Recording medium, display data generating device, and display data generating method

Abstract

The program causes the display data generating device (10) to function as the following functional units: a dynamic image acquisition unit (11) that acquires shooting data representing a shooting dynamic image obtained by shooting an environment including an instrument (30); an estimation unit (14) that estimates the position and orientation of the shooting device (20) that shot the shooting dynamic image based on the object shot in the shooting dynamic image; a log acquisition unit (13) that acquires log data representing a log of the action of the instrument (30) when the shooting dynamic image was shot; a model acquisition unit (12) that acquires model data representing a three-dimensional model of the instrument (30) and an object position corresponding to the object; and a display control unit (15) that generates and outputs display data for playing a model dynamic image and a shooting dynamic image in a time-synchronized manner and displaying the model dynamic image and the shooting dynamic image, the model dynamic image being obtained by changing the three-dimensional model according to the log and projecting the three-dimensional model configured based on the object position to the position and orientation estimated by the estimation unit (14).

Description

Technical Field

[0001] The present invention relates to a computer-readable non-transitory recording medium having a display data generation program recorded thereon, a display data generation device, and a display data generation method. Background Art

[0002] Factory Automation (FA) sites operate multiple instruments to perform various processes. When an anomaly occurs during one of these processes, the details of the anomaly are investigated by referencing previously recorded information. For example, post-process verification of images captured by the instrument or historical values ​​indicating the instrument's operating status is widely performed.

[0003] Comparing a captured image with a history of values ​​representing operational status is often difficult because it requires in-depth knowledge of the instrument. Therefore, for example, in situations where determining whether an object's dimensions fall within an acceptable range is necessary, a technique for overlaying a captured image of the object with a three-dimensional model of the object has been proposed to assist in abnormality analysis (see, for example, Patent Document 1). Specifically, by overlaying an image of the instrument and a three-dimensional model of the instrument to which a history of values ​​representing operational status has been applied, a substantive comparison between the captured image and the values ​​representing operational status is facilitated.

[0004] Patent Document 1: Japanese Patent Application Laid-Open No. 2021-96610 Summary of the Invention

[0005] Utilizing the technology of Patent Document 1 as described above makes it easy to verify whether an instrument is operating as intended at a certain timing. However, phenomena associated with anomalies in the field are not limited to instruments. Even if an instrument is operating as intended, anomalies may occur due to workpieces, equipment different from the instrument, operators taking unexpected actions, moving objects such as pests, and other objects. Furthermore, even if such an object is captured in an image, it is difficult to determine the correlation between the captured object and the anomaly or the object's impact on the instrument's operation. Therefore, there is room for further assistance to analysts who analyze the details of anomalies at the FA site.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to further assist an analyst who analyzes the details of an abnormality at a FA site.

[0007] In order to achieve the above-mentioned purpose, the display data generation program of the present invention enables a computer to function as the following units: a dynamic image acquisition unit, which acquires shooting data, which represents a first dynamic image obtained by shooting an instrument; an estimation unit, which estimates the position and orientation of a shooting device that shoots the first dynamic image based on the object shot in the first dynamic image; a log acquisition unit, which acquires log data, which represents a log of the action of the instrument when the first dynamic image is shot; a model acquisition unit, which acquires model data, which represents a three-dimensional model of the instrument and an object position corresponding to the object; and a display data generation unit, which generates and outputs display data, which is used to play a second dynamic image and a first dynamic image in a manner synchronized with time and display the second dynamic image and the first dynamic image, wherein the second dynamic image is obtained by changing the three-dimensional model according to the log and projecting the three-dimensional model configured based on the object position to the position and orientation estimated by the estimation unit.

[0008] Effects of the Invention

[0009] According to the present invention, a display data generation unit generates and outputs display data for playing and displaying a second dynamic image in a time-synchronized manner alongside the first dynamic image. The second dynamic image is obtained by changing the three-dimensional model of the instrument according to the log and projecting the three-dimensional model, configured based on the object's position, onto the position and orientation estimated by the estimation unit. Therefore, the second dynamic image, whose viewpoint of projecting the three-dimensional model is substantially the same as the viewpoint from which the instrument was captured, is played along with the first dynamic image in a time-synchronized manner. This allows analysts to easily compare the state of the actual environment containing the instrument with the series of instrument movements represented by the log, making it easier to determine the relevance of captured objects other than the instrument to an anomaly or the impact of such objects on the instrument's movement. This further assists analysts in analyzing the details of an anomaly at the FA site. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a diagram showing the configuration of a display system according to Embodiment 1.

[0011] Figure 2 This is a diagram showing an overview of display by the display system according to the first embodiment.

[0012] Figure 3 This is a diagram for explaining the placement of a workpiece according to the first embodiment.

[0013] Figure 4 This is a diagram showing normal placement of a workpiece according to the first embodiment.

[0014] Figure 5 This is a diagram showing abnormal placement of a workpiece according to the first embodiment.

[0015] Figure 6 This is a diagram showing the hardware configuration of the display data generating device according to the first embodiment.

[0016] Figure 7 This is a diagram showing an example of imaging data according to the first embodiment.

[0017] Figure 8 This is a diagram showing an example of model data according to the first embodiment.

[0018] Figure 9 This is a diagram showing an example of log data according to the first embodiment.

[0019] Figure 10 This is a diagram for explaining the estimation of the position and orientation of the imaging device according to the first embodiment.

[0020] Figure 11 This is a flowchart showing the display processing involved in the first embodiment.

[0021] Figure 12 This is a diagram showing a display example implemented by the display data generating device according to the first embodiment.

[0022] Figure 13 This is a diagram showing an example of region extraction according to the second embodiment.

[0023] Figure 14 This is a diagram showing a display example implemented by the display data generating device according to the second embodiment.

[0024] Figure 15 A diagram showing a display example implemented by a display data generating device according to a modification.

[0025] Figure 16 This is a diagram showing an example of an object according to a modification.

[0026] Figure 17 It is a diagram showing a configuration example of a display data generating device according to a modification. DETAILED DESCRIPTION

[0027] Hereinafter, the display data generating device 10 according to the embodiment of the present invention will be described in detail with reference to the drawings.

[0028] Implementation Method 1

[0029] like Figure 1As shown, the display data generating device 10 involved in this embodiment, together with the imaging device 20 for imaging the instrument 30 and the instrument 30 controlled by the control device 31, constitutes a display system 100. The display system 100 is a system constructed for the purpose of abnormality analysis in a facility represented by a factory. Here, an abnormality refers to a state that is determined by the manager to be out of the normal operating range of the facility. For example, damage to the workpiece processed by the instrument 30 and the deviation of the movement path of the workpiece from the pre-designed action route are equivalent to abnormalities. In addition, one abnormality may cause other abnormalities. For example, due to an abnormality related to the workpiece, the instrument 30 may become abnormal.

[0030] like Figure 2 As shown, the display system 100 synthesizes a dynamic image 101 obtained by photographing the environment of the device 30 and a dynamic image 102 of a three-dimensional model drawn based on the operation log of the device 30 and displays them on a screen 103 , thereby assisting the analyst in abnormality analysis.

[0031] The imaging device 20 is a camera having an imaging element, such as a camera recording unit connected to a PLC (Programmable Logic Controller) or a surveillance camera operating independently of the PLC. The imaging device 20 transmits captured data representing a dynamic image to the display data generation device 10 via an industrial network. The dynamic image includes frame images obtained by periodically capturing the instrument 30 and its surroundings. The frame rate of the dynamic image captured by the imaging device 20 is, for example, 10 fps (frames per second) or 30 fps. The imaging device 20 can capture dynamic images using either visible light or infrared light.

[0032] The instrument 30 is an FA instrument having a movable part. The following description focuses on an example in which the instrument 30 is an industrial robot having a robot arm having four axes of freedom of movement. The instrument 30 operates according to control instructions from the control device 31. In detail, Figure 2 As shown in the upper part of , the instrument 30 repeatedly performs a series of operations of gripping the workpiece 300 placed on the tray 301 and moving it into the inspector 302 , and then gripping the inspected workpiece 300 again and moving it to the tray 303 .

[0033] like Figure 3 As shown in FIG. 3 , it is assumed that the workpiece 300 is placed on the tray 301 while the axis of the workpiece 300 is kept horizontal. Figure 3 The cross-sectional view at line AA' Figure 4As shown, if one end on the left side and the other end on the right side of the workpiece 300 are horizontal, the workpiece 300 is normally placed on the pallet 301 .

[0034] On the other hand, Figure 5 As shown in FIG. 3 , if one end and the other end of the workpiece 300 are tilted relative to the tray 301, it can be said that the workpiece 300 is not properly placed on the tray 301. Such a placement may cause an abnormality in the gripping state of the instrument 30, an inappropriate inspection by the inspector 302, or an abnormal stop of the instrument 30, but it is not easy to Figure 2 One captured image such as the one shown in the upper portion directly identifies such a placement state as the cause of the abnormality. In addition, although the operation history of the instrument 30 is widely recorded, the posture of the workpiece 300 is usually not recorded as a sensing target.

[0035] In addition, Figure 4 、 5 If a sensor 304 for detecting the presence of a workpiece 300 is disposed at the position indicated by the hollow circle in the figure, the output of the sensor 304 is ON during normal loading and OFF during abnormal loading. By referring to this output, it is easier to determine that an abnormality has occurred in the pallet 301 than when the sensor 304 is not disposed.

[0036] However, in order to verify whether the workpiece 300 is not on the pallet 301 or whether the posture of the workpiece 300 is abnormal, it is necessary to confirm the captured image. Moreover, even if the abnormal posture of the workpiece 300 is confirmed from the image, it is not easy to verify whether the abnormal posture is caused by another abnormality or whether it is the cause of the other abnormality from a single captured image.

[0037] Return to Figure 1 The control device 31 is a control device represented by a PLC (Programmable Logic Controller). The control device 31 controls the instrument 30 by executing a pre-set ladder diagram program. Specifically, the control device 31 controls the values ​​of parameters represented by the rotation angle and angular velocity of each of the four axes of the instrument 30 via a servo amplifier. The control device 31 includes a recording unit that records a log of the parameter values ​​and transmits log data representing the log to the display data generation device 10 via a communication line such as a USB (Universal Serial Bus) cable.

[0038] The display data generating device 10 is a computer represented by an industrial PC (Personal Computer) and a tablet terminal. The display data generating device 10 has a hardware structure such as Figure 6 As shown, the processor 41 includes a main storage unit 42, an auxiliary storage unit 43, an input unit 44, an output unit 45, and a communication unit 46. The main storage unit 42, the auxiliary storage unit 43, the input unit 44, the output unit 45, and the communication unit 46 are connected to the processor 41 via an internal bus 47.

[0039] The processor 41 includes a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) as a processing circuit. The processor 41 implements various functions of the display data generating device 10 and performs the processing described below by executing a program 48 stored in the auxiliary storage unit 43. The program 48 is an example of a display data generation program.

[0040] The main storage unit 42 includes a RAM (Random Access Memory). The program 48 is loaded from the auxiliary storage unit 43 into the main storage unit 42. The main storage unit 42 is used as a work area for the processor 41.

[0041] The auxiliary storage unit 43 includes nonvolatile memory such as EEPROM (Electrically Erasable Programmable Read-Only Memory) and HDD (Hard Disk Drive). In addition to the program 48, the auxiliary storage unit 43 also stores various data used by the processor 41. Following instructions from the processor 41, the auxiliary storage unit 43 supplies data used by the processor 41 and stores data supplied from the processor 41.

[0042] The input unit 44 includes input devices such as a keyboard and a pointing device, and acquires information input by an analyst, that is, a user of the display data generating device 10 , and notifies the processor 41 of the acquired information.

[0043] The output unit 45 includes output devices such as an LED (Light Emitting Diode), an LCD (Liquid Crystal Display), and a speaker. The output unit 45 presents various information to the user according to the instructions of the processor 41.

[0044] The communication unit 46 includes a communication interface circuit for transmitting and receiving signals to and from an external device. The communication unit 46 receives a signal from the outside and outputs the data indicated by the signal to the processor 41. In addition, the communication unit 46 transmits a signal indicating the data output from the processor 41 to an external device.

[0045] The above hardware works in coordination, so that the display data generating device 10 can perform various functions. Figure 1 As shown, the system comprises: a dynamic image acquisition unit 11 that acquires a dynamic image of the instrument 30 captured by the camera 20; a model acquisition unit 12 that acquires model data representing a three-dimensional model of the instrument 30; a log acquisition unit 13 that acquires log data representing a log of the operation of the instrument 30; an estimation unit 14 that estimates the position of the camera 20 based on the captured dynamic image; a display control unit 15 that generates display data representing content to be displayed and displays it on the display unit 16; and a display unit 16 having a screen 103 that displays the content of the display data. The display unit 16 is mainly implemented by the output unit 45.

[0046] The dynamic image acquisition unit 11 is mainly implemented by the communication unit 46. The dynamic image acquisition unit 11 receives the shooting data representing the dynamic image from the shooting device 20 by requesting the shooting device 20 to provide the dynamic image captured within the time range including the time specified by the user. In the following, the captured dynamic image is sometimes referred to as a captured dynamic image. In addition, the acquisition of the shooting data implemented by the dynamic image acquisition unit 11 is not limited to receiving it from the shooting device 20, and it can also be read from a recording medium represented by a memory card or an external server device. The captured dynamic image is equivalent to an example of a first dynamic image obtained by capturing an environment including an instrument, and the dynamic image acquisition unit 11 is equivalent to an example of a dynamic image acquisition unit that obtains the shooting data representing the first dynamic image in the display data generation device 10.

[0047] exist Figure 7 An example of the captured data is shown schematically in FIG. Figure 7 As shown, the shooting data represents multiple frame images in association with their respective shooting dates and times. However, the shooting data may also include a value indicating the start date and time when the initial frame image was captured and a value indicating the frame rate, thereby effectively indicating the shooting date and time of each frame image. If the shooting data includes these values, the date and time when each frame image was captured is calculated based on the start date and time, frame rate, and frame number. Furthermore, the method for indicating the shooting time of each frame image is not limited to this and can be arbitrarily modified.

[0048] Return to Figure 1The model acquisition unit 12 is mainly implemented by the collaborative action of the processor 41, the input unit 44 and the communication unit 46. The model acquisition unit 12 obtains model data representing a three-dimensional model that simulates the shape of the instrument 30 from the user. In detail, the model acquisition unit 12 obtains the model data by reading the model data from a recording medium specified by the user or an address of an external server device. The shape of the instrument 30 is predetermined by a mechanical design using 3D CAD (3-Dimensions Computer Aided Design) software application. The model data is data created in such a mechanical design, and shows the spatial coordinates of the points, lines and surfaces that constitute the three-dimensional model.

[0049] exist Figure 8 An example of model data is schematically shown in FIG. Figure 8 As shown, the model data shows a three-dimensional model of the instrument using spatial coordinates and shows the object position. Figure 8 The three-dimensional model of the planar support table for supporting the instrument 30 is also shown, but the three-dimensional model of the support table can also be omitted. The object position shows the position corresponding to the object captured in the dynamic image. Figure 8 In the example, the positions of three markers representing the objects are shown as object positions. The object positions are used to determine the scale and angle of the three-dimensional model. The determination of scale and angle based on the object positions will be described in detail later. The model acquisition unit 12 in the display data generation device 10 serves as an example of a model acquisition unit that acquires model data representing the three-dimensional model of the instrument and the object positions corresponding to the objects.

[0050] The log acquisition unit 13 is mainly implemented by the communication unit 46. The log acquisition unit 13 receives log data from the control device 31 by requesting the control device 31 to provide a log recorded in a time range including the time specified by the user. Since the time specified by the user is the same as the time specified for the dynamic image acquisition unit 11, the time range for creating the log acquired by the log acquisition unit 13 overlaps with the time range of the shooting of the dynamic image acquired by the dynamic image acquisition unit 11. In addition, the acquisition of log data by the log acquisition unit 13 is not limited to receiving from the control device 31. The log acquisition unit 13 can also acquire log data by reading from a recording medium or an external server device. In the case where the instrument 30 records a log, the log data can also be acquired from the instrument 30. The log acquisition unit 13 is equivalent to an example of a log acquisition unit that acquires log data in the display data generation device 10, and the log data shows a log of the operation of the instrument when the first dynamic image is captured.

[0051] The log data is information that can reproduce the movement of the movable part of the instrument 30. Figure 9 An example of log data is shown in Figure 9 In the example shown in FIG, the log data shows the angles of the first, second, third, and fourth axes of the instrument 30 in chronological order, with one rotation being 360 degrees, in association with the recording date and time. Each record constituting the log data may be a regular record or an irregular record.

[0052] The estimation unit 14 is mainly implemented by the processor 41. The estimation unit 14 obtains the captured dynamic image from the dynamic image acquisition unit 11 and estimates the position and direction of the imaging device 20 based on the object captured in the captured dynamic image. Figure 10 As shown in FIG1 , three different markers pre-pasted on the support table of the instrument 30 are detected as objects. Then, the estimation unit 14 estimates the position and orientation of the camera 20 based on the coordinates on the image captured by the detected object. This estimation is performed using a known solution algorithm for the PnP (Perspective-n-Point) problem. Here, the coordinates of the markers pasted as objects are known. In addition, the position and orientation of the camera 20 are specified in the same coordinate system as the coordinates of the markers. The estimation unit 14 is equivalent to an example of an estimation unit in the display data generating device 10 that estimates the position and orientation of the camera that captured the first dynamic image based on the object captured in the first dynamic image.

[0053] The display control unit 15 is mainly implemented by the processor 41. The display control unit 15 obtains model data from the model acquisition unit 12 and log data from the log acquisition unit 13, and changes the three-dimensional model according to the log. Specifically, the display control unit 15 changes the movable part model corresponding to the movable part in the three-dimensional model based on the value shown in the log. For example, the display control unit 15 changes the movable part model corresponding to the movable part in the three-dimensional model according to the value shown in the log. Figure 9 The display control unit 15 rotates the movable part model corresponding to the first axis so that the angle of the movable part model matches the "30" angle indicated in the log. The display control unit 15 similarly rotates the movable part model for the other axes. Furthermore, the display control unit 15 sequentially applies this rotation to the three-dimensional model for the other records that record dates and times.

[0054] Furthermore, as described above, the display control unit 15 virtually configures the three-dimensional model, which changes according to the log, based on the object's position. Here, to define the coordinate system in the real environment, the object is attached to a support that remains stationary in space. In the model data, the object's position is defined in a coordinate system common to the three-dimensional model. Therefore, the three-dimensional model is defined in a coordinate system substantially common to the instrument 30 in the real environment. The display control unit 15 virtually operates this three-dimensional model according to the log.

[0055] Furthermore, the display control unit 15 obtains the estimation result of the estimation unit 14 from the estimation unit 14, and projects the three-dimensional model to the estimated position and orientation, thereby creating a dynamic image. That is, the display control unit 15 creates a dynamic image of the three-dimensional model that is photographed from the same position and orientation as the photographing device 20 that photographed the instrument 30. In the following, the dynamic image obtained by projecting the three-dimensional model is sometimes referred to as a model dynamic image. Figure 10 2 shows an example in which the estimated position of the imaging device 20 is represented by the coordinates (X1, Y1, Z1) and the estimated orientation is represented by (θ, ψ). Here, θ corresponds to the azimuth angle, and ψ corresponds to the elevation angle.

[0056] In addition, it is preferred that the projection method of the three-dimensional model and the projection method from the instrument 30 to the captured dynamic image be the same, but these projection methods may also be different. For example, the estimation unit 14 may also estimate the field angle and aberration as the shooting range in addition to the above-mentioned position and orientation, and the display control unit 15 may apply the estimated field angle and aberration to the projection of the three-dimensional model. In addition, the display control unit 15 may also generate a model dynamic image by parallel projecting the three-dimensional model onto a plane specified by the estimated position and orientation of the imaging device 20. In the case of the model dynamic image formed by parallel projection, parameters such as aberration are different from those of the captured dynamic image, but the model dynamic image can be a dynamic image that allows the user to verify the actions that comply with the log of the instrument 30.

[0057] Furthermore, the display control unit 15 plays and displays the generated model dynamic image on the display unit 16 through a 3D simulator. Here, the 3D simulator is a software application for displaying production equipment or control devices and their action simulation results in 3D. In addition, the display control unit 15 displays a semi-transparent captured dynamic image in a region on the screen 103 of the display unit 16 where the model dynamic image is displayed, overlapping with the model dynamic image. That is, the frame image of the model dynamic image generated based on the record of a specific recording date and time contained in the log and the frame image of the captured dynamic image captured at the same date and time as the recording date and time are displayed in sequence in an overlapping manner. The display control unit 15 generates display data for realizing such playback display of the model dynamic image and synchronous display with the captured dynamic image, and outputs it to the display unit 16, thereby causing these dynamic images to be displayed in an overlapping manner on the screen of the display unit 16.

[0058] Furthermore, regarding the simultaneously displayed frames of the model dynamic image and the captured dynamic image, the date and time recorded in the log used to generate the model dynamic image and the date and time of the captured dynamic image do not need to be completely consistent; a certain degree of synchronization error is permitted. For example, a synchronization error of 10ms or 100ms is permitted. A synchronization error can also occur within a range that does not affect the analysis of anomalies when the analyst visually recognizes and compares the model dynamic image and the captured dynamic image simultaneously. The display control unit 15 can simply play the model dynamic image and the captured dynamic image simultaneously while the synchronization error remains within a certain range.

[0059] The model dynamic image corresponds to an example of a second dynamic image, which is obtained by changing the three-dimensional model according to the log and projecting the three-dimensional model, which is arranged based on the object position, to the position and orientation estimated by the estimation unit. Furthermore, the display control unit 15 corresponds to an example of a display data generation unit in the display data generation device 10, which generates and outputs display data for displaying the second dynamic image and the first dynamic image by playing them synchronously.

[0060] Next, use Figures 11-12 The display processing executed by the display data generating device 10 will be described. Figure 11 The display processing shown corresponds to an example of a display data generating method executed by the display data generating device 10 .

[0061] During the display process, the processor 41 of the display data generating device 10 determines whether a time has been specified by the user (step S1). This time corresponds to the date and time at which the user requested the dynamic image to be played. Furthermore, in the determination of step S1, it is also possible to determine whether a start time and an end time have been specified, whether a length of time for the dynamic image to be played has been specified along with the start time, and whether a time range before and after the specified time has been specified along with the specified time. Alternatively, the user may specify a time with a predetermined length of time before and after. Regarding the determination of step S1, a yes result is sufficient if at least a portion of the time at which the dynamic image to be played has been determined.

[0062] If the determination in step S1 is negative (step S1: No), the display data generation device 10 repeats the determination in step S1 and waits for the user to specify a time. On the other hand, if the determination in step S1 is positive (step S1: Yes), the video acquisition unit 11 acquires the captured video captured within the range including the time specified in step S1 (step S2). Furthermore, the log acquisition unit 13 acquires the logs recorded within the range including the time specified in step S1 (step S3). Furthermore, the model acquisition unit 12 acquires model data provided by the user (step S4).

[0063] Next, the estimation unit 14 estimates the position and orientation of the camera 20 based on the object captured in the captured dynamic image indicated by the captured data obtained in step S2 (step S5). The estimation unit 14 may also estimate the position and orientation of the camera 20, which was fixed from the beginning to the end of the captured dynamic image, based on the object captured in one frame image. The estimation unit 14 may also improve the estimation accuracy of the position and orientation of the fixed camera 20 based on the objects captured in multiple frame images. In addition, the estimation unit 14 may sequentially estimate the position and orientation of the unfixed camera 20 for each frame image.

[0064] Next, the display control unit 15 generates a model dynamic image (step S6) by causing the three-dimensional model shown in the model data obtained in step S4 to change according to the log shown in the log data obtained in step S3, and projecting the three-dimensional model to the position and direction estimated in step S5. Specifically, the display control unit 15 applies the records corresponding to the recording dates and times contained in the log to the three-dimensional model, and then projects based on the estimation result of step S5, thereby generating a frame image corresponding to the recording date and time. Here, if the camera 20 is not fixed and there is no estimation result for the same shooting date and time as the recording date and time, the display control unit 15 can use the most recent estimation result or linearly interpolate the previous and next estimation results.

[0065] Next, the display control unit 15 generates display data for playing the captured dynamic image and the model dynamic image in synchronization with each other and outputs the display data to the display unit 16 (step S7). Figure 12 The user's operation of the play button 104 and the pause button 105 controls the playback of the superimposed captured dynamic image and the model dynamic image. As a result, the display unit 16 displays the two dynamic images based on the display data (step S8). The display process then ends.

[0066] Furthermore, the content and order of each step in the above-described display process can be arbitrarily changed. For example, the order of steps S2, S3, and S4 can be reversed, or these steps can be performed in parallel. Furthermore, while step S2 must be performed before step S5, steps S3 and S4 are not subject to this requirement and can simply be performed before step S6.

[0067] As described above, the display control unit 15 generates and outputs display data for playing and displaying a model dynamic image and a captured dynamic image in a time-synchronized manner. This model dynamic image is obtained by changing the instrument's three-dimensional model according to the log and projecting the three-dimensional model, arranged based on the object's position, onto the position and orientation estimated by the estimation unit 14. Therefore, the model dynamic image, whose viewpoint of projection of the three-dimensional model is substantially the same as the viewpoint of the instrument's image, is played along with the captured dynamic image in a time-synchronized manner. This allows analysts to easily compare the state changes of the actual environment including the instrument with the series of instrument movements represented in the log, making it easier to determine the relevance of captured objects other than the instrument to anomalies or their impact on instrument operation. This further assists analysts in analyzing the details of anomalies at the FA site.

[0068] Furthermore, analysts can combine 3D models with real-world images to confirm the status of production equipment and FA devices when they fail. This makes it easier to pinpoint the cause of the failure, enabling faster recovery and improving FA device availability.

[0069] Implementation Method 2

[0070] Next, Embodiment 2 will be described, focusing on the differences from Embodiment 1. Components identical or equivalent to those in Embodiment 1 will be assigned the same reference numerals, and their descriptions will be omitted or simplified. This embodiment differs from Embodiment 1 in that the results of the comparison between the model dynamic image and the captured dynamic image are emphasized.

[0071] like Figure 13As shown, the display control unit 15 involved in this embodiment extracts a set of pixels that have changed between the frames of the captured dynamic image, namely, area A11, and a set of pixels that have not changed, namely, area A12. Here, the pixels that have not changed can also be pixels whose pixel value change is less than a predetermined threshold. Similarly, the display control unit 15 extracts a set of pixels that have changed between the frames of the model dynamic image, namely, area A21, and a set of pixels that have not changed, namely, area A22. Then, the display control unit 15 extracts an area A30 corresponding to the difference between the area A11 that has changed in the captured dynamic image and the area A21 that has changed in the model dynamic image. In detail, the set of pixels in area A21 that are not included in area A11 is extracted as area A30.

[0072] Here, unchanged area A12 represents the area of ​​the imaged instrument 30 and its surroundings that remains unchanged, while unchanged area A22 represents the area of ​​the 3D model that remains unchanged. Neither area A12 nor A22 should be of interest to the analyst. Furthermore, the overlap between areas A11 and A21 that exhibit change is assumed to be due to the instrument 30 operating as expected, and therefore should not be of interest to the analyst.

[0073] On the other hand, area A30 corresponds to an area where the instrument 30 is not operating as expected, or an area where an object not depicted as a three-dimensional model is captured in the captured dynamic image. Therefore, it is possible that this area is associated with an abnormality and should be brought to the attention of the analyst. Furthermore, area A11 corresponds to an example of a first area that has changed in the captured dynamic image, and area A21 corresponds to an example of a second area that has changed in the model dynamic image.

[0074] Therefore, the display control unit 15 is Figure 14 The portion extracted as area A30 between the preceding and following frame images is highlighted and displayed by thick shading. Specifically, the display control unit 15 overlays and displays the portion corresponding to area A30 in the captured dynamic image with the model dynamic image, making the transparency lower than that of other portions. Alternatively, area A30 can be emphasized by flashing the dotted line surrounding area A30.

[0075] As described above, the display control unit 15 corresponds to an example of a display data generating unit that generates display data for emphasizing a portion of the first area based on a comparison between the first area that changes in the captured dynamic image and the second area that changes in the model dynamic image.

[0076] While the example described above emphasizes a portion of region 11, considering that changes in the captured dynamic image may differ from those in the model dynamic image, the entire region where changes have occurred in the captured dynamic image may also be emphasized. Alternatively, the amount of change in each pixel in the captured dynamic image may be compared with the amount of change in each pixel in the model dynamic image, and a set of pixels where the difference in these amounts of change exceeds a predetermined threshold may be extracted as region A30.

[0077] As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment.

[0078] For example, an example in which the captured moving image is displayed in a superimposed manner on the model moving image has been described, but the model moving image may also be displayed in a superimposed manner on the captured moving image.

[0079] In addition, it is also possible to replace the display of overlapping the model dynamic image and the shooting dynamic image, such as Figure 15 As shown, the model dynamic image and the shot dynamic image are displayed side by side and played synchronously.

[0080] In addition, while the example described above shows the capture date and time of each frame of a moving image, and the log data shows the recording date and time of each record, this is not limiting. For example, the capture data may show the length of time that elapsed from the activation of the control device 31 until each frame was captured, and the log data may show the length of time that elapsed from the activation of the control device 31 until each record was recorded. The capture time of each frame image shown in the capture data and the recording time of each record shown in the log data may be based on a common reference time.

[0081] Furthermore, although an example has been described in which the model data indicates the object position in addition to the three-dimensional model, the present invention is not limited thereto, and the model acquisition unit 12 may acquire the object position separately from the model data.

[0082] Furthermore, the estimating unit 14 detects the object captured in the captured moving image, but the position of the object captured in the captured moving image on the image may be designated by the user.

[0083] In addition, the example of using three marks as objects has been described, but the number of marks may be greater than or equal to 4. In addition, objects different from marks may be used. For example, a mark may be printed in advance as an object. Figure 16 A mark of a two-dimensional pattern as shown in the upper part of is affixed to the support, according to Figure 16 The position and orientation of the imaging device 20 are estimated based on the deformation state of the captured object as shown in the lower part of .

[0084] In addition, the display data generating device 10 may also be as follows Figure 17 The display control unit 15 is replaced by a display data generating unit 15a that outputs display data to an external display device 160. For example, a cloud server on the Internet, i.e., the display data generating unit 10, may distribute a web page containing a script for synchronously playing the model dynamic image and the captured dynamic image as display data to the user terminal, i.e., the display device 160. Figure 17 In the example of , the display data generating unit 15a corresponds to an example of a display data generating means.

[0085] Furthermore, the functions of the display data generating device 10 can also be realized by dedicated hardware or a general computer system.

[0086] For example, the program 48 executed by the processor 41 may be stored in a computer-readable non-transitory recording medium and distributed, and the program 48 may be installed in a computer, thereby forming a device that executes the above-mentioned processing. Examples of such recording media include floppy disks, CD-ROMs (Compact Disc Read-Only Memory), DVDs (Digital Versatile Discs), and MOs (Magneto-Optical Discs).

[0087] Alternatively, the program 48 may be stored in advance on a disk device of a server device on a communication network such as the Internet, and may be superimposed on a carrier wave and downloaded to a computer.

[0088] Furthermore, the above-described processing can also be realized by transferring the program 48 via a communication network and starting its execution.

[0089] Furthermore, the above-described processing can also be realized by executing all or part of the program 48 on a server device, and by having the computer execute the program while transmitting and receiving information related to the processing via a communication network.

[0090] Furthermore, when the above functions are realized by sharing the responsibility of the OS (Operating System) or when the above functions are realized by cooperation between the OS and applications, only the parts other than the OS may be stored in a medium and distributed, or downloaded to a computer.

[0091] Furthermore, the means for realizing the functions of the display data generating device 10 is not limited to software, and a part or all of the functions may be realized by dedicated hardware including circuits.

[0092] The present invention can be implemented in various embodiments and variations without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are provided to illustrate the present invention and are not intended to limit the scope of the present invention. That is, the scope of the present invention is not defined by the embodiments but by the claims. Furthermore, variations implemented within the scope of the claims and their equivalents are considered to fall within the scope of the present invention.

[0093] Industrial Applicability

[0094] The present invention is suitable for analyzing faults occurring at FA sites.

[0095] Description of the label

[0096] 100 display system, 10 display data generation device, 11 dynamic image acquisition unit, 12 model acquisition unit, 13 log acquisition unit, 14 estimation unit, 15 display control unit, 15a display data generation unit, 16 display unit, 20 imaging device, 30 instrument, 31 control device, 41 processor, 42 main storage unit, 43 auxiliary storage unit, 44 input unit, 45 output unit, 46 communication unit, 47 internal bus, 48 ​​program, 101, 102 dynamic image, 103 screen, 104 play button, 105 pause button, 160 display device, 300 workpiece, 301, 303 trays, 302 inspector, 304 sensor.

Claims

1. A computer-readable non-transitory recording medium having a display data generation program recorded thereon, the display data generation program causing a computer to function as: a dynamic image acquisition unit that acquires imaging data representing a first dynamic image obtained by imaging an environment including the device; an estimating unit for estimating a position and a direction of a camera that captured the first moving image based on an object captured in the first moving image; a log acquisition unit that acquires log data representing a log of the operation of the device when the first moving image is captured; a model acquisition unit that acquires model data representing a three-dimensional model of the apparatus and an object position corresponding to the object; and A display data generating unit generates and outputs display data for displaying a second dynamic image played in a time-synchronized manner and the first dynamic image, wherein the second dynamic image is obtained by changing the three-dimensional model according to the log and projecting the three-dimensional model configured based on the object position to the position and orientation estimated by the estimating unit.

2. The recording medium according to claim 1, wherein The display data is data for displaying the first moving image and the second moving image in a superimposed manner.

3. The recording medium according to claim 1, wherein The display data is data for displaying the first moving image and the second moving image in parallel.

4. The recording medium according to any one of claims 1 to 3, wherein The display data generating unit generates the display data for highlighting a portion or all of the first region based on a comparison between a first region that has changed in the first moving image and a second region that has changed in the second moving image.

5. A display data generating device for generating display data for displaying a first dynamic image obtained by photographing an environment including an instrument using a photographing device, The display data generating device comprises: a moving image acquisition unit that acquires captured data representing the first moving image; an estimating unit configured to estimate a position and an orientation of the imaging device based on an object captured in the first dynamic image; a log acquisition unit that acquires log data representing a log of the operation of the device when the first moving image is captured; a model acquisition unit that acquires model data representing a three-dimensional model of the apparatus and an object position corresponding to the object; and A display data generating unit generates and outputs the display data for displaying a second dynamic image played in a time-synchronized manner and the first dynamic image, wherein the second dynamic image is obtained by changing the three-dimensional model according to the log and projecting the three-dimensional model configured based on the object position to the position and orientation estimated by the estimating unit.

6. A method for generating display data, comprising: The moving image acquisition unit acquires imaging data representing a first moving image obtained by imaging an environment including the device. The estimating unit estimates the position and orientation of the imaging device that captured the first moving image based on the object captured in the first moving image. The log acquisition unit acquires log data indicating a log of the operation of the device when the first moving image is captured. The model acquisition unit acquires model data representing a three-dimensional model of the device and an object position corresponding to the object. The display data generation unit generates and outputs display data for displaying a second dynamic image played in a time-synchronized manner and the first dynamic image, wherein the second dynamic image is obtained by changing the three-dimensional model according to the log and projecting the three-dimensional model configured based on the object position to the position and orientation estimated by the estimation unit.

Images