Display control device, display control method and program

Abstract

The display control device (10) includes an acquisition unit (11) that acquires time-series control data (131) that indicates device values ​​used to control a controlled device in association with time, and image data (132) that includes images of an object related to control repeatedly photographed at timings corresponding to the times associated with the control values, an image processing unit (14) that processes the images to obtain result values ​​related to the position of the object, and a display control unit (15) that displays on a display unit (16) the progress of the device values ​​indicated by the control data and the progress of the result values ​​in a form that corresponds them based on a common reference time.

Description

[Technical Field]

[0001] The present disclosure relates to a display control device, a display control method, and a program. [Background technology]

[0002] In FA (Factory Automation) sites, processes are carried out by operating equipment to process objects. In the event of an abnormality occurring in this process or for the purpose of improving yield, log data relating to the operating equipment may be compared and matched with log data relating to the objects being processed (see, for example, Patent Document 1).

[0003] Patent Document 1 describes a technology for extracting image features such as brightness or the amount of received light from an image captured by a camera, and monitoring an object using time-series data of the image features and time-series data of device values ​​of a programmable controller. This technology makes it possible to compare the state of the captured object, which appears as image features, with device values ​​corresponding to the operation of the equipment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-189481 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of Patent Document 1, time-series data of image features is created from background parts other than the object shown in the image. Therefore, the time-series data of image features does not necessarily represent the state of the object that is of interest in the processing process performed by the operation of the equipment. This could result in a decrease in the efficiency of the matching work using images captured of the processing process performed at the factory automation site.

[0006] The present disclosure has been made in light of the above-mentioned circumstances, and aims to improve the efficiency of matching work using images captured of processing steps carried out at an FA site. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the display control device of the present disclosure comprises an acquisition means for acquiring time-series control data that indicates control values ​​used to control a controlled device in association with time, and image data including images of an object related to control repeatedly photographed at timings corresponding to the times associated with the control values, an image processing means for obtaining a result value related to the position of the object by processing the image, and a display control means for displaying on a display device the progress of the control values ​​indicated by the control data and the progress of the result value in a form that corresponds based on a common reference time. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to improve the efficiency of matching work using images captured of processing steps carried out at the FA site. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of a support system according to an embodiment. [Figure 2] 1 is a diagram showing an example of a processing step according to an embodiment; [Figure 3] FIG. 1 is a diagram showing a hardware configuration of a display control device according to an embodiment. [Figure 4] FIG. 1 is a diagram showing a functional configuration of a display control device according to an embodiment; [Figure 5] FIG. 1 is a diagram showing an example of control data and image data according to an embodiment; [Figure 6] 1 is a flowchart showing a display control process according to an embodiment; [Figure 7] FIG. 1 is a first diagram showing an example of a setting screen for image processing parameters according to an embodiment; [Figure 8]FIG. 2 is a second diagram showing an example of a setting screen for image processing parameters according to an embodiment; [Figure 9] FIG. 3 is a third diagram showing an example of a setting screen for image processing parameters according to an embodiment; [Figure 10] FIG. 4 is a fourth diagram showing an example of a setting screen for image processing parameters according to an embodiment; [Figure 11] FIG. 10 is a diagram showing an example of a setting screen for graph display according to an embodiment; [Figure 12] FIG. 10 is a diagram showing an example of a setting screen for conversion parameters according to an embodiment; [Figure 13] FIG. 10 is a diagram showing an example of conversion into physical units according to an embodiment. [Figure 14] FIG. 1 is a diagram for explaining a noise removal process according to an embodiment; [Figure 15] FIG. 10 is a diagram for explaining adjustment of the scale and offset value of a graph according to an embodiment. [Figure 16] FIG. 1 is a diagram for explaining a least squares error according to an embodiment. [Figure 17] FIG. 1 is a first diagram showing an example of a graph display according to an embodiment; [Figure 18] FIG. 10 is a diagram showing an example of a flag setting screen according to an embodiment; [Figure 19] FIG. 2 is a second diagram showing an example of a graph display according to an embodiment; [Figure 20] FIG. 10 is a diagram showing an example of a graph display according to a modified example. [Figure 21] FIG. 1 is a first diagram showing the correspondence between control data and image data according to a modified example. [Figure 22] FIG. 2 is a second diagram showing the correspondence between control data and image data according to a modified example. [Figure 23] FIG. 10 is a diagram showing a functional configuration of a display control device according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure relates to a system for supporting a manager who manages a process performed by controlling controlled devices in a facility such as a factory or a plant.

[0011] Embodiment The support system 1000 according to this embodiment supports the manager in the matching work by displaying the progress of control and the progress of the result values ​​obtained by image processing of the object of the processing process based on a common reference time.

[0012] As shown in FIG. 1, the support system 1000 includes a display control device 10 that controls the content of information displayed to the administrator, a PLC (Programmable Logic Controller) 20 that controls the controlled equipment, a servo amplifier 21 and a servo motor 22 that are the controlled equipment, and cameras 31 and 32 that photograph the workpiece 40 to be processed.

[0013] The PLC 20 is also referred to as a programmable controller. The PLC 20 controls the servo amplifier 21 and the servo motor 22 using device values ​​211 stored in an internal memory 210. The device values ​​211 are values ​​input to the PLC 20 from an external device including the servo amplifier 21, intermediate values ​​used in arithmetic processing by the PLC 20, or values ​​output from the PLC 20 to an external device including the servo amplifier 21. For example, when the PLC 20 outputs a result of calculation based on the sensing result of a sensor (not shown) as a command value to the servo amplifier 21, the sensing result, the intermediate value used in the calculation, and the command value can each be the device value 211. Because the device value 211 represents the control state in this way, a log of the device value 211 is used to represent the control history. The device value 211 corresponds to an example of a control value used to control a controlled device.

[0014] The servo amplifier 21 and the servo motor 22 are controlled by the PLC 20 to process the workpiece 40. The processing of the workpiece 40 may be transportation, machining, assembly, or other processing. The servo amplifier 21 is connected to the PLC 20 via a signal line, and is also connected to the servo motor 22 via a signal line and a power line.

[0015] The following description focuses on an example in which the workpiece 40 is processed by placing it on top of another workpiece 40a, as shown in FIG. 2. The workpiece 40 is transported by an arm 23 attached to a servo motor 22 as indicated by the thick arrow in FIG. 2, and the mark 41 on the workpiece 40 is positioned so that it overlaps the mark 41a on the workpiece 40a in the vertical Z-axis direction. This process is performed sequentially on multiple workpieces 40, but anomalies such as poor printing of the marks 41, 41a, poor machining of the workpieces 40, 40a, or interference between the arm 23 and the workpieces 40, 40a can cause the process to fail as intended. When such an anomaly occurs, the log of device values ​​211 is compared with the captured image of the workpiece 40 to identify the cause of the anomaly and resolve the issue. The workpieces 40, 40a and some or all of the marks 41, 41a correspond to examples of objects to be photographed.

[0016] Returning to Fig. 1, as shown in Fig. 1, a workpiece 40 to be processed is repeatedly photographed from different angles by cameras 31 and 32. The photographed images are transmitted to the display control device 10 via a communication line 301. The communication line 301 is, for example, a USB (Universal Serial Bus) cable or a Coaxpress-compatible cable. Alternatively, instead of the communication line 301, the images may be transmitted via a network such as a LAN (Local Area Network).

[0017] The cameras 31 and 32 simultaneously capture images upon receiving an I / O (Input / Output) signal transmitted from the PLC 20 via the communication line 202. Because the cameras 31 and 32 capture images triggered by a common I / O signal, the capture timing of the cameras 31 and 32 is synchronized. Furthermore, if the I / O signal is transmitted by the PLC 20 when reading out device values ​​and creating a log, the capture timing of the image and the recording timing of the device values ​​will be synchronized.

[0018] The display control device 10 is a UI (User Interface) terminal typified by an industrial PC (Personal Computer), and is connected to the PLC 20 via a communication line 201 such as a USB cable. The display control device 10 functions as an engineering tool for creating and editing a control program that defines the control content to be performed by the PLC 20, and may write the control program to the PLC 20 and have the PLC 20 execute it.

[0019] The display control device 10 has a hardware configuration as shown in Fig. 3. That is, the display control device 10 is configured as a computer having a processor 101, a main memory unit 102, an auxiliary memory unit 103, an input unit 104, an output unit 105, and a communication unit 106. The main memory unit 102, the auxiliary memory unit 103, the input unit 104, the output unit 105, and the communication unit 106 are all connected to the processor 101 via an internal bus 107.

[0020] The processor 101 includes a CPU (Central Processing Unit) as a processing circuit. The processor 101 realizes various functions and executes the processes described below by executing a program P1 stored in the auxiliary storage unit 103. The program P1 corresponds to an example of a display control program.

[0021] The main memory unit 102 includes a RAM (Random Access Memory). A program P1 is loaded into the main memory unit 102 from the auxiliary memory unit 103. The main memory unit 102 is used as a working area for the processor 101.

[0022] The auxiliary storage unit 103 includes a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) and an HDD (Hard Disk Drive). In addition to the program P1, the auxiliary storage unit 103 stores various data used in the processing of the processor 101. The auxiliary storage unit 103 supplies the processor 101 with data used by the processor 101 in accordance with instructions from the processor 101. The auxiliary storage unit 103 also stores data supplied from the processor 101.

[0023] The input unit 104 includes input devices such as hardware switches, input keys, a keyboard, and a pointing device. The input unit 104 acquires information input by a user of the display control device 10 and notifies the processor 101 of the acquired information.

[0024] The output unit 105 includes output devices such as a light emitting diode (LED), a liquid crystal display (LCD), and a speaker. The output unit 105 presents various information to the user in accordance with instructions from the processor 101.

[0025] The communication unit 106 includes a communication interface circuit for communicating with an external device. The communication unit 106 receives a signal from the outside and outputs data indicated by this signal to the processor 101. The communication unit 106 also transmits a signal indicating the data output from the processor 101 to the external device. Note that while one communication unit 106 is representatively shown in FIG. 3, the display control device 10 may have multiple communication units 106. For example, the display control device 10 may have a communication unit 106 for communicating via the communication line 201 in FIG. 1 and a communication unit 106 for communicating via the communication line 301 in FIG. 1, separately.

[0026] The above-described hardware configurations work together to enable the display control device 10 to perform various functions. In detail, as shown in Fig. 4, the display control device 10 has, as its functions, an acquisition unit 11 that acquires control data 131 from the PLC 20 and stores it in the storage unit 13, acquires image data 132 from the cameras 31 and 32 and stores it in the storage unit 13, a reception unit 12 that receives parameters used for image processing from a user, the storage unit 13 that stores various information, an image processing unit 14 that performs image processing on the image data 132, a display control unit 15 that controls the display content of the display unit 16, and the display unit 16 that displays information to the user. Note that the arrows connecting the functional units in Fig. 4 merely indicate the main transmission paths of information, and information can also be transmitted via paths not shown in Fig. 4, as will be described later.

[0027] The acquisition unit 11 is realized mainly by cooperation between the processor 101 and the communication unit 106. The control data 131 acquired by the acquisition unit 11 is time-series data of device values ​​211 to which timestamps are attached, as exemplified in Fig. 5. In the example of Fig. 5, the timestamp, one or more device values ​​to be collected, and an imaging trigger indicating whether or not an I / O signal was transmitted when the device value was read from the memory 210 are associated with each other. In the figure, the imaging trigger value "1" indicates that an I / O signal was transmitted as an imaging trigger, and the imaging trigger value "0" indicates that an I / O signal was not transmitted.

[0028] Furthermore, the image data 132 acquired by the acquisition unit 11 is time-series data that associates captured images with frame counts, as exemplified in Fig. 5. In the example of Fig. 5, an identifier for identifying the camera that captured the image is shown in parentheses. For example, camera

[31] represents camera 31, whose identifier is "31." The identifier and the reference symbol in the figure are the same.

[0029] As described above, an image is captured at the timing when the device value associated with the imaging trigger of "1" is recorded, and therefore, this imaging trigger identifies the synchronized device value and image. In detail, the device value with the first imaging trigger "1" assigned in the control data 131 and the first image constituting the image data 132 are identified as corresponding in order. The acquisition unit 11 corresponds to an example of an acquisition means that acquires time-series control data that indicates, in association with time, control values ​​used to control the controlled device, and image data including images of control-related objects repeatedly captured at timings corresponding to the times associated with the control values.

[0030] Returning to FIG. 4 , the reception unit 12 is mainly realized by the input unit 104. The reception unit 12 receives from the user image processing parameters 133 used for image processing, conversion parameters 134 for converting pixel units into physical quantity units, and range parameters 135 indicating a range related to the position of the workpiece 40 as a physical quantity, and stores these received parameters in the storage unit 13. The reception unit 12 corresponds to an example of a first reception means for receiving conversion parameters for converting pixel units constituting an image into physical quantity units. Note that, if conversion to physical quantity units is not performed as described below, the reception unit 12 may omit receiving the conversion parameters 134 and the range parameters 135. Here, conversion from pixel units to physical quantity units means converting the relationship between pixels into the physical quantity of an object. In other words, it means converting the relationship between two pixels showing parts of an object into the value of the physical quantity related to each part. Specific examples of the conversion parameters and conversion will be described later.

[0031] The storage unit 13 is mainly realized by at least one of the main storage unit 102 and the auxiliary storage unit 103.

[0032] The image processing unit 14 is mainly realized by the processor 101. The image processing unit 14 performs image processing, such as pattern matching, on the image included in the image data 132. The image processing unit 14 performs image processing based on the image processing parameters 133 to obtain a result value related to the position of the workpiece 40 shown in the image. This result value is, for example, at least one value of the position, velocity, acceleration, attitude, angular velocity, angular acceleration, and area of ​​the workpiece 40 itself or a part of the workpiece 40 estimated from the image, or a value indicating the shape. The value indicating the shape is, for example, the rate of match with a pre-registered 3D model shape or 2D silhouette shape.

[0033] The image processing unit 14 may obtain a result value after estimating the position of the workpiece 40, or may obtain a result value without estimating the position of the workpiece 40. For example, the image processing unit 14 may directly estimate the velocity or acceleration of the workpiece 40 as a result value without identifying the position of the workpiece 40. These velocities and accelerations are derived from the first and second derivatives of the position of the workpiece 40, and therefore can be considered result values ​​related to the position of the workpiece 40. Note that the orientation and shape of the workpiece 40 depend on the positions of each part of the workpiece 40, and therefore can be considered result values ​​related to the positions of these parts. Furthermore, when the workpiece 40 having moving parts or a powder or fluid is the object of the processing process, the area of ​​the object is determined by the positions of each part of these objects. Therefore, the area can be considered a result value related to the position of the part of the object.

[0034] Furthermore, the image processing unit 14 may obtain a result value in pixel units, or may obtain a result value in physical quantity units by converting pixel units to physical quantity units using the conversion parameters 134. That is, the result value may be expressed in pixel units or in physical units. A physical quantity refers to an attribute, state, or property that an object actually possesses, which can be measured by a method other than an optical method, regardless of the presence or absence of a camera. The unit of the physical quantity is, for example, a unit of the International System of Units using meters and seconds, or a unit according to the imperial system.

[0035] The following mainly describes an example in which image processing is performed using the transformation parameters 134 to obtain a result value of a physical quantity. Details of the image processing will be described later. The image processing unit 14 corresponds to an example of an image processing means that processes an image to obtain a result value related to the position of an object.

[0036] Furthermore, the image processing unit 14 corrects the transition of the result value based on the range parameter 135. For example, if the movement speed of the workpiece 40 estimated by the image processing unit 14 through image processing exceeds the upper limit value indicated by the range parameter 135, the image processing unit 14 corrects the transition of the result value indicating the position or speed so that it does not exceed the upper limit value.

[0037] The display control unit 15 is mainly realized by the processor 101. The display control unit 15 acquires the transition of the result value from the image processing unit 14, and reads out the control data 131 from the storage unit 13. Then, the display control unit 15 causes the display unit 16 to display the transition of the result value and the transition of the device value in a form in which these transitions correspond to each other. In detail, the display control unit 15 causes the display unit 16 to draw graphs showing each of these transitions by superimposing them in an area defined by a common time axis. Details of the drawing by the display control unit 15 will be described later. The display control unit 15 corresponds to an example of a display control means that causes the display device to display the transition of the control value indicated by the control data and the transition of the result value in a form in which they correspond to each other based on a common reference time.

[0038] The display unit 16 is mainly realized by the output unit 105. The display unit 16 displays information to the user in accordance with instructions from the display control unit 15. The display unit 16 corresponds to an example of a display device.

[0039] Next, the display control process executed by the display control device 10 will be described in detail with reference to Figures 6 to 19. The display control process in Figure 6 starts when a user performs a specific operation on application software of the display control device 10. This application software may be the above-mentioned engineering tool.

[0040] 6, in the display control process, the acquisition unit 11 acquires control data 131 and image data 132 (step S1) and stores these data in the storage unit 13. Next, the reception unit 12 receives image processing settings from the user and acquires image processing parameters 133 (step S2).

[0041] 7 shows an example of a screen for setting image processing. When a camera is selected from list 501 on the screen of FIG. 7 and "Capture" button 502 is pressed, reception unit 12, in cooperation with acquisition unit 11, causes the selected camera to capture a new image and displays the captured image in real time in image display area 520. When "Read image file" button 503 is pressed, reception unit 12 reads the image specified by the user from storage unit 13 and displays the read image in image display area 520.

[0042] The receiving unit 12 also receives a selection of any filter from the list 51 as a designation of pre-processing for image processing. This list 51 includes "Gaussian", "Median", "Average", "Maximum", and "Minimum".

[0043] The reception unit 12 also receives the designation of an image processing mode selected from the list 52. This image processing mode is, for example, edge detection, circle detection, line detection, corner detection, color detection, normalized correlation, pattern matching, blob detection, feature matching (matching method, optical flow method), or a calculation mode described below. According to the selected image processing mode, the reception unit 12 displays an input field 54 for parameters to be handled in that image processing mode.

[0044] The reception unit 12 also displays the results of trial image processing using the temporarily specified image and parameters in area 511. In particular, when updating the display content in the image display area 520 and when changing the values ​​of the parameters entered in the input field 54, the reception unit 12 causes the image processing unit 14 to trial image processing and displays the trial results in area 511. As a result, the position of the workpiece 40 or a part of the workpiece 40, or the results of intermediate processing such as filter processing, edge point recognition, and straight line conversion processing, are displayed enlarged as figures or polygons in area 511. When the image processing mode is pattern matching, the reception unit 12 displays an image of the model registered from the input field 54 in area 512 adjacent to area 511.

[0045] When the "Automatic Parameter Adjustment" button 53 is pressed, the reception unit 12 adjusts the parameters by having the image processing unit 14 try out image processing on the specified image while changing the parameter values ​​and searching for appropriate values. The parameters to be adjusted are parameters for which a numerical value must be specified, other than parameters indicating a range. In image processing modes where a matching rate can be calculated, such as pattern matching and detection of shapes such as rectangles or circles, or color detection, the parameters are adjusted to values ​​that maximize the matching rate. Furthermore, if the trial image processing results in an estimated value of the position of the workpiece 40 or a part thereof, the parameters are adjusted to values ​​that meet the following conditions, prioritized in order: "detected," "low overdetection," "small change in the estimated value when each parameter value is changed to a nearby value," and "fast processing time." The nearby value of a parameter value is a value where the difference from the parameter value, or the ratio of the difference to the parameter value, is a predetermined value.

[0046] Furthermore, the reception unit 12 displays the trial results of the image processing in a result column 55. In detail, the reception unit 12 displays a numerical value or character string indicating the result of the image processing and the length of time required for the image processing, and if a matching rate is calculated, the reception unit 12 also displays the calculated matching rate.

[0047] FIG. 8 shows an example of a screen displayed when the "Live Image Processing" button 504 is pressed. At this time, images are repeatedly captured by the selected camera in real time, and image processing trials are continuously performed on each captured image. The results of the image processing trials are displayed in the graph display area 530. In the example of FIG. 8, the transitions of the X and Y coordinates in the image of the pattern detected by pattern matching are displayed as graphs. Furthermore, when the "Display Trajectory" check box 521 is checked, the reception unit 12 draws the trajectory of the detected pattern in the image display area 520, as indicated by a thick line.

[0048] 9 shows an example of a screen that appears when the "Collected Data Image Processing" button 506 is pressed and one of the multiple image data 132 is selected from the list 505. At this time, image processing is attempted all at once on the image data 132 that represent time-series images, and the results are displayed in the graph display area 530. When the check box 521 is checked, the trajectory of the detected detection target is drawn in the image display area 520, similar to the example in FIG. 8.

[0049] 5 shows that the single image data 132 includes time-series images taken by the multiple cameras 31 and 32, but the image data 132 may also be time-series images taken by a single camera. Furthermore, time-series images taken by the same camera in different time periods may each be treated as different image data 132. The selection of image data 132 using the list 505 in FIG. 9 corresponds to the selection of image data 132 taken by a single camera in a specific time period.

[0050] Furthermore, when the knob of bar 522 is operated, reception unit 12 displays an image associated with the time corresponding to the position of the knob in image display area 520, and moves line 531 in graph display area 530 to correspond to the position of the knob. Similarly, when the position of line 531 is operated along the horizontal axis of the graph, i.e., the time axis, reception unit 12 updates the image in image display area 520 to correspond to the position of line 531, and changes the position of the knob of bar 522.

[0051] Furthermore, when the check box indicating whether or not to execute is checked in the input field 513 for "search area tracking" and processed data indicating the image processing results is selected from the list as the tracking target, the reception unit 12 sets a search area for searching for the detection target based on the partial area in which the detection target is detected in the image of the selected processed data. Search area tracking is used to reduce the calculation load, for example, in pattern matching when a pattern to be detected from image data 132 is expected to appear in approximately the same position as other image data 132.

[0052] The target of the search area may not be processed data showing the results of image processing that has already been executed, but may be scheduled data for which image processing is scheduled to be executed. When scheduled data is selected as the target of the search area, image processing related to the scheduled data is executed preferentially, and then a search area based on the results of the image processing is set and image processing is executed on the image data 132 selected from the list 505. In addition, to avoid circular definition of the target of the search area, the image data 132 selected from the list 505 cannot be specified as the target of the search area for image processing related to the scheduled data.

[0053] 9, data indicating the results of image processing using the input parameters can become new processed data or scheduled data. The names of this processed data and scheduled data are specified in input field 50. Furthermore, when image data 132 is selected from list 505 and then the "Save" button 56 is pressed, processed data or scheduled data based on the image data 132 is registered, and this registered data may be subject to graph display, which will be described later.

[0054] In addition to the processed data and scheduled data, "previous image processing result" or "previous image difference" can be selected as the tracking target. When "previous image processing result" is selected, the search area of ​​the current image is set with the image processing result of the immediately preceding image among the series of images included in the image data 132 selected from the list 505 as the tracking target. When "previous image difference" is selected, the immediately preceding image and the current image are compared, and an area where a difference exceeding a threshold occurs is set as the search area. When there are multiple areas where differences occur, multiple search areas including each area are set and image processing is performed.

[0055] When tracking the search area, if the object to be detected is not found or detection fails due to factors such as deformation of the workpiece 40, the search area may be reset to the entire image and image processing may be performed again.

[0056] FIG. 10 shows an example of a screen when the "Calculation" mode is selected as the image processing mode from list 52. In this case, the processing result of the "Calculation" mode is the result of performing the calculation indicated by the calculation formula entered in input field 58 on the image processing results contained in the processed data displayed in list 57. In the figure, a, c, and e represent the X coordinate values ​​obtained as a result of image processing for each piece of processed data, and b, d, and f represent the Y coordinate values. By specifying the calculation formula appropriately, the distance between patterns, center point, center of gravity, or tilt can be easily obtained. Calculation formulas are added or deleted using the "Add" and "Delete" buttons in input field 58.

[0057] When the "Execute image processing" button is pressed after an arithmetic expression is entered in the input field 58, the results calculated using the entered arithmetic expression are displayed directly below the input field. Also, when the "Live image processing" button 504 or the "Acquired data image processing" button 506 is pressed, the progress of the calculation results using the entered arithmetic expression is displayed in the graph display area 530.

[0058] 7 to 10, image processing is trialed as needed, and settings of image processing parameters 133 are accepted, and a combination of the accepted image processing parameters 133 and image data 132 to be image processed using the image processing parameters 133 is registered. Hereinafter, the combination of the image processing parameters 133 and image data 132 may be referred to as image processing data. The image processing data may be the processed data or planned data described above.

[0059] Returning to FIG. 6, following step S2, the accepting unit 12 accepts the settings for graph display and acquires the conversion parameters 134 and range parameters 135 (step S3).

[0060] FIG. 11 shows an example of a screen displayed when accepting graph display settings. On this screen, settings are entered for displaying two graphs superimposed on one or more areas. Specifically, settings are entered for "Graph No. 1" and "Graph No. 2" to be superimposed and displayed in "Graph Area A1" in FIG. 11, and settings are entered for "Graph No. 1" and "Graph No. 2" to be superimposed and displayed in "Graph Area A2" in FIG. 11. For each graph, image processing results or control data 131 are selected from list 611. The image processing results to be selected are the processed data or planned data registered in FIGS. 7 to 10. For example, "Image Processing Setting B1(X)" in FIG. 11 indicates the transition of the X-coordinate value in the image output when image processing is performed using the parameter values ​​registered under the name "Image Processing Setting B1." Furthermore, "Axis Ax.1-Current Value" and "Axis Ax.2-Current Value" in FIG. 11 indicate the transition of the device values ​​registered under these names. When comparing the control data 131 and the image data 132, typically, one of the image processing data is selected from the list 611 as one of the graphs to be superimposed and displayed, and one of the device value trends is selected from the list 611 as the other graph.

[0061] In addition, the type of result value to be displayed is selected from list 612. For example, if any image processing result is selected from list 611 and then "velocity" is selected from list 612, the velocity obtained by differentiating the position as a result of the image processing is displayed on a graph. From list 613, the unit of the value of the graph to be displayed is selected. For example, if "position" is selected from list 612 and then "mm" is selected from list 613, the position indicated by the number of pixels as a result of the image processing is converted to the position of the workpiece 40 in millimeters and displayed as a graph.

[0062] Furthermore, when the "noise removal" check box 601 is checked for each graph, the noise removal process described below is executed. Furthermore, when the "inter-graph fitting" check box 602 is checked for each graph area, the fitting process described below is executed for the two graphs displayed in that area.

[0063] When the "Data Settings" button 64 on the screen of FIG. 11 is pressed, the reception unit 12 transitions the screen to the setting screen shown in FIG. 12. In the upper part of the setting screen of FIG. 12, the image processing data and control data 131, which indicate the image processing settings, are checked or unchecked to determine whether they should be included as a selection candidate in the list 611 of FIG. 11. In addition, in the lower part of the setting screen of FIG. 12, the resolution of each camera 31, 32 is set. This resolution represents the ratio between the number of pixels between two pixels constituting a captured image and the distance between the parts of the workpiece 40 corresponding to each pixel. This resolution is used when a physical unit other than the number of pixels is selected from the list 613 of FIG. 11. Specifically, when "None" is selected from the list 613 of FIG. 11, a graph in pixel units is displayed. When "cm," "mm," or "um" is selected, a graph in physical units converted from the number of pixels using the set resolution is displayed. The following describes an example in which "cm," "mm," or "um" is selected and conversion from the number of pixels to physical units is performed. The resolution corresponds to an example of a conversion parameter.

[0064] 12 corresponds to an example of a range parameter, and the value of this range parameter is received by the receiving unit 12.

[0065] Returning to Fig. 11, when the "Save setting data" button 62 is pressed, the settings made in steps S1 to S3 are saved together with an identification name. When the "Read setting data" button 63 is pressed, the setting contents saved with the specified identification name are read by specifying the identification name. When the image processing settings and graph display settings are complete, the user presses the "Start data display" button 61.

[0066] Returning to FIG. 6, following step S3, the image processing unit 14 executes image processing according to the settings (step S4), and converts the values ​​obtained as a result of the image processing into physical quantities, as necessary. For example, as illustrated in FIG. 13, the image processing unit 14 sequentially detects detection targets by pattern matching and converts the pixel-unit positions of the detected patterns into physical unit positions using the set resolution. In the example of FIG. 13, for the X and Y coordinate values ​​corresponding to each timestamp, the pixel-unit positions indicated by "px" are converted into micrometer-unit positions indicated by "um." Note that if the results of the image processing performed during the information setting of steps S1 and S2 can be reused, the image processing in step S3 may be omitted.

[0067] Next, the image processing unit 14 executes differentiation and noise removal in accordance with the settings (step S5). Note that in Fig. 11, if a type that does not require differentiation is selected from the list 612 for all graphs, differentiation is omitted. Also, in Fig. 11, if it is specified in the check box 601 that noise removal is not to be executed for all graphs, noise removal is omitted.

[0068] Differentiation is a process of calculating velocity or acceleration from a time series of positions. The central difference method is used to derive velocity. Specifically, the velocity at a specific time is determined by differentiating the values ​​immediately before and after the value at that specific time in the time series data. For example, if the position at time t is x(t), the velocity at time t is calculated from x(t-1) and x(t+1).

[0069] To convert to acceleration, the above-mentioned differentiation operation is performed twice on the value at a specific time in the time series data and the values ​​immediately before and after that time. For example, the acceleration at time t is calculated by further differentiating the velocity obtained by differentiating x(t) and x(t-1) and the velocity obtained by differentiating x(t) and x(t+1). In the differentiation process, to avoid any time lag before and after the process, the differential value at the specific time is calculated from the values ​​immediately before and after the specific time, as described above.

[0070] In the noise removal process, noise components are removed by low-pass filtering, as shown in FIG. 14 . While methods such as the moving average method, frequency space cutoff, and Gaussian convolution can be used as noise removal techniques, the moving average method is employed, in which the noise-removed value at a specific time is calculated by averaging N values ​​around the specific time. For example, when N is 3, the moving average of x(t−1), x(t), and x(t+1) becomes the noise-removed value at time t. The value of N may be set by the user or may be specified in advance. In the noise removal process, as in the differentiation process described above, the processed value at a specific time is calculated from the values ​​around the specific time to avoid a time lag before and after the process.

[0071] Returning to FIG. 6, following step S5, the display control unit 15 performs inter-graph fitting (step S6). Note that if execution of inter-graph fitting is not specified in FIG. 11, step S6 is omitted. In inter-graph fitting, as shown in detail in FIG. 15, it is set that the values ​​should be displayed in the same area, and the display control unit 15 sets two analysis points for the transitions of each of the two values ​​obtained in steps S1 to S5. The analysis points are arranged at predetermined regular intervals, but the time corresponding to the analysis points may also be specified by the user.

[0072] Next, each graph showing the value transition is converted into a linear graph that takes the value at the analysis point. Then, using the graph with the smallest slope as the reference, the slope of the reference linear graph is changed to adjust the scale so that the distances d1 and d2 between the graphs at the analysis point match. Furthermore, the offset value of the reference graph is adjusted to maximize overlap between the scaled graphs. The least squares method can be used to overlap two graphs. Figure 16 shows an example of evaluating the distance between two graphs using the least squares method. The reference graph is moved up and down, and the sum of the squares of the values ​​at each time is taken as the distance between the graphs, and the offset value that minimizes this distance is calculated. Adjusting the scale and offset value of the graphs in this way makes it easier for users to match two graphs.

[0073] 6, following step S6, the display control unit 15 causes the display unit 16 to display the image and graph (step S7). Specifically, a screen such as that shown in FIG. 17 is displayed on the display unit 16. On this screen, an image included in the image data 132 is displayed in the image display area 71, and a dashed frame indicating the detection result is displayed superimposed on the image for the detection target detected from this image.

[0074] In addition, the graph display area 72 displays graphs showing the transitions of result values ​​obtained through image processing, conversion to physical quantities, differentiation, and noise removal processing, and graphs showing the transitions of device values, superimposed on each other using a common time axis. The graphs displayed in the graph display area 72 can be enlarged or reduced horizontally or vertically by mouse and keyboard operations. Furthermore, pressing the "Add Graph" button 701 adds a new graph area to the graph display area 72, and pressing the "Delete" button 702 deletes the corresponding graph area. This allows further editing of the displayed graphs while they are still displayed. Furthermore, whether or not a graph is displayed is determined by whether or not a check box 721 adjacent to the name of the graph is checked.

[0075] The accepting unit 12 corresponds to an example of a fourth accepting means that accepts designation of control data and image data for creating a graph from the plurality of control data and the plurality of image data, and also accepts designation of whether or not to display the graph. The image processing unit 14 corresponds to an example of an image processing means that obtains a result value from the designated image data accepted by the fourth accepting means, and the display control unit 15 corresponds to an example of a display control means that switches whether or not to display the graph in accordance with the designation accepted by the fourth accepting means.

[0076] 9, the knob of the bar 711 and the line 703 in the graph area are linked, and an image at a time corresponding to the knob or the line 703 is displayed in the image display area 71. The image in the image display area 71 can be played back, played back in reverse, or paused as a moving image by operating the button 712. Furthermore, the image in the image display area 71 can be advanced or rewound frame by operating the button 713.

[0077] A flag 714 is set on the bar 711. The flag 714 is attached at the time when the difference between two transitions whose graphs are displayed in the same graph area becomes larger than a predetermined threshold. When the flag 714 is selected by hovering the cursor over it, an image of that time is displayed in the image display area 71, and the line 703 in the graph display area 72 also moves in conjunction with it to a position corresponding to that time. The flag 714 is set by pressing a "Set Flag" button 715.

[0078] When button 715 is pressed, the screen shown in FIG. 18 is displayed. On this setting screen, information is set in two setting modes: "Command Value Interval" and "Value Interval." The "Command Value Interval" mode is a mode for setting a threshold value for attaching a flag 714 to the time of the command value when the time difference between the rise or fall of the device value (command value) and the result value obtained by image processing, i.e., the distance along the horizontal axis of the graph, is greater than the threshold value. If the transition of the device value (command value) is not handled, information setting in the "Command Value Interval" mode is not performed. On the other hand, the "Value Interval" mode is a mode for setting a threshold value for attaching a flag 714 to the time when the difference in value between two transitions, i.e., the distance along the vertical axis between the graphs, is greater than the threshold value.

[0079] 18, the threshold expressed in "%" indicates the percentage or more of the overall average difference that should be detected. The "search range" is the search range for checking the difference, and refers to the distance in the time direction in the "command value interval" mode and the distance in the value magnitude direction in the "value interval" mode. If no object to be detected is found within this search range, flag 714 is not assigned and the object is not included in the calculation of the average difference.

[0080] 17 shows an example in which there is one image data 132 and one image is played back in the image display area 71, but this is not limiting. If multiple image data 132 are set to be the subject of image processing, multiple images may be displayed in the image display area 71 as shown in FIG.

[0081] As described above, the display control device 10 obtains the transition of result values ​​related to the position of the workpiece 40 from photographed images of the workpiece 40, and displays the transition of result values ​​and the transition of device values ​​in a form that corresponds to a common reference time. Here, correspondence based on a common reference time means that the reference point and time scale in the time direction match, and that the times at which values ​​were generated at any point during the overlapping period of the two transitions match. This makes it easy to compare the state of an object with its control history. Therefore, the efficiency of the matching work using photographed images of processing steps performed at an FA site can be improved.

[0082] The conversion parameters 134 indicate the relationship between the number of pixels from one pixel to another pixel that constitutes an image and the distance from a portion of the workpiece 40 corresponding to one pixel to another portion corresponding to another pixel when the workpiece 40 is captured in the image. The result value relating to the position of the workpiece 40 includes at least one value of the position, velocity, acceleration, attitude, angular velocity, angular acceleration, and area of ​​the workpiece 40, or a value indicating the shape. This allows the user to directly determine whether the processing status of the workpiece 40 has progressed as intended.

[0083] The control data 131 may also be converted into a value related to the position as a physical quantity. For example, if the control data 131 is set to be expressed in units of travel distance from the list 613 in Fig. 11, the travel distance of the workpiece 40 when the servo motor 22 makes one rotation and the value of "pulse / rev" indicating the number of pulses per rotation, which are set as shown in the lower part of Fig. 12, may be used to obtain a transition in the result value related to the position of the arm 23 from the device value indicating the number of pulses, and this transition may be displayed.

[0084] Furthermore, the display control unit 15 displays the two trends as graphs superimposed on each other in a graph area defined by a common time axis. This allows the user to visually grasp the trends and easily perform matching work. The graph showing the trends in the device values ​​corresponds to an example of a first graph, and the graph showing the trends in the result values ​​obtained by image processing corresponds to an example of a second graph.

[0085] Furthermore, the display control unit 15 compares the transition of the device value with the transition of the result value obtained by image processing, and adjusts at least one of the scale and offset value for one of the two graphs before displaying them. This allows the graphs to be displayed as having similar shapes, making it easier to visually compare the graphs. Note that the adjustment of the scale and offset value may be performed for both graphs.

[0086] Furthermore, the receiving unit 12 receives range parameters 135 indicating the range of the physical position of the workpiece 40, the image processing unit 14 corrects the results of the image processing based on the range parameters 135, and the display control unit 15 displays a graph obtained by correcting the transition of the result values ​​based on the range parameters 135. As a result, even if unrealistic movement of the workpiece 40 is obtained as a result of image processing due to factors such as noise contained in the image, a graph that keeps the movement within a realistic range is displayed. The receiving unit 12 corresponds to an example of a third receiving means that receives range parameters.

[0087] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments.

[0088] For example, as shown in Fig. 20, when multiple patterns are detected from an image and the number of graphs increases, it can be cumbersome to determine which pattern each graph corresponds to. Therefore, the display control unit 15 may indicate the correspondence between the patterns and the graphs by lines 81 as shown in Fig. 20.

[0089] 20, the receiving unit 12 corresponds to an example of a second receiving means that receives a designation of a designated time by using the knob on the line 703 or the bar 711. The display control unit 15 corresponds to an example of a display control means that displays an image associated with the designated time together with the first graph and the second graph, and when a result value is obtained from a partial area corresponding to a pattern included in the displayed image, highlights and displays the partial area and associates the partial area with the second graph.

[0090] Also, although an example has been described in which a value in pixel units is converted into a result value that represents the position of the workpiece 40 in physical units based on the conversion parameters, the present invention is not limited to this. For example, even if the conversion parameters 134 are not given, if the 3D model shape of the workpiece 40 is given in advance, it is possible to obtain physical quantities of the workpiece 40, such as velocity or acceleration, from the image by fitting the projected image of the workpiece 40 shown in the image to the 3D model shape. Furthermore, even if there is no prior information, if one or both of the cameras 31 and 32 are depth cameras, it is possible to obtain physical quantities directly from the image.

[0091] Furthermore, although an example has been described in which timestamps are assigned to device values ​​in the control data 131 and the logging of device values ​​and the capture of images are synchronized by an I / O signal, the present invention is not limited to this. For example, as shown in Fig. 21, if a timestamp is also assigned to an image in the image data 132, it is possible to identify synchronized device values ​​and images by comparing the timestamps in the control data 131 and the image data 132. Furthermore, as shown in Fig. 22, if the frame count "1" in the image data 132 corresponds to the timestamp "1002034" in the control data 131 and synchronization information indicating a 4 ms capture interval is provided, it is possible to identify synchronized device values ​​and images based on this synchronization information.

[0092] Although the example has been described in which two graphs showing the transitions of device values ​​and the transitions of image processing result values ​​are displayed in correspondence with each other, the present invention is not limited to this. For example, synchronized device values ​​and image result values ​​may be displayed in the same column in a table format as shown in Figures 5, 21, and 22. Furthermore, when the results of processing one image are expressed as a spectrum, the time series may be displayed as a spectrogram in time-frequency representation.

[0093] Furthermore, although an example in which the control data 131 is provided from the PLC 20 has been described, this is not limiting and the control data may be provided from an industrial PC, a servo amplifier, a machine tool, or other FA device other than the display control device 10.

[0094] Furthermore, the display control device 10 may be configured without the display unit 16. As shown in Fig. 23, the display control unit 15 may cause an external display device 16a to display information.

[0095] Furthermore, it may be possible to control controlled equipment other than the servo amplifier 21 and the servo motor 22. Examples of such controlled equipment include factory automation equipment such as machine tools, actuators, and robots.

[0096] Furthermore, the communication line that enables communication between the devices may be a network such as a LAN or a wide area network.

[0097] Furthermore, although the above description has focused on an example in which the object to be photographed is the workpiece 40 as a control target, the present invention is not limited to this. For example, fluids with irregular shapes, such as grease in a grease production line or sauce in a food production line, or powders and granular materials, may be objects to be processed in a processing step and photographed at the same time.

[0098] Furthermore, the object to be photographed does not have to be the processing target. For example, the object may be the controlled device itself, a support table for a workpiece, or a member of a belt conveyor.

[0099] The functions of the display control device 10 according to the above-described embodiment can be realized by dedicated hardware or by an ordinary computer system.

[0100] For example, program P1 can be stored and distributed on a computer-readable recording medium such as a flexible disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), or an MO (Magneto-Optical disk), and by installing program P1 on a computer, a device that executes the above-mentioned processing can be configured.

[0101] Furthermore, the program P1 may be stored in a disk device of a server device on a communication network such as the Internet, and may be downloaded to a computer by superimposing it on a carrier wave, for example.

[0102] The above process can also be achieved by starting and executing the program P1 while transferring it via a network such as the Internet.

[0103] Furthermore, the above-described processing can also be achieved by executing all or part of program P1 on a server device, and executing program P1 while the computer sends and receives information about the processing via a communications network.

[0104] In addition, if the above functions are realized by an operating system (OS) or by a collaboration between the OS and an application, only the parts other than the OS may be stored on a medium and distributed, or may be downloaded to a computer.

[0105] Furthermore, the means for realizing the functions of the display control device 10 is not limited to software, and some or all of the functions may be realized by dedicated hardware or circuits.

[0106] The present disclosure allows various embodiments and modifications without departing from the broad spirit and scope of the present disclosure. Furthermore, the above-described embodiments are intended to explain the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of equivalent disclosures are considered to be within the scope of the present disclosure. [Industrial Applicability]

[0107] The present disclosure is suitable for improving the efficiency of maintenance work at FA sites. [Explanation of symbols]

[0108] 10 display control device, 11 acquisition unit, 12 reception unit, 13 memory unit, 14 image processing unit, 15 display control unit, 16 display unit, 16a display device, 20 PLC, 21 servo amplifier, 22 servo motor, 23 arm, 31, 32 camera, 40, 40a work, 41, 41a mark, 50, 54, 58, 513 input field, 51, 52, 57, 501, 505, 611 to 613 list, 53, 56, 61 to 64, 502 to 504, 506, 701, 702, 712, 713, 715 button, 55 result field, 71, 520 image display area, 72, 530 graph display area, 81, 531, 703 line, 101 processor, 102 main memory unit, 103 Auxiliary memory unit, 104 input unit, 105 output unit, 106 communication unit, 107 internal bus, 131 control data, 132 image data, 133 image processing parameters, 134 conversion parameters, 135 range parameters, 201, 202, 301 communication line, 210 memory, 211 device value, 511, 512 area, 521, 601, 602, 721 check column, 522, 711 bar, 714 flag, 1000 support system, P1 program.

Claims

1. an acquisition means for acquiring time-series control data indicating control values ​​used to control a controlled device in association with time, and image data including images of an object related to the control repeatedly captured at timings corresponding to the times associated with the control values; image processing means for processing said images to obtain a result value relating to the position of said object; a display control means for displaying on a display device a transition of the control value indicated by the control data and a transition of the result value in a manner that corresponds to each other based on a common reference time; A display control device comprising:

2. a first receiving means for receiving a conversion parameter for converting a unit of pixels constituting the image into a unit of a physical quantity; the image processing means obtains the resultant value by processing the image based on the transformation parameters; The display control device according to claim 1 .

3. the transformation parameter indicates a relationship between the number of pixels from one pixel to another pixel that constitutes the image and a distance from a part of the object that corresponds to the one pixel to another part that corresponds to the other pixel when the object is captured in the image, the resultant value includes at least one value of a position, a velocity, an acceleration, an attitude, an angular velocity, an angular acceleration, and an area of ​​the object, or a value indicating a shape of the object; The display control device according to claim 2 .

4. the display control means displays a first graph showing a transition of the control value and a second graph showing a transition of the result value in a graph area defined by a common time axis, the first graph and the second graph being superimposed on each other; The display control device according to claim 1 .

5. the display control means compares the transition of the control value with the transition of the result value, and adjusts at least one of a scale and an offset value for at least one of the first graph and the second graph, and displays the graph. The display control device according to claim 4 .

6. The system further includes a second reception means for receiving a designation of the designated time, The display control means displaying the image associated with the specified time together with the first graph and the second graph on the display device; When the result value is obtained from a partial region included in the displayed image, the partial region is highlighted and displayed in association with the second graph. The display control device according to claim 4 .

7. a third receiving means for receiving a range parameter indicating a range related to a position of the object as a physical quantity, the display control means displays the second graph obtained by correcting the transition of the result value based on the range parameter. The display control device according to claim 4 .

8. a fourth receiving means for receiving designation of the control data and the image data for which a graph is to be created from the plurality of control data and the plurality of image data, and for receiving designation of whether or not to display the graph; the image processing means obtains the result value from the specified image data accepted by the fourth accepting means, the display control means switches between displaying and not displaying the graph in accordance with the specification accepted by the fourth accepting means. The display control device according to claim 4 .

9. the image processing means obtains the transition of the resultant values ​​by performing low-pass filtering on a series of values ​​obtained by processing the image; The display control device according to claim 1 .

10. an acquiring means acquiring time-series control data indicating control values ​​used to control the controlled device in association with time, and image data including images of an object related to the control, the image data being repeatedly captured; image processing means for processing the image to obtain a result value relating to the position of the object; a display control means for controlling a display device to display the transition of the control value indicated by the control data and the transition of the result value in a manner that corresponds to each other; A display control method comprising:

11. On the computer, Acquire time-series control data indicating control values ​​used to control the controlled device in association with time, and image data including images of an object related to the control, which are repeatedly photographed; processing the image to obtain a result value related to the position of the object; displaying, on a display device, a transition of the control value indicated by the control data and a transition of the result value in a form in which they correspond to each other; A program to make it happen.

Images