Image-related information determination device and computer-readable storage medium

Abstract

Provided is a video relating information determination device which generates recorded video of an industrial machine, and which: acquires data indicating a machine state of the industrial machine; acquires video indicating an action of the industrial machine; searches data and detects appearance of a trigger event; searches past data and determines whether or not an event set has appeared in the past, the event set being a combination of events corresponding to the trigger event; determines relating information for video corresponding to the event set; and links the video and the relating information to each other on the basis of an appearance time of the event.

Description

Technical Field

[0001] The present invention relates to a video-related information determination device and a computer-readable storage medium.

Background Art

[0002] Currently, there are technologies for monitoring the state of industrial machines and utilizing them for improving control efficiency and machine maintenance. In such technologies, data related to industrial machines is collected and analyzed to understand the state of industrial machines, prevent machine failures, confirm product quality, and utilize it for improving production management.

[0003] Industrial machines and the machines detected by sensors provided around them are recorded in a data collection device called a data logger. A large number of data loggers are provided in the factory to record the state of the machines in the factory. In addition, a PLC (Programmable Logic Controller) that controls the operation sequence of a machine and a numerical control device (Numerical Control) that controls a machine tool also have a recording unit inside and store control information and state information.

[0004] The collected data is converted into a form that is easy for an operator to visually recognize. Patent Document 1 discloses a "machine system including a machine body for processing a workpiece, a camera for video-recording the processing process of the workpiece by the machine body, an information processing device having a memory and storing the moving image data obtained by the video-recording by the camera in the memory, and the information processing device protecting the moving image data for a predetermined set time including the time when the abnormality occurs when an abnormality occurs in the processing process."

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] Patent Document 1 describes a system that, when an anomaly (error) occurs, protects video data for a predetermined period of time, including the time the anomaly occurred. Having logs and images from the time of the error helps to clarify the circumstances surrounding the error and aids in investigating its cause. Similarly, creating highly visible video footage can be used for the maintenance of industrial machinery and improving productivity.

[0007] In the field of data management for industrial machinery, there is a challenge in improving visibility. [Means for solving the problem]

[0008] A video-related information determination device, which is one aspect of this disclosure, A video acquisition unit that acquires video showing the operation of industrial machinery, and the aforementioned A data acquisition unit that acquires data indicating the machine status of industrial machinery, and a data history storage unit that stores data of industrial machinery, An event set is a combination of multiple events that occur in a time order, including at least one trigger event, and an associated information storage unit stores associated information corresponding to the event set. The system includes a related information determination unit that searches data, detects the occurrence of trigger events, searches past data stored in a data history storage unit, determines whether an event set, which is a combination of events corresponding to a trigger event, has occurred in the past, and determines related information corresponding to the event set. A storage medium in one aspect of this disclosure is executed by one or more processors, We acquire video footage showing the operation of the industrial machine, and the above Obtain data indicating the status of industrial machinery. , the aforementioned Search for the data, The associated information storage unit includes an event set, which is a combination of multiple events that occur in a temporal order, including at least one trigger event, and associated information corresponding to the event set, and stores them together. Detect the occurrence of a trigger event, The data of the aforementioned industrial machine Search past data stored as history, and the event set, which is a combination of events corresponding to the trigger event, From the aforementioned trigger event Determine whether or not it has occurred in the past. If it is manifested, refer to the related information storage unit. Determine the relevant information corresponding to the event set. death , Based on the event occurrence times of the events included in the aforementioned event set, the video and the related information are linked. The processor stores instructions that it can read. [Effects of the Invention]

[0009] According to one aspect of the present invention, visibility can be improved.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing the application range of the video-related information determination device. [Figure 2] It is a block diagram of the video-related information determination device of the first disclosure. [Figure 3] It is a conceptual model of the related information determination unit. [Figure 4] It is a block diagram of the video-related information determination device of the second disclosure. [Figure 5] It is a diagram showing an example of a machining center. <00000�2>It is a diagram showing an example of a trigger event and an event set. [Figure 7] It is a diagram showing the determination process of the work unit. [[ID=2९]] [Figure 8] It is a diagram showing an example of a video generated by the video-related information determination device. [Figure 9] It is a diagram showing an example of a video generated by the video-related information determination device. [Figure 10] [ It is a diagram showing an example of a seek bar. <00oo081> [Figure 11] It is a diagram for explaining work measurement. [Figure 12] It is a diagram showing an example of a video generated by the video-related information determination device. [Figure 13] It is a diagram showing the hardware configuration of the video-related information determination device. [[ID=4७]]

Modes for Carrying Out the Invention

[0011] Hereinafter, the video-related information determination device 100 to which the present disclosure is applied will be described. The video-related information determination device 100 acquires a video showing the operation of an industrial machine arranged in a factory and data showing the state of the industrial machine. The video-related information determination device 100 presents information that is considered necessary for the user of the industrial machine by using the videos and data that have already been accumulated. Examples of the target industrial machines include machine tools, numerical control devices, robots, and the like.

[0012] The video generated by the video-related information determination device 100 is used as a recorded video of the industrial machine. FIG. 1 shows the application range of the video-related information determination device 100. The video-related information determination device 100 is applied to a PC used by a factory manager, a server on the cloud, a PC of a machine manufacturer or a control machine manufacturer, a mobile terminal used by a factory manager, and the like. In addition, the video-related information determination device 100 can be applied to information processing devices such as a numerical control device and a PLC (Programmable Logic Controller). The related information determined by the video-related information determination device 100 is utilized for the management (maintenance, production management, etc.) of the industrial machine.

[0013] [First Disclosure] FIG. 2 is a block diagram of the video-related information determination device 100 of the first disclosure. The video-related information determination device 100 includes a data acquisition unit 1, a related information determination unit 3, a data history storage unit 4, and an event set storage unit 5.

[0014] The data acquisition unit 1 acquires data related to the numerical control device and machine tools from data loggers, internal and external servers, PLCs, numerical control devices, etc. The data acquired by the data acquisition unit includes status information of the numerical control device, operation input information, and control information of the machine tool. The status information of the numerical control device is set in the internal variables of the numerical control device and includes the mode of the numerical control device, tool offset amount, alarms, time, number of machined parts, modal information position information (coordinate system), etc. Operation input information includes input from touch panels, physical buttons, dials, handles, etc. Control information of the machine tool includes power consumption, servo motor load, sensor values ​​(temperature and pressure), etc.

[0015] The data history storage unit 4 stores the history of the data acquired by the data acquisition unit 1. The related information determination unit 3 determines related information to support the video based on the data acquired by the data acquisition unit 1. Figure 3 is a conceptual model of the related information determination unit 3. The related information determination unit 3 searches the data acquired by the data acquisition unit 1 and detects trigger events. The event set storage unit 5 stores event sets corresponding to trigger events. The related information determination unit 3 reads the event set corresponding to the detected trigger event and searches past data stored in the data history storage unit 4 to check whether the event set corresponding to the detected trigger event has occurred in the past. If the event set corresponding to the detected trigger event has occurred in the past, the related information corresponding to the event set is determined to be displayed as related information to support the video.

[0016] [Second Disclosure] The second disclosed video-related information determination device 100 generates video by linking related information with video. Figure 4 is a block diagram of the second disclosed video-related information determination device 100. The video-related information determination device 100 comprises a data acquisition unit 1, a video acquisition unit 2, a related information determination unit 3, a data history storage unit 4, an event set storage unit 5, a related information storage unit 6, and a linking unit 7.

[0017] The video acquisition unit 2 acquires video of machinery and equipment in the factory from data loggers, internal and external servers, PLCs, numerical control devices, etc. In this disclosure, video of a machining center, in which a machine tool and a numerical control device are integrated, is acquired. Figure 5 shows the external appearance of the machining center. The machining center in this disclosure is equipped with two cameras. One camera photographs the workpiece mounted on the table, and the other camera photographs the control panel. The display screen of the control panel may be captured.

[0018] The video acquired by the video acquisition unit 2 has time information added to it, which can be merged with information useful to the operator. In this disclosure, "video" includes both moving images and still images. Images that display multiple still images in sequence are also included in the definition of video. Furthermore, while this disclosure displays the video feed of the machining center's interior and the control panel side by side, it is also acceptable to display only one of them.

[0019] The related information storage unit 6 stores related information corresponding to the event set. The linking unit 7 reads the related information from the related information storage unit 6 determined by the related information determination unit 3. Based on the event occurrence time, the linking unit 7 determines the time of the related information and links the video acquired by the video acquisition unit 2 with the related information by time.

[0020] As explained above, the video-related information determination device 100 of the second disclosure detects trigger events from data relating to industrial machinery. When the video-related information determination device 100 detects a trigger event, it reads out the event set corresponding to the trigger event. If the events included in the event set exist in the data history, the video-related information determination device 100 associates the related information corresponding to the event set with the video. By linking video footage with related information to support it, it becomes possible to generate highly visible video footage, as the information needed by industrial machine operators can be presented.

[0021] [Specific example of a video-related information determination device 1] Next, a specific example of the video-related information determination device 100 will be described. In this example, based on the user's operation history, the device determines the tasks performed while the video was being filmed and reads out information useful for those tasks as related information.

[0022] Figure 6 shows an example of a trigger event and event set. The task "Work Coordinate Measurement" is associated with the trigger event "Setting Work Coordinate Data," and the event set consisting of five events: "Replacing with Touch Probe," "Displaying Work Measurement Screen," "Jog or Handle Operation," "Touch Probe Contacts Workpiece," and "Setting Work Coordinate Data." Furthermore, the task "Work Coordinate Measurement" is associated with the related information "Work Coordinate List."

[0023] Of the five events, "Replacing with touch probe" is the starting event. As the following events occur in order from the starting event—"Display of work measurement screen," "Jog or handle operation," "Touch probe contacts workpiece," and "Setting workpiece coordinate data"—the video at the time of the event occurrence is associated with the operation "Workpiece coordinate measurement."

[0024] Furthermore, the "Start Machining" operation is linked to an event set consisting of three events: the trigger event "Pressing the Cycle Start button," and the event set "Pressing the button to change NC mode to MEM," "Displaying the automatic operation screen," and "Pressing the Cycle Start button." In addition, the "Start Machining" operation is linked to related information: "Workpiece drawing" and "Machining process information (machining type, tools used, etc.)."

[0025] Of the three events, "pressing the button to change NC mode to MEM" is the start event. If the events "displaying the automatic operation screen" and "pressing the cycle start button" are executed in order after the start event, the video and the operation "start machining" will be linked.

[0026] Here, we show an example of searching for events by working backward in time from a trigger event to determine the unit of work. Referring to Figure 7, the process for determining the work unit will be explained. The related information determination unit 3 searches the data acquired by the data acquisition unit 1 and detects the trigger event "Work coordinate data setting". The related information determination unit 3 reads the event set corresponding to the detected trigger event. Here, the relationship between a trigger event and an event set is not one-to-one; a single trigger event may correspond to multiple event sets. In such cases, searching for events by working backward in time from the trigger event will eventually converge to a single event set (unit of work).

[0027] In the example in Figure 7, when the trigger event "Setting work coordinate data" is detected, the system searches for the preceding event, "Touch probe touches workpiece." If the event "Touch probe touches workpiece" exists, it searches for the preceding event, "Jog or handle operation." The system searches for events by working backward in time, and when it reaches the starting event, it determines the work unit corresponding to the event set.

[0028] Once the video work unit is determined, the relevant information useful for that work can be determined. Refer to Figures 8 and 9 to see examples of images generated by the image-related information determination device 100. The video in Figure 8 is divided into three areas: the upper left of the screen shows the display screen of the numerical control unit during the execution of the "Work Coordinate Data Setting" operation; the upper right of the screen shows the in-machine video during the execution of the "Work Coordinate Data Setting" operation; and the "Work Coordinate List" is displayed at the bottom of the screen. Adding an image of the "Work Coordinate List" to the video of the task "Setting Work Coordinate Data" allows users to see the task and the work coordinates set by that task. The "Work Coordinate List" is useful information (related information) for confirming the details of the task. Linking the video with related information improves user visibility.

[0029] The video in Figure 9 is divided into three areas: the upper left of the screen shows the display screen of the numerical control device when the operation "Start Machining" is executed, the upper right of the screen shows the in-machine video when the operation "Start Machining" is executed, and the bottom of the screen displays the "Workpiece Drawing". Adding an image of the "workpiece drawing" to the video of the "start of machining" operation allows users to confirm that the workpiece is being machined according to the design. Although the image is not displayed on the screen during the execution of the "start of machining" operation, it is useful information (related information) for confirming the work content. Linking the video with related information improves user visibility.

[0030] The linking unit 7 links video and related information to the start and end times of work units. The linking unit 7 may also link video and related information according to the time of events included in the work unit. The linking unit 7 may also generate a seek bar indicating the start and end times of work units. Figure 10 is an example of a seek bar. From the start time to the end time of the work "Work Coordinate Data Setting", the "Work Coordinate List" is displayed as related information. From the start time to the end time of the work "Machining Start", the "Work Drawing" is displayed as related information.

[0031] As explained above, in the specific example 1 of the video-related information determination device, event sets and work units are associated, and video is generated that links useful information for confirming work content (related information) with the video, thereby improving visibility.

[0032] [Specific example of a video-related information determination device 2] In specific example 2, the trigger event is an alarm. The trigger event (alarm) is associated with the alarm cause (event set). The related information determination unit 3 searches the data acquired by the data acquisition unit 1, and when it detects a trigger event (alarm), it reads out the alarm cause (event set) associated with the alarm. The related information determination unit 3 searches past data stored in the data history storage unit 4 and, if an event set corresponding to the detected alarm occurred in the past, determines that the related information corresponding to the event set will be displayed as related information to supplement the video. Related information includes parameter settings, sensor position, tool offset, image detection results, sensor values ​​such as laser, acceleration, vibration, and tilt, motor current, torque, machine drawings, and manuals. For example, if an interference error is detected, the machine drawing of the interfering area will be displayed.

[0033] An example of an alarm is described below. In workpiece measurement, the dimensions of the workpiece after machining are measured using a touch probe or the like, as shown in Figure 11. In this example, the diameter of the machined hole is measured. The numerical control device compares the actual hole diameter with the designed hole diameter, and if the actual hole diameter exceeds the tolerance value of the designed hole diameter, it generates an alarm. The alarm information is stored as an alarm number (for example, "MC3770") in the data history storage unit 4. When the related information determination unit 3 detects an alarm number as a trigger event, it reads the event set associated with the alarm. The related information determination unit 3 determines from the event set that it is an "alarm caused by measuring the workpiece after machining." The related information determination unit 3 reads the workpiece drawing as related information corresponding to the event set.

[0034] As shown in Figure 12, the video-related information determination device 100 adds a workpiece drawing to the video at the time of the alarm. The operator compares the actual diameter of the hole formed in the workpiece with the design diameter of the hole and adjusts the machine. For example, if there is insufficient cutting, the tool offset value is adjusted to adjust the amount of cutting. If there is too much cutting, since a workpiece that has been over-cut cannot be corrected, the dimensions of the workpiece drawing are checked and the tool offset value is adjusted. For the next workpiece, machining is performed with the new tool offset value.

[0035] As explained above, in specific example 2, an alarm is detected as a trigger event, and related information corresponding to the alarm cause (event set) associated with the trigger event is linked to the video.

[0036] The hardware configuration of the video-related information determination device 100 to which this disclosure applies will be described below. Figure 13 is a hardware configuration diagram of the video-related information determination device 100. As shown in Figure 13, the video-related information determination device 100 includes a CPU 111 that controls the video-related information determination device 100 as a whole, a ROM 112 that records programs and data, and a RAM 113 for temporarily expanding data. The CPU 111 reads the system program recorded in the ROM 112 via the bus and controls the entire video-related information determination device 100 according to the system program.

[0037] The non-volatile memory 114 is backed up, for example, by a battery (not shown), so that its stored state is maintained even when the power to the video-related information determination device 100 is turned off. The non-volatile memory 114 stores various data, such as programs read from external devices 120 via interfaces 115, 118, and 119, and user operations input via the input unit 30.

[0038] Interface 115 is an interface 115 for connecting the video-related information determination device 100 to an external device 120 such as an adapter. Programs and various parameters are read from the external device 120. [Explanation of Symbols]

[0039] 100 Video-related information determination device 1. Data Acquisition Unit 2. Video acquisition unit 3. Related Information Decision Department 4. Data history storage unit 5 Event Set Memory Unit 6. Related Information Storage Unit 7. Linking section 111 CPU 112 ROM 113 RAM 114 Non-volatile memory

Claims

1. A video acquisition unit that acquires video showing the operation of an industrial machine, A data acquisition unit that acquires data indicating the mechanical status of the aforementioned industrial machine, A data history storage unit for storing data of the aforementioned industrial machine, An event set is a combination of multiple events that occur in a time order, including at least one trigger event, and an associated information storage unit stores associated information corresponding to the event set. A related information determination unit searches the aforementioned data, detects the occurrence of the trigger event contained in the related information storage unit, searches past data stored in the data history storage unit, determines whether an event set, which is a combination of events corresponding to the trigger event, has occurred in the past from the trigger event, and if it has occurred, refers to the related information storage unit to determine related information corresponding to the event set. Based on the event occurrence times of the events included in the event set, a linking unit links the video acquired by the video acquisition unit with the related information. A video-related information determination device equipped with the following features.

2. The video-related information determination device according to claim 1, wherein the linking unit generates a seek bar indicating the occurrence time of the event.

3. The trigger event is user operation data, and the event set is a combination of user operation data. The video-related information determination device according to claim 1, wherein the related information determination unit determines the work performed at the time the video was filmed based on the event set, and determines related information for the work.

4. The video-related information determination device according to claim 1, wherein the trigger event is an alarm of the industrial machine, and the event set is the cause of the alarm.

5. One or more processors execute, We obtained video footage showing the operation of industrial machinery. Data indicating the status of the aforementioned industrial machine is acquired, The system searches the data and detects the occurrence of the trigger event contained in the related information storage unit, which stores an event set (a combination of multiple events that occur in a time order including at least one trigger event) and related information corresponding to the event set in association with it; it searches past data that stores the industrial machine data as history and determines whether the event set (a combination of events corresponding to the trigger event) has occurred in the past since the trigger event; and if it has occurred, it refers to the related information storage unit to determine the related information corresponding to the event set. Based on the event occurrence times of the events included in the aforementioned event set, the video and the related information are linked. A storage medium for storing instructions that the processor can read.

Images