Video generation device and computer-readable storage medium

Abstract

Provided is a video generation device which generates a recorded video of an industrial machine, acquires data indicating a state of the industrial machine, searches data indicating a state of the industrial machine, detects a trigger event indicating work which the industrial machine has carried out, determines a viewpoint corresponding to the trigger event, and generates video based on the determination of the viewpoint.

Description

Image generation device and computer-readable storage medium

[0001] The present invention relates to an image generation device and a computer-readable storage medium.

[0002] Currently, there are technologies that monitor the status of industrial machinery and utilize them for improving control efficiency and machine maintenance. These technologies collect and analyze data related to industrial machinery to understand the status of the machinery, prevent breakdowns, check product quality, and improve production management.

[0003] Industrial machinery and the machinery detected by sensors installed around it are recorded in data collection devices called data loggers. Many data loggers are installed in factories to record the status of the machinery within the factory. In addition, programmable logic controllers (PLCs) that control the operating sequence of machines and numerical control devices that control machine tools also have internal recording units that store control information and status information.

[0004] The collected data is converted into a format that is easy for the operator to view. Patent Document 1 states that "when the time loading means acquires control data acquired from the control device, image data acquired from the camera, and sound data acquired from the controlled device, it creates time-added control data, time-added image data, and time-added sound data by adding the time at which each data was acquired, and stores these in a temporary file. Then, the log data generating means reads the most recent set of time-added control data, time-added image data, and time-added sound data from the time-added control data, time-added image data, and time-added sound data stored in the temporary file, and generates log data containing the time, control data, image data, and sound data, thereby enabling each data to be played back in a synchronized manner." According to Patent Document 1, the images, sounds, and logs of the controlled device are played back in a synchronized manner.

[0005] Patent Document 2 states that "a camera that takes pictures of the robot is connected to the controller, and image data showing the robot's behavior obtained by the image capture is supplied to the controller in the form of an electrical signal," and "when an error (event) occurs, an error record related to the error is created separately from the operation log." According to Patent Document 2, when an event occurs while an image is being captured, the event and the image are recorded in association with each other.

[0006] JP 2001-125612 A JP 2019-150923 A

[0007] In Patent Document 1, an image and a log are displayed simultaneously, while in Patent Document 2, a log (data) is recorded when an event occurs. If a log and an image are available when an error occurs, the situation at the time of the error can be clarified and the cause of the error can be identified. Similarly, if a highly visible image is created, it can be used to maintain industrial machinery and improve productivity.

[0008] In the field of industrial machinery data management, there is a challenge in improving visibility.

[0009] An image generating device according to one aspect of the present disclosure includes a data acquisition unit that acquires data indicating a machine state of industrial machinery, a viewpoint determination unit that searches the data indicating the state of the industrial machinery, detects a trigger event indicating work performed on the industrial machinery, and determines a viewpoint corresponding to the trigger event, and an image generating unit that generates an image of the industrial machinery based on the viewpoint determined by the viewpoint determination unit.A storage medium according to one aspect of the present disclosure stores processor-readable instructions that, when executed by one or more processors, acquire data indicating the state of the industrial machinery, search the data indicating the state of the industrial machinery, detects a trigger event indicating work performed on the industrial machinery, determine a viewpoint corresponding to the trigger event, and generate an image based on the viewpoint determination.

[0010] According to one embodiment of the present invention, the visibility of an image can be improved.

[0011] 1 is a diagram showing the application range of an image generation device. FIG. 1 is a block diagram of an image generation device of the first disclosure. FIG. 2 is a conceptual model of a viewpoint determination unit of the first disclosure. FIG. 3 is a block diagram of an image generation device of the second disclosure. FIG. 4 is a conceptual model of a viewpoint determination unit of the second disclosure. FIG. 5 is a diagram showing an example of an event set. FIG. 6 is a diagram showing an example of work before machining starts on a lathe. FIG. 7 is a diagram showing an example of an image generated by an image generation unit. FIG. 8 is a diagram showing an example of work before machining starts on a machining center. FIG. 9 is a diagram showing an example of an image generated by an image generation unit. FIG. 10 is a diagram showing a trigger event and an event set for work measurement work (end face). FIG. 11 is a diagram showing a trigger event and an event set for work measurement (hole distance measurement). FIG. 12 is a diagram showing a method for searching for an event set. FIG. 13 is a diagram showing the relationship between task units and viewpoints. FIG. 14 is a diagram showing an example of a seek bar with task units arranged on a time axis. FIG. 15 is a diagram showing an example of combining two images into one. FIG. 16 is a diagram showing the hardware configuration of an image generation device.

[0012] The following describes an image generation device 100 to which the present disclosure is applied. The image generation device 100 acquires data showing the status of industrial machines installed in a factory. The image generation device 100 uses already accumulated data to determine the viewpoint of an image showing the operation of the industrial machines. Target industrial machines include, but are not limited to, machine tools, numerical control devices, robots, etc.

[0013] The image generation device 100 is applied to information processing devices such as PCs (personal computers), servers, and mobile terminals. Specifically, as shown in FIG. 1 , the image generation device 100 is applied to, but not limited to, a PC used by a factory manager, a server on the cloud, a PC used by an operator of a machine manufacturer or a control machine manufacturer, and a mobile terminal used by a factory manager. Images generated by the image generation device 100 are used for managing industrial machinery (maintenance, production management, etc.). The image generation device 100 may also be applied to information processing devices such as a numerical control device and a PLC (Programmable Logic Controller).

[0014] 2 is a block diagram of an image generation device 100 according to the first disclosure. The image generation device 100 includes a data acquisition unit 1, an image generation unit 2, and a viewpoint determination unit 3.

[0015] The data acquisition unit 1 acquires data related to industrial machinery. The data acquired by the data acquisition unit 1 includes status information of the industrial machinery, operation input information, control information of the industrial machinery, etc. In this disclosure, a numerical control device and a machine tool are taken as examples of industrial machinery. Status information of the industrial machinery includes the mode of the numerical control device, tool offset amount, alarm, time, number of machined parts, modal information, position information (coordinate system), power consumption of the machine tool, load of the servo motor, data measured by sensors (temperature and pressure), etc. Operation input information is operation input from an operator to a touch panel, button, dial, handle, etc.

[0016] The image generator 2 generates a simulation image showing the operation of the industrial machine using existing simulation technology. The viewpoint of the simulation image can be switched. Although the present disclosure provides an example of generating a simulation image, the image generated by the image generator 2 may be a two-dimensional image based on the result of the viewpoint determination unit 3 instead of a simulation image.

[0017] The viewpoint determination unit 3 determines the viewpoint of the simulation video generated by the video generation unit 2. Fig. 3 is a conceptual model of the viewpoint determination unit 3. The viewpoint determination unit 3 monitors the data acquired by the data acquisition unit 1 and detects a trigger event. The viewpoint determination unit 3 determines a viewpoint corresponding to the detected trigger event. Specific examples of trigger events will be described later.

[0018] The video generation unit 2 generates a simulation video from the viewpoint determined by the viewpoint determination unit 3. Alternatively, a two-dimensional image may be generated based on the result of the viewpoint determination unit 3, rather than a simulation video. This allows a recorded video from a viewpoint suitable for checking the operation of industrial machinery to be created.

[0019] As described above, the image generating device 100 of the first disclosure acquires data related to industrial machinery, detects trigger events, and generates recorded video with a viewpoint suitable for checking the operation of the industrial machinery. According to the present disclosure, it is possible to switch the viewpoint of the video in accordance with the work of the industrial machinery and generate video with high visibility.

[0020] 4 is a block diagram of an image generation device 100 according to a second disclosure. The image generation device 100 includes a data acquisition unit 1, an image generation unit 2, a data history storage unit 7, a viewpoint determination unit 3, and an event set storage unit 8.

[0021] The image generation device 100 of the second disclosure includes a data history storage unit 7 and an event set storage unit 8. The data history storage unit 7 stores the history of data acquired by the data acquisition unit 1. The event set storage unit 8 stores a combination of data (event set) that occurs when a task is performed. FIG. 5 shows a conceptual model of the viewpoint determination unit 3. The viewpoint determination unit 3 searches the data acquired by the data acquisition unit 1 and detects a trigger event. The viewpoint determination unit 3 reads an event set corresponding to the detected trigger event and searches past data stored in the data history storage unit 7 to determine whether an event set corresponding to the detected trigger event has occurred in the past. If an event set corresponding to the detected trigger event has occurred in the past, it is determined that a task has been performed. An event set corresponds to a task unit from the start to the end of the task. In the present disclosure, an event that occurs at the start of a task is called a start event. The period from the start event to the trigger event constitutes one task unit. Specific examples of event sets will be described later.

[0022] The video generation unit 2 generates a video based on the result of the viewpoint determination unit 3. The time at which an event occurred may be added to the video generated by the video generation unit 2. This time can be used to create a seek bar (described later) or to merge with other videos.

[0023] As described above, the image generation device 100 of the second disclosure determines a viewpoint depending on whether an event (trigger event and event set) associated with an activity occurs. The image generation device 100 generates a simulation image based on the determined viewpoint. By switching the viewpoint, it is possible to generate an image with high visibility.

[0024] [Specific Example of the Second Disclosure] A specific example of the image generation device 100 of the second disclosure will be described below. In this example, the events will be limited to the operation signal of the operator and the screen display.

[0025] Switching of viewpoints in a numerical control device that controls a machine tool will be described with reference to Figures 6, 11, and 12. It is assumed that the viewpoint determination unit 3 searches for a trigger event from the data acquired by the data acquisition unit 1. It is assumed that the viewpoint determination unit 3 detects the trigger event "pressing the cycle start button." Figure 6 shows an example of an event set. In Figure 6, the trigger event "pressing the cycle start button" is linked to an event set consisting of three events: "pressing the button to change the NC mode to MEM," "displaying the automatic operation screen," and "pressing the cycle start button."

[0026] Of the three events, "Pressing the button to change the NC mode to MEM" is the start event. If the events "Displaying the automatic operation screen" and "Pressing the cycle start button" are executed in this order from the start event, it is clear that the task unit "Start of machining" has been executed.

[0027] The viewpoint determination unit 3 detects "pressing the cycle start button" and "pressing the button to change the NC mode to MEM" from the operation signal. "Display of automatic operation screen" can be detected from the "screen ID" included in the operation signal history. "Screen ID" is an ID assigned to each screen. It can be detected from the "screen ID" that the "automatic operation screen" has been displayed. If events are executed in the order defined in the event set, the viewpoint determination unit 3 can determine that the operation "start machining" has been executed. Note that in the above explanation, the order of occurrence of events is fixed, but the order may be random.

[0028] As preparation before starting machining, the tool is moved at rapid traverse to the machining start position. When machining is performed using a new machining program that has not previously been used, the tool is generally paused at the machining start position, or slowed down by operating the feed rate override on the industrial machine's control panel. This is because if the distance between the workpiece and the tool at the machining start position is the expected distance, it can be determined that the "machining program" created by the programmer and the setting values ​​for the "tool offset" and "workpiece coordinate settings" set at the machining site are correct. In the work before machining begins, it is useful to have a video that makes it easy to see the distance between the workpiece and the tool.

[0029] The appropriate viewpoint varies depending on the type of machine tool and the direction of tool movement. Figure 7 shows an example of a lathe. In a lathe, the Z axis is usually parallel to the rotation axis of the spindle, and the Z axis is the middle finger, with the thumb direction being the X axis in a right-handed Cartesian coordinate system. To check the distance between the tool and the workpiece before cutting begins, a viewpoint perpendicular to the XZ plane is best. Figure 8 shows an image generated from a viewpoint perpendicular to the XZ plane. By switching the viewpoint, it is easy to see the distance between the workpiece and the tool. Note that the image in Figure 8 is a two-dimensional image based on the results of the viewpoint determination unit 3.

[0030] FIG. 9 shows an example of a machining center. Similarly, in a machining center, the tool moves at high speed to the cutting start position and then "pauses" or "slows down." In this example, the tool that has started cutting moves parallel to the X-axis of the workpiece coordinate system. In this example, the distance between the tool and the workpiece is easy to check when the viewpoint is perpendicular to the XZ plane. FIG. 10 shows an image generated from a viewpoint perpendicular to the XZ plane. By switching the viewpoint, the distance between the workpiece and the tool can be easily seen. Note that the image in FIG. 10 is a two-dimensional image based on the results of the viewpoint determination unit 3. The viewpoint determination unit 3 may analyze the machining program and determine the viewpoint based on the movement direction of the tool.

[0031] 11 is a diagram showing a trigger event and an event set for a workpiece measurement operation (end face). The trigger event for a workpiece measurement operation (end face) is "workpiece coordinate data setting." The trigger event "workpiece coordinate data setting" is linked to an event set consisting of three events: "display of workpiece measurement screen," "selection of workpiece measurement (end face) on the workpiece measurement screen," and "workpiece coordinate data setting."

[0032] Of the three events, "display of work measurement screen" is the start event. If the events "select work measurement (end face) on work measurement screen" and "set work coordinate data" are executed in order from the start event, it can be determined that the task unit "start of work measurement (end face)" has been executed. The viewpoint determination unit 3 searches the data acquired by the data acquisition unit 1 to detect data indicating the occurrence of an event set.

[0033] Next, a method for identifying a task unit by searching for events while tracing back in time from a trigger event will be described. FIG. 12 is a diagram showing the trigger event and event set for workpiece measurement (hole-to-hole measurement). The trigger event in FIG. 12 is "workpiece coordinate data setting." The trigger event "workpiece coordinate data setting" is linked to an event set consisting of six events: "switch to touch probe," "display of workpiece coordinate measurement screen," "selection of hole-to-hole measurement," "axis operation by jog," "touch probe contact with workpiece," and "workpiece coordinate data setting."

[0034] The trigger event "Workpiece coordinate data setting" in Fig. 12 is the same trigger event as the task unit "Workpiece measurement (end face)" in Fig. 11. In other words, multiple event sets may correspond to one trigger event. In such a case, if events are searched for by tracing back in time from the trigger event, they will eventually converge to one event set (task unit).

[0035] A specific example is shown in Figure 13. When a trigger event called "workpiece coordinate data setting" is detected, the previous event, "touch probe contacts workpiece," is searched for. If the event "touch probe contacts workpiece" exists, the event immediately before that, "axis operation by jog," is searched for. If the event "touch probe contacts workpiece" does not exist, events included in other event sets (for example, Figure 11) are searched for. Events are searched backward in time, and when the start event is reached, the work unit corresponding to the event set is determined.

[0036] Once the task unit is determined, the viewpoint is determined. Figure 14 shows the relationship between task units and viewpoints. For the "start machining" task (machining center), the viewpoint is determined to be one in which the Z axis is vertical and the workpiece and tool are viewed from the side. For the "workpiece measurement (end face)" task, the viewpoint is determined to be one in which the Z axis is vertical and the workpiece and tool are viewed from the side. For the "workpiece measurement (hole distance measurement)", the viewpoint is determined to be one in which the Z axis is vertical and the workpiece is viewed from above. The viewpoint can be defined by the direction and distance relative to the object. For example, the viewpoint may be selected from predefined directions and distances as seen from the tool and workpiece. Alternatively, the viewpoint may be determined automatically by calculating through simulation the direction and distance that will prevent any obstacles from getting between the object and the viewpoint.

[0037] By arranging task units on the time axis, a video seek bar can be created, as shown in Figure 15. Creating a seek bar allows you to check the overall work flow, improving the visibility of the data.

[0038] You can also create a video seek bar by placing the occurrence time of an event on the timeline. By creating a seek bar, you can not only visually grasp the workflow, but also confirm the work in text rather than just the video, improving the visibility of the data.

[0039] [Third Disclosure] In the third disclosure, an image on the display screen of the numerical control device and an image generated by the image generation unit 2 are displayed side by side. FIG. 16 shows an example in which an image on the display screen of the numerical control device and an image generated by the image generation unit 2 are displayed side by side. The image generation device 100 merges the image on the display screen of the numerical control device and the generated image in terms of time, and generates a single image that combines two images synchronized in terms of time. In the example of FIG. 16, an image on the display screen of the numerical control device and an image generated by the image generation unit 2 are displayed side by side. The image on the display screen may be captured instead of being captured by a camera. By displaying the image on the display screen and the simulation image side by side, the operator's operation and the state of the industrial machine can be checked simultaneously.

[0040] The hardware configuration of an image generation device 100 to which the present disclosure is applied will be described below. Fig. 17 is a hardware configuration diagram of the image generation device 100. As shown in Fig. 17, the image generation device 100 includes a CPU 111 that controls the entire image generation device 100, a ROM 112 that records programs and data, and a RAM 113 for temporarily expanding data. The CPU 111 reads a system program recorded in the ROM 112 via a bus and controls the entire image generation device 100 in accordance with the system program.

[0041] The nonvolatile memory 114 is backed up by, for example, a battery (not shown), and the stored state is maintained even when the power to the image generation device 100 is turned off. The nonvolatile memory 114 stores various data such as programs read from external devices via the interfaces 115, 118, and 119, and user operations input via the input unit 71.

[0042] The interface 115 mediates the connection between the image generation device 100 and an external device such as an adapter, and reads programs and various parameters from the external device.

[0043] REFERENCE SIGNS LIST 100 Image generation device 1 Data acquisition unit 2 Image generation unit 3 Viewpoint determination unit 7 Data history storage unit 8 Event set storage unit 111 CPU 113 RAM 114 Non-volatile memory

Claims

1. A data acquisition unit that acquires data indicating a machine state of the industrial machine; a viewpoint determination unit that searches data indicating a state of the industrial machine, detects a trigger event indicating an operation performed by the industrial machine, and determines a viewpoint corresponding to the trigger event; an image generating unit that generates an image of the industrial machine based on the viewpoint determined by the viewpoint determining unit; An image generating device comprising:

2. a data history storage unit that stores a history of the data acquired by the data acquisition unit; 2. The image generating device of claim 1, wherein the viewpoint determination unit stores an event set which is a combination of a plurality of events, and when the trigger event is detected, searches whether data indicating the occurrence of an event included in the event set exists in the data history storage unit, and if data indicating the occurrence of an event included in the event set exists in the data history storage unit, determines a work unit corresponding to the event set and determines a viewpoint corresponding to the work unit.

3. 2. The video generating apparatus according to claim 1, wherein the viewpoint determining section searches for data indicating occurrence of an event included in the event set, tracing back in time from the trigger event as a starting point.

4. The image generating device according to claim 2 , further comprising: a time when the event occurred added to the image generated by the image generating unit.

5. The image generating device according to claim 2 , further comprising: a seek bar that indicates occurrence times of the events included in the event set.

6. The image generating apparatus according to claim 2 , further comprising: a seek bar that indicates an occurrence time of the task unit.

7. The image generating device according to claim 1 , wherein the image generating unit generates a simulation image showing an operation of the industrial machine, or a two-dimensional image based on a result of the viewpoint determining unit.

8. When executed by one or more processors, Acquire data showing the condition of industrial machines, retrieving data indicative of a state of the industrial machine to detect a trigger event indicative of an operation performed on the industrial machine; and determining a viewpoint corresponding to the trigger event; generating an image based on said viewpoint determination; A storage medium that stores instructions readable by the processor.