Image generation device and computer-readable storage medium

Abstract

Provided is a video generation device which generates a recorded video of an industrial machine, acquires data indicating a state of the industrial machine, searches data indicating a state of the industrial machine, detects a trigger event indicating work which the industrial machine has carried out, determines a viewpoint corresponding to the trigger event, and generates video based on the determination of the viewpoint.

Description

Technical Field

[0001] The present invention relates to an image generation device and a computer-readable storage medium.

Background Art

[0002] Currently, there are technologies for monitoring the state of industrial machines and utilizing them for improving control efficiency and machine maintenance. In such technologies, data related to industrial machines is collected, and by analyzing the collected data, the state of industrial machines is grasped, and it is used to prevent failures of industrial machines, confirm the quality of products, and improve production management.

[0003] Industrial machines and the machines detected by sensors provided around them are recorded in a data collection device called a data logger. A large number of data loggers are provided in the factory to record the state of the machines in the factory. In addition, a PLC (Programmable Logic Controller) that controls the operation sequence of a machine and a numerical control device (Numerical Control) that controls a machine tool also have a recording unit inside and store control information and state information.

[0004] The collected data is converted into a format that is easy for the operator to view. Patent Document 1 states that "when the time loading means acquires control data from the control device, image data from the camera, and sound data from the controlled device, it creates time-added control data, time-added image data, and time-added sound data by adding the time at which each data was acquired, and stores these in a temporary file. Then, the log data formation means reads out the time-added control data, time-added image data, and time-added sound data that are closest in time from among the time-added control data, time-added image data, and time-added sound data stored in the temporary file, and forms log data having time, control data, image data, and sound data, thereby enabling playback of each data in synchronization." According to Patent Document 1, the image, sound, and log of the controlled device are played back in synchronization with time.

[0005] Patent Document 2 states that "a camera for photographing the robot is connected to the controller, and image data showing the robot's behavior obtained from the photograph is supplied to the controller in the form of electrical signals," and "when an error (event) occurs, an error record related to the error is created separately from the operation log." According to Patent Document 2, when an event occurs when an image is taken, the event and the image are recorded in association. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2001-125612 [Patent Document 2] Japanese Patent Publication No. 2019-150923 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Patent Document 1 displays images and logs simultaneously, while Patent Document 2 records logs (data) when an event occurs. Having logs and images from the time of an error helps to clarify the circumstances surrounding the error and aids in identifying its cause. Similarly, creating highly visible video footage can be used for the maintenance of industrial machinery and improving productivity.

[0008] In the field of industrial machinery data management, there is a challenge in improving visibility. [Means for solving the problem]

[0009] An image generation device according to one aspect of this disclosure includes: a data acquisition unit that acquires data indicating the mechanical state of an industrial machine; a viewpoint determination unit that searches for data indicating the state of the industrial machine, detects trigger events indicating work performed on the industrial machine, and determines a viewpoint corresponding to the trigger event; and a view generation device that generates images of the industrial machine based on the viewpoint determined by the viewpoint determination unit. Simulation demonstrating operation It comprises a video generation unit that generates video. Another aspect of the present disclosure, a video generation device, comprises: a data acquisition unit that acquires data indicating the mechanical state of an industrial machine; a viewpoint determination unit that searches for data indicating the state of the industrial machine, detects trigger events indicating work performed on the industrial machine, and determines a viewpoint corresponding to the trigger event; and a video generation unit that generates video of the industrial machine based on the viewpoint determined by the viewpoint determination unit, wherein the viewpoint determination unit searches for data indicating the occurrence of events included in the event set, working backward in time from the trigger event. A video generation device in yet another aspect of the present disclosure includes a data acquisition unit that acquires data indicating the machine state of an industrial machine, a viewpoint determination unit that searches for data indicating the state of the industrial machine, detects trigger events indicating work performed on the industrial machine, and determines a viewpoint corresponding to the trigger event, a video generation unit that generates video of the industrial machine based on the viewpoint determined by the viewpoint determination unit, and a data history storage unit that stores the history of data acquired by the data acquisition unit. The viewpoint determination unit stores an event set which is a combination of multiple events, and when it detects a trigger event, it searches whether data indicating the occurrence of an event included in the event set exists in the data history storage unit, and if data indicating the occurrence of an event included in the event set exists in the data history storage unit, it determines a work unit corresponding to the event set, determines a viewpoint corresponding to the work unit, and further performs any of (1) to (3) described below. (1) The time of the event is added to the video generated by the video generation unit. (2) Create a seek bar that shows the occurrence times of the events included in the event set. (3) Create a seek bar that shows the time of occurrence of each work unit. A storage medium in one aspect of this disclosure, by being executed by one or more processors, acquires data indicating the state of an industrial machine, retrieves data indicating the state of an industrial machine, detects trigger events indicating work performed on the industrial machine, determines a viewpoint corresponding to the trigger event, and based on the determination of the viewpoint Simulation showing the operation of industrial machinery It stores instructions that the processor can read to generate images. A storage medium in another aspect of the present disclosure stores processor-readable instructions which, when executed by one or more processors, acquire data indicating the state of an industrial machine, retrieve data indicating the state of an industrial machine, detect trigger events indicating work performed on the industrial machine, determine a viewpoint corresponding to a trigger event by retrieving data indicating the occurrence of events included in the event set, starting from the trigger event and working backward in time, and generate video based on the determined viewpoint. [Effects of the Invention]

[0010] According to one aspect of the present invention, the visibility of images can be improved. [Brief explanation of the drawing]

[0011] [Figure 1]It is a diagram showing the applicable range of the video generation device. [Figure 2] It is a block diagram of the video generation device of the first disclosure. [Figure 3] It is a conceptual model of the viewpoint determination unit of the first disclosure. [Figure 4] It is a block diagram of the video generation device of the second disclosure. [Figure 5] It is a conceptual model of the viewpoint determination unit of the second disclosure. [Figure 6] It is a diagram showing an example of an event set. [Figure 7] It is a diagram showing an example of the work before starting machining on a lathe. [Figure 8] It is a diagram showing an example of the video generated by the video generation unit. [Figure 9] It is a diagram showing an example of the work before starting machining in a machining center. [Figure 10] It is a diagram showing an example of the video generated by the video generation unit. [Figure 11] It is a diagram showing the trigger event and event set for the workpiece measurement operation (end face). [Figure 12] It is a diagram showing the trigger event and event set for the workpiece measurement (measurement between holes). [Figure 13] It is a diagram showing the search method of the event set. [Figure 14] It is a diagram showing the relationship between the work unit and the viewpoint. [Figure 15] It is a diagram showing an example of a seek bar in which the work units are arranged on the time axis. [Figure 16] It is a diagram showing an example of combining two videos into one. [Figure 17] It is a hardware configuration diagram of the video generation device.

Embodiments for Carrying Out the Invention

[0012] The following describes a video generation device 100 to which this disclosure applies. The video generation device 100 acquires data indicating the status of industrial machinery placed in a factory. Using the already accumulated data, the video generation device 100 determines the viewpoint for the video showing the operation of the industrial machinery. The target industrial machinery includes, but is not limited to, machine tools, numerical control devices, and robots.

[0013] The video generation device 100 is applicable to information processing devices such as PCs (personal computers), servers, and mobile terminals. Specifically, as shown in Figure 1, the video generation device 100 is applicable to PCs used by factory managers, servers on the cloud, PCs used by operators of machine manufacturers and control machine manufacturers, and mobile terminals used by factory managers, but is not limited to these. The video generated by the video generation device 100 is used for the management of industrial machinery (maintenance, production management, etc.). Furthermore, the video generation device 100 may be applied to information processing devices such as numerical control devices and PLCs (Programmable Logic Controllers).

[0014] [First Disclosure] Figure 2 is a block diagram of the first disclosed video generation device 100. The video generation device 100 comprises a data acquisition unit 1, a video generation unit 2, and a viewpoint determination unit 3.

[0015] The data acquisition unit 1 acquires data related to industrial machinery. The data acquired by the data acquisition unit 1 includes status information of the industrial machinery, operation input information, and control information of the industrial machinery. In this disclosure, numerical control devices and machine tools are used as examples of industrial machinery. State information of industrial machinery includes the mode of the numerical control device, tool offset amount, alarms, time, number of machined parts, modal information, position information (coordinate system), power consumption of the machine tool, servo motor load, and data measured by sensors (temperature and pressure). Operation input information refers to operations input from the operator to touch panels, buttons, dials, handles, etc.

[0016] The video generation unit 2 generates a simulation video showing the operation of an industrial machine using existing simulation technology. The viewpoint of the simulation video can be switched. This disclosure provides an example of generating a simulation video, but the video generated by the video generation unit 2 may be a 2D video based on the results of the viewpoint determination unit 3, rather than a simulation video.

[0017] The viewpoint determination unit 3 determines the viewpoint of the simulated video generated by the video generation unit 2. Figure 3 is a conceptual model of the viewpoint determination unit 3. The viewpoint determination unit 3 monitors the data acquired by the data acquisition unit 1 and detects trigger events. The viewpoint determination unit 3 determines the viewpoint corresponding to the detected trigger event. Specific examples of trigger events will be described later.

[0018] The video generation unit 2 generates a simulated video from the viewpoint determined by the viewpoint determination unit 3. Alternatively, instead of a simulated video, a 2D image based on the result of the viewpoint determination unit 3 may be generated. This creates a recorded video from a viewpoint suitable for verifying the operation of industrial machinery.

[0019] As described above, the video generation device 100 of the first disclosure acquires data related to industrial machinery, detects trigger events, and generates recorded video from a viewpoint suitable for verifying the operation of the industrial machinery. According to this disclosure, it is possible to switch the viewpoint of the video in accordance with the operation of the industrial machinery and generate highly visible video.

[0020] [Second Disclosure] Figure 4 is a block diagram of the second disclosure video generation device 100. The video generation device 100 comprises a data acquisition unit 1, a video generation unit 2, a data history storage unit 7, a viewpoint determination unit 3, and an event set storage unit 8.

[0021] The second disclosed video generation device 100 includes a data history storage unit 7 and an event set storage unit 8. The data history storage unit 7 stores the history of data acquired by the data acquisition unit 1. The event set storage unit 8 stores combinations of data (event sets) that occur when an operation is performed. Figure 5 is a conceptual model of the viewpoint determination unit 3. The viewpoint determination unit 3 searches the data acquired by the data acquisition unit 1 and detects trigger events. The viewpoint determination unit 3 reads the event set corresponding to the detected trigger event, searches the past data stored in the data history storage unit 7, and checks whether the event set corresponding to the detected trigger event has occurred in the past. If the event set corresponding to the detected trigger event has occurred in the past, it is determined that a certain operation has been performed. An event set corresponds to a work unit from the start to the end of a task. In this disclosure, the event that occurs at the start of a task is called the start event. The period from the start event to the trigger event constitutes one work unit. Specific examples of event sets will be described later.

[0022] The video generation unit 2 generates video based on the result of the viewpoint determination unit 3. The video generation unit 2 may also add the time the event occurred to the video it generates. This time can be used to create the seek bar described later or to merge it with other videos.

[0023] As explained above, the video generation device 100 of the second disclosure determines the viewpoint based on whether or not an event (trigger event and event set) associated with the work has occurred. Based on the determined viewpoint, the video generation device 100 generates a simulated video. By switching viewpoints, it is possible to generate a video with high visibility.

[0024] [Specific examples of the second disclosure] The following describes a specific example of the video generation device 100 of the second disclosure. In this example, the events are limited to operator operation signals and screen displays.

[0025] Refer to Figures 6, 11, and 12 to explain the switching of viewpoints in a numerical control device that controls a machine tool. As a prerequisite, the viewpoint determination unit 3 searches for trigger events from the data acquired by the data acquisition unit 1. The viewpoint determination unit 3 is assumed to have detected the trigger event "Pressing the cycle start button". Figure 6 is an example of an event set. In Figure 6, the trigger event "Pressing the cycle start button" is linked to an event set consisting of three events: "Pressing the button to change NC mode to MEM", "Displaying the automatic operation screen", and "Pressing the cycle start button".

[0026] Of the three events, "pressing the button to change NC mode to MEM" is the start event. If the events "display of the automatic operation screen" and "pressing the cycle start button" are executed in order after the start event, it indicates that the "start of machining" work unit has been executed.

[0027] The viewpoint determination unit 3 detects "pressing the cycle start button" and "pressing the button to change the NC mode to MEM" from the operation signals. "Display of the automatic operation screen" can be detected from the "screen ID" included in the history of operation signals. The "screen ID" is an ID assigned to each screen. The display of the "automatic operation screen" can be detected from the "screen ID". The viewpoint determination unit 3 can determine that the operation "start machining" has been performed if the events are executed in the order defined in the event set. In the explanation above, the order in which events occur is fixed, but it is not necessary to change the order.

[0028] As preparation before starting machining, the tool is moved to the machining start position at a rapid traverse. When machining using a new machining program that has not been used before, the tool is generally paused at the machining start position or decelerated by overriding the feed rate on the industrial machine's control panel. This is because if the distance between the workpiece and the tool at the machining start position is the expected distance, it can be determined that the "machining program" created by the programmer and the "tool offset" and "workpiece coordinate settings" set at the machining site are correct. In pre-machining operations, video footage that clearly shows the distance between the workpiece and the tool is useful.

[0029] The appropriate viewpoint varies depending on the type of machine tool and the direction of tool movement. Figure 7 shows an example of a lathe. In a lathe, the Z-axis is typically parallel to the spindle's rotation axis, and in a right-hand Cartesian coordinate system, the Z-axis is the middle finger and the X-axis is the direction of the thumb. To check the distance between the tool and the workpiece before starting cutting, a viewpoint perpendicular to the XZ plane is best. Figure 8 shows an image generated from a viewpoint perpendicular to the XZ plane. Switching viewpoints makes it easier to visualize the distance between the workpiece and the tool. Note that the image in Figure 8 is a two-dimensional image based on the results of the viewpoint determination unit 3.

[0030] Figure 9 shows an example of a machining center. Similarly, in a machining center, the tool moves to the cutting start position at rapid traverse and then "pauses" or "decelerates." In this example, the tool, once cutting has begun, moves parallel to the X-axis of the workpiece coordinate system. In this example, viewing the tool from a viewpoint perpendicular to the XZ plane makes it easier to check the distance between the tool and the workpiece. Figure 10 shows an image generated from a viewpoint perpendicular to the XZ plane. Switching viewpoints makes it easier to visualize the distance between the workpiece and the tool. Note that the image in Figure 10 is a two-dimensional image based on the results of the viewpoint determination unit 3. The viewpoint determination unit 3 may analyze the machining program and determine the viewpoint based on the direction of tool movement.

[0031] Figure 11 shows the trigger event and event set for workpiece measurement (end face). The trigger event for workpiece measurement (end face) is "Setting workpiece coordinate data". The trigger event "Setting workpiece coordinate data" is linked to an event set consisting of three events: "Display workpiece measurement screen", "Select workpiece measurement (end face) on the workpiece measurement screen", and "Set workpiece coordinate data".

[0032] Of the three events, "Displaying the work measurement screen" is the start event. If the events "Select work measurement (end face) on the work measurement screen" and "Setting work coordinate data" are executed in order from the start event, it indicates that the work unit "Starting work measurement (end face)" has been executed. The viewpoint determination unit 3 searches the data acquired by the data acquisition unit 1 and detects data that indicates the occurrence of an event set.

[0033] Next, we will explain how to search for events by working backward in time from a trigger event and identify the unit of work. Figure 12 shows the trigger event and event set for workpiece measurement (hole spacing measurement). The trigger event in Figure 12 is "Setting workpiece coordinate data". The trigger event "Setting workpiece coordinate data" is linked to an event set consisting of six events: "Replacing with touch probe", "Displaying workpiece coordinate measurement screen", "Selecting hole spacing measurement", "Axis operation by jog", "Touch probe contacts workpiece", and "Setting workpiece coordinate data".

[0034] The trigger event "Data setting for work coordinates" in Figure 12 is the same trigger event as the work unit "Work measurement (end face)" in Figure 11. In other words, one trigger event may correspond to multiple event sets. In such cases, searching for events by working backward in time from the trigger event will eventually converge to a single event set (work unit).

[0035] Figure 13 shows a specific example. When a trigger event called "Work coordinate data setting" is detected, the system searches for the preceding event, "Touch probe touches workpiece." If the event "Touch probe touches workpiece" exists, the system searches for the event immediately preceding it, "Axis operation by jog." If the event "Touch probe touches workpiece" does not exist, the system searches for events included in other event sets (for example, Figure 11). The system searches for events by working backward in time, and when it reaches the starting event, it determines the work unit corresponding to the event set.

[0036] Once the work unit is determined, the viewpoint is determined. Figure 14 shows the relationship between the work unit and the viewpoint. In the "Start Machining" operation (machining center), the viewpoint is determined so that the workpiece and tool are viewed from the side, with the Z-axis as vertical. In the "Workpiece Measurement (End Face)" operation, the viewpoint is determined so that the workpiece and tool are viewed from the side, with the Z-axis as vertical. In the "Workpiece Measurement (Hole Distance Measurement)" operation, the viewpoint is determined so that the workpiece is viewed from above, with the Z-axis as vertical. The viewpoint can be defined by its direction and distance from the object. For example, the viewpoint may be selected from predefined directions and distances relative to the tool and workpiece. Alternatively, the viewpoint may be automatically determined by calculating the direction and distance that avoids obstacles between the viewpoint and the object through simulation.

[0037] By arranging the work units on a time axis, a seek bar can be created for the video, as shown in Figure 15. Creating a seek bar allows the overall workflow to be seen, improving the visibility of the data.

[0038] Furthermore, it is possible to create a video seek bar by placing the event occurrence times on the timeline. By creating a seek bar, the workflow can be visually understood, and the work can be confirmed not only visually but also in text, improving the visibility of the data.

[0039] [Third Disclosure] In the third disclosure, the video from the display screen of the numerical control device and the video generated by the video generation unit 2 are displayed side by side. Figure 16 is an example of the video from the display screen of the numerical control device and the video generated by the video generation unit 2 being displayed side by side. The video generation device 100 merges the video from the display screen of the numerical control device and the generated video by time, and generates a single video by combining the two time-synchronized videos. In the example shown in Figure 16, the video from the display screen of the numerical control device and the video generated by the video generation unit 2 are displayed side by side. The video from the display screen may be captured using a camera instead of being captured by a camera. By displaying the video from the display screen and the simulation video side by side, the operator's actions and the status of the industrial machine can be checked simultaneously.

[0040] The hardware configuration of the video generation device 100 to which this disclosure is applied will be described below. Figure 17 is a hardware configuration diagram of the video generation device 100. As shown in Figure 17, the video generation device 100 includes a CPU 111 that controls the video generation device 100 as a whole, a ROM 112 that stores programs and data, and a RAM 113 for temporarily expanding data. The CPU 111 reads the system program stored in the ROM 112 via the bus and controls the entire video generation device 100 according to the system program.

[0041] The non-volatile memory 114 is backed up, for example, by a battery (not shown), so that its memory state is maintained even when the power to the video generation device 100 is turned off. Various data such as programs read from external devices via interfaces 115, 118, and 119, and user operations input via the input unit 71 are stored in the non-volatile memory 114.

[0042] Interface 115 mediates the connection between the video generation device 100 and external devices such as adapters. Programs and various parameters are read from the external device. [Explanation of Symbols]

[0043] 100 Video Generation Devices 1. Data Acquisition Unit 2. Video Generation Unit 3. Viewpoint determination unit 7. Data history storage unit 8 Event Set Memory Unit 111 CPU 113 RAM 114 Non-volatile memory

Claims

1. A data acquisition unit that acquires data indicating the mechanical status of industrial machinery, A viewpoint determination unit searches for data indicating the status of the industrial machine, detects trigger events indicating operations performed by the industrial machine, and determines a viewpoint corresponding to the trigger event. A video generation unit generates a simulated video showing the operation of the industrial machine based on the viewpoint determined by the viewpoint determination unit, A video generation device equipped with the following features.

2. The data acquisition unit includes a data history storage unit that stores the history of the data acquired by the data acquisition unit, The video generation apparatus according to claim 1, wherein the viewpoint determination unit stores an event set which is a combination of multiple events, and when it detects the trigger event, it searches whether or not data indicating the occurrence of an event included in the event set exists in the data history storage unit, and if data indicating the occurrence of an event included in the event set exists in the data history storage unit, it determines a work unit corresponding to the event set and determines a viewpoint corresponding to the work unit.

3. A data acquisition unit that acquires data indicating the mechanical state of an industrial machine, A viewpoint determination unit searches for data indicating the status of the industrial machine, detects trigger events indicating operations performed by the industrial machine, and determines a viewpoint corresponding to the trigger event. A video generation unit generates an image of the industrial machine based on the viewpoint determined by the viewpoint determination unit, Equipped with, The viewpoint determination unit is a video generation device that searches for data indicating the occurrence of events included in the event set, starting from the trigger event and working backward in time.

4. A data acquisition unit that acquires data indicating the mechanical state of an industrial machine, A viewpoint determination unit searches for data indicating the status of the industrial machine, detects trigger events indicating operations performed by the industrial machine, and determines a viewpoint corresponding to the trigger event. A video generation unit generates an image of the industrial machine based on the viewpoint determined by the viewpoint determination unit, The data acquisition unit includes a data history storage unit that stores the history of the data acquired by the data acquisition unit, The viewpoint determination unit stores an event set which is a combination of multiple events, and when it detects the trigger event, it searches the data history storage unit to see if data indicating the occurrence of an event included in the event set exists, and if data indicating the occurrence of an event included in the event set exists in the data history storage unit, it determines the work unit corresponding to the event set and determines the viewpoint corresponding to the work unit. A video generation device that adds the time of occurrence of the event to the video generated by the video generation unit.

5. A data acquisition unit that acquires data indicating the mechanical state of an industrial machine, A viewpoint determination unit searches for data indicating the status of the industrial machine, detects trigger events indicating operations performed by the industrial machine, and determines a viewpoint corresponding to the trigger event. A video generation unit generates an image of the industrial machine based on the viewpoint determined by the viewpoint determination unit, The data acquisition unit includes a data history storage unit that stores the history of the data acquired by the data acquisition unit, The viewpoint determination unit stores an event set which is a combination of multiple events, and when it detects the trigger event, it searches the data history storage unit to see if data indicating the occurrence of an event included in the event set exists, and if data indicating the occurrence of an event included in the event set exists in the data history storage unit, it determines the work unit corresponding to the event set and determines the viewpoint corresponding to the work unit. A video generation device that creates a seek bar indicating the occurrence times of events included in the aforementioned event set.

6. A data acquisition unit that acquires data indicating the mechanical state of an industrial machine, A viewpoint determination unit searches for data indicating the status of the industrial machine, detects trigger events indicating operations performed by the industrial machine, and determines a viewpoint corresponding to the trigger event. A video generation unit generates an image of the industrial machine based on the viewpoint determined by the viewpoint determination unit, The data acquisition unit includes a data history storage unit that stores the history of the data acquired by the data acquisition unit, The viewpoint determination unit stores an event set which is a combination of multiple events, and when it detects the trigger event, it searches the data history storage unit to see if data indicating the occurrence of an event included in the event set exists, and if data indicating the occurrence of an event included in the event set exists in the data history storage unit, it determines the work unit corresponding to the event set and determines the viewpoint corresponding to the work unit. A video generation device that creates a seek bar indicating the time of occurrence of the aforementioned work unit.

7. One or more processors execute, We acquire data that shows the status of industrial machinery. The system searches for data indicating the status of the industrial machine, detects trigger events indicating operations performed by the industrial machine, and determines the viewpoint corresponding to the trigger event. A simulation video is generated showing the operation of the industrial machine based on the determination of the viewpoint. A storage medium for storing instructions that the processor can read.

8. One or more processors execute, We acquire data that shows the status of industrial machinery. By searching for data indicating the status of the industrial machine, detecting trigger events indicating operations performed by the industrial machine, and searching for data indicating the occurrence of events included in the event set while working backward in time from the trigger event, the perspective corresponding to the trigger event is determined. To generate an image based on the aforementioned viewpoint determination, A storage medium for storing instructions that the processor can read.

Images