VIDEO GENERATION DEVICE AND COMPUTER-READABLE STORAGE MEDIUM

Abstract

A video generation device is provided that produces a recorded video of an industrial machine, obtains data indicating a state of the industrial machine, searches through data indicating a state of the industrial machine, detects a trigger event indicating an activity performed by the industrial machine, determines a viewpoint corresponding to the trigger event, and generates a video based on the determination of the viewpoint.

Description

TECHNICAL FIELD

[0001] The present invention relates to a video generating apparatus and a computer-readable storage medium. GENERAL STATE OF THE ART

[0002] Currently, there is a technology that monitors the conditions of an industrial machine and uses the monitoring results to improve control performance or for machine maintenance. This technology collects data related to the industrial machine and analyzes the collected data to track the conditions of the industrial machine. The data is used to prevent malfunctions in the industrial machine, verify product quality, and improve production management.

[0003] An industrial machine and its peripheral devices, monitored by sensors, are recorded in a data collection device called a data logger. Several data loggers are installed in a factory to record the status of the machines in the factory.

[0004] Programmable logic controllers (PLCs) for controlling machine operations and numerical control devices (numerical controllers) for controlling machine tools also have internal recording units to store control information, status information, etc.

[0005] Items of the collected data are converted into a form recognizable by an operator. Patent Literature Example 1 discloses that "a time loading means, after acquiring control data from a controller, image data from a camera, and sound data from a device to be controlled, generates control data with added time, image data with added time, and sound data with added time by adding the times at which the data were respectively acquired, and stores these data items in a temporary file."Then, a log data generating means reads a set of control data with time added, image data with time added, and sound data with time added, each having the closest time, from the items of control data with time added, image data with time added, and sound data with time added stored in the temporary file, and generates log data including times, control data, image data, and sound data, whereby the data can be reproduced while being synchronized." According to Patent Document 1, the images, sound, and log of the device to be controlled are reproduced while being synchronized with time.

[0006] Patent Literature Example 2 discloses that "a camera is connected to the controller to capture images of a robot, and image data obtained by capturing images of the robot and showing the robot's behavior is supplied to the controller in the form of electrical signals," and that "when an error (event) occurs, an error record related to the error is separately generated from an operation log." According to Patent Literature Example 2, when the event occurs during image capture, the event is recorded in conjunction with the images. PRIOR ART DOCUMENTS PATENT LITERATURE EXAMPLES Patent Literature Example 1: Japanese Patent Laid-Open No. 2001-125612 Patent Literature Example 2: Japanese Patent Laid-Open No. 2019-150923 SUMMARY OF THE INVENTION PROBLEMS THAT THE INVENTION INTENDED TO SOLVE

[0007] Patent Literature Example 1 displays images and logs simultaneously, while Patent Literature Example 2 records the log (data) at the time the event occurred. The log and images during the event can clarify the circumstances at the time the fault occurred and help identify the cause of the fault. If images with good visibility are generated, these images can be used to improve the maintenance and productivity of industrial machinery.

[0008] Therefore, in the field of data management of industrial machines, there is a problem of improving detectability. MEANS TO SOLVE THE PROBLEM

[0009] One aspect of the present disclosure is a video generation device comprising a data acquisition unit for acquiring data indicating a mechanical state of an industrial machine; a viewpoint determination unit for searching data indicating the state of the industrial machine and detecting a trigger event including a task performed by the industrial machine to determine a viewpoint corresponding to the trigger event; and a video generation unit for generating a video of the industrial machine based on the viewpoint determined by the viewpoint determination unit.

[0010] One aspect of the present disclosure is a storage medium storing instructions readable by one or more processors, wherein the one or more processors execute the instructions to obtain data indicative of a state of an industrial machine; search data indicative of the state of the industrial machine; detect a trigger event indicative of a task performed by the industrial machine and determine a viewpoint corresponding to the trigger event; and generate a video based on the determination regarding the viewpoint. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0011] One aspect of the present invention can improve image recognizability. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows an application area of ​​a video generating device; Fig. 2 is a block diagram showing a video generating apparatus according to a first disclosure; Fig. 3 shows a conceptual model of a viewpoint determining unit in the first disclosure; Fig. 4 is a block diagram showing a video generating apparatus according to a second disclosure; Fig. 5 shows a conceptual model of a viewpoint determining unit in the second disclosure; Fig. 6 shows an example of an event set; Fig. Figure 7 shows an example of a task before starting a machining operation on a lathe; Fig. 8 shows an example of a video generated by a video generating unit; Fig. Figure 9 shows an example of a task before starting machining in a machining center; Fig. 10 shows an example of the video generated by the video generating unit; Fig. 11 shows a trigger event and an event set for the workpiece measurement task (end face); Fig. 12 shows a trigger event and an event set for a workpiece measurement (hole spacing measurement); Fig. 13 shows a method for searching the event set; Fig. 14 shows a relationship between a work unit and a viewpoint; Fig. 15 shows an example of a search bar with the work unit associated with the timeline; Fig. Figure 16 shows an example that combines two videos into one; Fig. 17 shows a hardware structure of the video generating apparatus. WAYS FOR CARRYING OUT THE INVENTION

[0012] A video generation device 100 employing the present disclosure will now be described. The video generation device 100 is configured to acquire data indicating a state of an industrial machine installed in a factory. The video generation device 100 uses previously collected data to determine a viewpoint of a video showing the operation of the industrial machine. Target industrial machines include, but are not limited to, machine tools, numerical controllers, and robots.

[0013] The video generation device 100 is applied to an information processing device such as a personal computer (PC), a server, and a mobile terminal. More specifically, the video generation device 100 is configured as shown in Fig. 1, for example, a PC used by a factory administrator, a server in the cloud, a PC used by an operator of a machine manufacturer, a control machine manufacturer, or the like, and a mobile terminal used by the factory administrator are applied, but are not limited to these. Videos generated by the video generation device 100 are used for industrial machine control (e.g., maintenance, production control).

[0014] The video generating apparatus 100 can also be applied to an information processing apparatus such as a numerical controller and a programmable logic controller (PLC). First Revelation

[0015] Fig. Figure 2 is a block diagram of the video generation device 100 according to a first disclosure. The video generation device 100 includes a data acquisition unit 1, a video generation unit 2, and a viewpoint determination unit 3.

[0016] The data acquisition unit 1 is configured to acquire data related to industrial machines. The data acquired by the data acquisition unit 1 includes state information of the industrial machines, operation input information, control information of the industrial machines, and the like. The present disclosure shows a numerical controller and a machine tool as examples of the industrial machines. The state information of the industrial machines includes, for example, the operation mode of the numerical controller, a tool offset, alarms, times, the number of machined parts, modal information, position information (the coordinate system), power consumption in the machine tool, the load of a servo motor, or measurement data obtained from sensors (such as temperature and pressure).The operation input data is operation input made by an operator on a touch panel, button, dial, handle, and the like.

[0017] The video generation unit 2 is configured to generate a simulation video showing the operation of the industrial machinery using a conventional simulation technique. A viewpoint of the simulation video can be changed. Although the present disclosure shows an example of generating a simulation video, the video generated by the video generation unit 2 may be a two-dimensional image based on a result obtained by the viewpoint determination unit instead of the simulation video.

[0018] The viewpoint determining unit 3 is configured to determine a viewpoint of the simulation video generated by the video generating unit 2. Fig. Figure 3 shows a conceptual model of the viewpoint determination unit 3. The viewpoint determination unit 3 monitors the data acquired by the data acquisition unit 1 and then detects a trigger event. The viewpoint determination unit 3 determines a viewpoint corresponding to the detected trigger event. A concrete example of the trigger event will be described later.

[0019] The video generation unit 2 generates a simulation video of the viewpoint determined by the viewpoint determination unit 3. Instead of the simulation video, a two-dimensional image can be generated based on the result obtained by the viewpoint determination unit 2. This generates a recorded video of a viewpoint suitable for testing the operation of the industrial machine.

[0020] As described above, the video generation device 100 of the first disclosure acquires the data related to the industrial machine, detects the trigger event, and generates the recorded video from the viewpoint suitable for checking the operation of the industrial machine. According to this disclosure, the viewpoint of the video can be changed according to the task of the industrial machine, thus generating a video with high recognizability. Second Revelation

[0021] Fig. 4 is a block diagram of the video generation unit 100 according to a second disclosure. The video generation unit 100 includes a data acquisition unit 1, a video generation unit 2, a data history storage unit 7, a viewpoint determination unit 3, and an event set storage unit 8.

[0022] The video generation device 100 of the second disclosure includes the data history storage unit 7 and the event set storage unit 8. The data history storage unit 7 is configured to store data histories acquired by the data acquisition unit 1. The event set storage unit 8 is configured to store a combination of data (an event set) that occurs when tasks are executed.

[0023] Fig. Figure 5 shows a conceptual model of the viewpoint determination unit 3. The viewpoint determination unit 3 searches data acquired by the data acquisition unit 1 and detects a trigger event. The viewpoint determination unit 3 reads an event record corresponding to the detected trigger event and then searches history data stored in the data history storage unit 8 to determine whether the event record corresponding to the detected trigger event has occurred in the past. If any event record corresponding to the detected trigger event has occurred, it is determined that a task has been performed in the past.

[0024] The event set corresponds to work units from the beginning to the end of a task. In this disclosure, an event that occurs at the beginning of the task is referred to as a start event. A work unit ranges from the start event to a trigger event. A concrete example of the event set will be described later.

[0025] The video generation unit 2 generates a video based on the result obtained from the viewpoint determination unit 3.

[0026] The time at which the event occurred can be added to the video generated by the video generation unit 2. This time can be used to generate a seek bar or merge with other videos, as described later.

[0027] As described above, the video generation device 100 of the second disclosure determines the viewpoint according to whether the events (trigger event and event set) associated with the task have occurred. The video generation device 100 generates a simulation video based on the determined viewpoint. By changing the viewpoints, a video with high recognizability can be generated. Embodiment of the second disclosure

[0028] Now, an embodiment of the video generating device 100 of the second disclosure will be described. This embodiment will be described by limiting the events to operation signals from an operator and screen displays.

[0029] With reference to Fig. 6, Fig. 11 and Fig. 12, descriptions will be made regarding the change of the viewpoint in a numerical control system that controls machine tools.

[0030] The descriptions are based on the assumption that the viewpoint determination unit 3 finds a trigger event from data acquired by the data acquisition unit 1.

[0031] It is assumed that the viewpoint determination unit 3 detects a trigger event “pressing the cycle start button”. Fig. 6 shows an example of an event set. In Fig. 6, the trigger event “pressing the cycle start button” is bound to an event set consisting of three events, ie, “pressing the button to change the NC mode to MEM”, “displaying the automatic operation screen”, and “pressing the cycle start button”.

[0032] Among the three events, the "pressing the key to change the NC mode to MEM" event is the start event. When the "displaying the automatic operation screen" and "pressing the cycle start key" events are executed sequentially after the start event, it can be determined that a "machining start" work unit is being executed.

[0033] The viewpoint determination unit 3 detects the events "pressing the cycle start button" and "pressing the button to change the NC mode to MEM" from an operation signal. The event "displaying the automatic operation screen" can be detected from a "screen identifier" object included in the history of the operation signal. The "screen identifier" object is an identifier assigned to each screen. From the "screen identifier" object, it can be detected that an "automatic operation screen" object is being displayed. When the events are executed in the order defined by the event set, the viewpoint determination unit 3 determines that the "machining start" task is being executed.

[0034] The order in which the events occur is fixed in the description above, but the events may occur in any particular order.

[0035] In preparation before starting machining, a tool is moved to a machining start position at rapid traverse. In a case where machining is performed using a new machining program that has not been used before, the tool is usually temporarily stopped at the machining start position or decelerated by a feedrate override operation on an operation panel of the industrial machine. This is because if the distance between a workpiece and the tool at the machining start position is the expected distance, it can be determined that the "machining program" created by a programmer and the setting values ​​of the "tool offset" and "workpiece coordinate setting" set at a machining location are correct.

[0036] In the task that is carried out before machining begins, a video that makes it easier to see the distance between the workpiece and the tool is useful.

[0037] The appropriate viewpoint changes depend on the type of machine tool and the direction of the tool movement.

[0038] Fig. 7 shows an example of a lathe.

[0039] On a lathe, the direction leading to the spindle's rotational axis is generally the z-axis, and the direction of the thumb in a right-handed Cartesian coordinate system, when the z-axis is considered the middle finger, is the x-axis. To check the distance between the tool and the workpiece before starting cutting, the viewpoint should be orthogonal to the XZ plane.

[0040] Fig. Figure 8 shows a video generated from the viewpoint orthogonal to the XZ plane. By changing the viewpoint, the distance between the workpiece and the tool is easily visible. The video, which is shown in Fig. 8 is a two-dimensional image based on the result obtained by the viewpoint determining unit 3.

[0041] Fig. 9 shows an example of a machining center.

[0042] In the machining center, the tool is also moved at rapid traverse to a cutting start position and then "temporarily stopped" or "decelerated." In this embodiment, the tool that begins cutting is moved parallel to the x-axis of the workpiece's coordinate system. In this example, the distance between the tool and the workpiece can be easily checked from a viewpoint orthogonal to the XZ plane.

[0043] Fig. Figure 10 shows a video generated at the viewpoint orthogonal to the XZ plane. By changing the viewpoint, the distance between the workpiece and the tool can be easily determined. The video, which is shown in Fig. 10 is a two-dimensional image generated based on the result obtained by the viewpoint determining unit 3.

[0044] The viewpoint determining unit 3 can analyze the machining program to determine the viewpoint based on the direction of the tool movement.

[0045] Fig. Figure 11 shows a trigger event and event set for the workpiece measurement (end face) task. The trigger event for the workpiece measurement (end face) task is "workpiece coordinate data setting." The trigger event "workpiece coordinate data setting" is bound to an event set consisting of three events: "displaying the workpiece measurement screen," "selecting the workpiece measurement (end face) on the workpiece measurement screen," and "tool coordinate data setting."

[0046] Among the three events, the "Workpiece Measurement Screen Display" event is the start event. If the "Workpiece Measurement (End Face) Selection on the Workpiece Measurement Screen Display" and "Tool Coordinate Data Setting" events are executed sequentially after the start event, it can be recognized that a "Workpiece Measurement (End Face) Start" work item is being executed.

[0047] The viewpoint determination unit 3 searches the data acquired by the data acquisition unit 1 and detects data indicating the occurrence of the event set.

[0048] Next, descriptions will be given of a method for identifying the work unit by searching for the events going back in time starting with the trigger event.

[0049] Fig. Figure 12 shows a trigger event and an event set for workpiece measurement (hole spacing measurement). The trigger event in Fig. 12 is the "Tool Coordinate Data Setting" event. The "Tool Coordinate Data Setting" trigger event is bound to an event set consisting of six events: "Exchange with a probe," "Display of the workpiece coordinate measurement screen," "Select hole pitch measurement," "Actuate the axis in jog mode," "Contact of the probe with the workpiece," and "Tool Coordinate Data Setting."

[0050] The trigger event “Data setting for the tool coordinates” in Fig. 12 is the trigger event of the work unit “Workpiece measurement (end face)” in Fig. 11. This means that multiple sets of events can correspond to a single trigger event. In such a case, the events will eventually converge into a single set of events (a work unit) if the events are searched for by working backward in time, starting with the trigger event.

[0051] Fig. Figure 13 shows a concrete example. If the trigger event "Data setting for the tool coordinates" is detected, the immediately preceding event "Contact of the measuring probe with the workpiece" is searched for. If the event "Contact of the measuring probe with the workpiece" is present, the immediately preceding event "Actuating the axis in jog mode" is searched for. If the event "Actuating the axis in jog mode" is not present, an event that is contained in another event set (e.g., in Fig. 11). If the event is searched for by going back in time and reaching the initial event, a work unit corresponding to the event set is determined.

[0052] Once the work unit is determined, the focus point is determined. Fig. Figure 14 shows the relationship between the work unit and the viewpoint. For the "Start of Machining (Machining Center)" task, the viewpoint is determined so that the workpiece and tool are viewed from the side with the z-axis as the vertical axis. For the "Workpiece Measurement (End Face)" task, the viewpoint is determined so that the workpiece and tool are viewed from the side with the z-axis as the vertical axis. For the "Workpiece Measurement (Hole Distance Measurement)" task, the viewpoint is determined so that the workpiece and tool are viewed from above with the z-axis as the vertical axis.

[0053] The viewpoint can be defined according to the direction and distance relative to an object. For example, the viewpoint can be selected from predefined directions and distances seen from a tool and a workpiece. Alternatively, a direction and distance can be calculated through simulation without obstructing the view of the object, and the viewpoint can be determined automatically.

[0054] If the work unit is arranged on a time axis, as in Fig. As shown in Figure 15, a search bar can be created for a video. Creating the search bar allows for verification of the entire workflow, thereby increasing the recognizability of the data.

[0055] A seek bar for the video can also be created by placing the time of an event's occurrence on the timeline. Creating a seek bar not only allows for a visual representation of the workflow but also allows the task to be reviewed not only through the video but also through text. This increases the recognizability of the data. Third Revelation

[0056] According to a third disclosure, a video on the display screen of the numerical controller and a video generated by the video generation unit 2 are displayed side by side. Fig. 16 shows an example in which the video on the display screen of the numerical controller and the video generated by the video generation unit 2 are displayed side by side. The video generation device 100 merges the video on the display screen of the numerical controller with the generated video according to the timing, and generates one video by combining two videos synchronized based on the timing.

[0057] The example in Fig. Figure 16 shows the video on the display screen of the numerical control system and the video generated by the video generation unit 2 side by side. The video on the display screen can also be recorded instead of being photographed by a camera. By displaying the video on the display screen and a simulation video side by side, the operator's actions and the conditions of the industrial machine can be checked simultaneously.

[0058] Now, a hardware structure of the video generating apparatus 100 employing the present disclosure will be described. Fig. 17 shows a hardware structure of the video generating device 100. The video generating device 100 has, as in Fig. As shown in Figure 17, the CPU 111 includes a central processing unit (CPU) 111 for controlling the entire video generation device 100, a read-only memory (ROM) 112 for storing programs and data items, and a random access memory (RAM) 113 for temporarily storing data. The CPU 111 reads a system program stored in the ROM 112 via a bus and controls the entire video generation device 100 according to the system program.

[0059] A non-volatile memory 114 is backed up, for example, by a battery so that memory states can be retained even when a power supply of the video generation device 100 is turned off. The non-volatile memory 114 is configured to store, for example, programs read from external devices via interfaces 115, 118, 119 and various data input by user operation via an input unit 71.

[0060] The interface 115 mediates the connection between the video generation device 100 and external devices such as adapters. Programs, various parameters, etc., are read from the external devices. LIST OF REFERENCE SYMBOLS 100 video generating device 1 data acquisition unit 2 Video generation unit 3 Viewpoint determination unit 7 Data history storage unit 8 Event record storage unit 111 CPU 113 RAM 114 non-volatile memory QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2001

[0006] JP 2019-150923

[0006]

Claims

[1] Video generating device, comprising a data acquisition unit for acquiring data indicating a mechanical state of an industrial machine; a viewpoint determination unit for searching data indicating a state of the industrial machine and detecting a trigger event including a task performed by the industrial machine to determine a viewpoint corresponding to the trigger event; and a video generating unit for generating a video of the industrial machine based on the viewpoint determined by the viewpoint determining unit. [2] The video generating apparatus according to claim 1, comprising a data history storage unit for storing a history of the data acquired by the data acquiring unit, wherein the viewpoint determining unit stores an event set formed by a combination of a plurality of events, and upon detection of the trigger event, checks the data history storage unit for the presence of data indicating the occurrence of the events included in the event set, and upon the presence of the data indicating the occurrence of the events included in the event set, determines in the data history storage unit a work unit corresponding to the event set, and thereby determines a viewpoint corresponding to the work unit. [3] The video generating apparatus according to claim 1, wherein the viewpoint determining unit searches data indicating the occurrence of the events included in the event set by going back in time starting from the trigger event. [4] The video generating apparatus according to claim 2, wherein times of occurrence of the events are added to the video generated by the video generating unit. [5] A video generating apparatus according to claim 2, wherein a search bar indicating the times of occurrence of the events included in the event set is generated. [6] The video generating apparatus according to claim 2, wherein a seek bar indicating a time of occurrence of the work unit is generated. [7] The video generating device according to claim 1, wherein the video obtaining unit is a video generating unit for generating a two-dimensional video based on a simulation video indicating an operation of the industrial machine or a result obtained by the viewpoint determining unit. [8] Storage medium that stores instructions that can be read by one or more processors, wherein the one or more processors execute the instructions to obtain data indicative of a state of an industrial machine; to search data indicating the state of the industrial machine, detect a trigger event indicating a task performed by the industrial machine, and determine a viewpoint corresponding to the trigger event; and to generate a video based on the determination of the viewpoint.

Images