Image generation device and computer-readable storage medium

Abstract

This video generation device acquires a video of an industrial machine, said video including a display screen of the industrial machine, acquires data indicating a machine condition of the industrial machine, determines the type of display screen, extracts a portion of the data on the basis of the type of display screen and the industrial machine condition information, and generates a video simultaneously displaying the extracted data and the acquired video.

Description

[Technical Field]

[0001] The present invention relates to an image generation device and a computer-readable storage medium. [Background technology]

[0002] Currently, there are technologies that record the status of industrial machinery and use them to improve control efficiency and maintain the machinery. Such technologies collect and analyze data related to industrial machinery to understand the status of the machinery, prevent breakdowns, check product quality, and improve production management.

[0003] The information detected by industrial machinery and sensors installed around it is recorded in a data collection device called a data logger. Many data loggers are installed in factories to record the status of the machinery within the factory. In addition, PLCs (Programmable Logic Controllers), which control the operation sequence of machines, and numerical controllers, which control machine tools, also have internal recording units that store control information and status information.

[0004] The collected data is converted into a format that is easy for the operator to view. Patent Document 1 states, "When the time loading means acquires control data acquired from the control device, image data acquired from the camera, and sound data acquired from the controlled device, it creates time-added control data, time-added image data, and time-added sound data by adding the time at which each data was acquired, and stores these in a temporary file. Then, the log data creation means reads the most recent set of time-added control data, time-added image data, and time-added sound data from the time-added control data, time-added image data, and time-added sound data stored in the temporary file, and creates log data containing the time, control data, image data, and sound data, thereby enabling synchronized playback of each data." According to Patent Document 1, the images, sounds, and logs of the controlled device are played back in time synchronization.

[0005] Patent Document 2 states that "a camera that takes pictures of the robot is connected to the controller, and image data showing the robot's behavior obtained by the image capture is supplied to the controller in the form of an electrical signal," and "when an error (event) occurs, an error record related to the error is created separately from the operation log." According to Patent Document 2, when an event occurs while an image is being captured, the event and the image are recorded in association with each other. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2001-125612 [Patent Document 2] Japanese Patent Application Publication No. 2019-150923 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, an image and a log are displayed simultaneously, while in Patent Document 2, a log (data) is recorded when an event occurs. If there is a log and an image when an error occurs, the situation at the time of the error can be clarified and the cause of the error can be identified.

[0008] In the field of industrial machinery data management, there is a challenge in improving visibility. [Means for solving the problem]

[0009] An image generation device according to one aspect of the present disclosure includes an image acquisition unit that acquires an image of the industrial machine, including a display screen of the industrial machine; a data acquisition unit that acquires data indicating the mechanical status of the industrial machine; a screen discrimination unit that discriminates the type of display screen; a data extraction unit that extracts a portion of the data based on the type of display screen and status information of the industrial machine; and an image generation unit that generates an image that simultaneously displays the data extracted by the data extraction unit and the image acquired by the data acquisition unit. A storage medium that is one aspect of the present disclosure stores processor-readable instructions that, when executed by one or more processors, acquire video of the industrial machine, including the display screen of the industrial machine, acquire data indicating the mechanical status of the industrial machine, determine the type of display screen, extract data to be used for generating the video based on the type of display screen and the status information of the industrial machine, and create a video that simultaneously displays the extracted data and the acquired video. [Effects of the Invention]

[0010] According to one embodiment of the present invention, the visibility of recorded video can be improved. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an application range of an image generation device. [Figure 2] FIG. 1 is a block diagram of a video generation device. [Figure 3] FIG. 1 is an external view of a machining center. [Figure 4] 10 is a table showing data extraction conditions. [Figure 5]10 is a flowchart showing a procedure for setting a tool compensation amount. [Figure 6] 10 is an example of a tool compensation screen. [Figure 7] FIG. 10 is a diagram illustrating machine coordinate values ​​of a tool measuring device. [Figure 8] FIG. 10 is a diagram illustrating the setting of work coordinates. [Figure 9] FIG. 2 is a diagram showing the positional relationship between a tool and a tool measuring instrument. [Figure 10] FIG. 10 is a diagram illustrating setting of a tool length. [Figure 11] FIG. 1 is a diagram showing an example of a conventional video. [Figure 12] FIG. 10 is a diagram illustrating an example of an image to which the present disclosure is applied. [Figure 13] FIG. 2 is a hardware configuration diagram of the video generation device. DETAILED DESCRIPTION OF THE INVENTION

[0012] The video generation device 100 to which the present disclosure is applied will be described below. The video generation device acquires video of industrial machines installed in a factory and data showing the status of the industrial machines. The video and data are stored in a data logger, PLC, internal and external servers, numerical control devices, etc. The video generation device 100 acquires the already stored data and generates recorded video showing the operation of the industrial machines. The recorded video is composed of video of the industrial machines and data explaining the status of the industrial machines.

[0013] The image generation device 100 is applied to information processing devices such as PCs (personal computers), servers, and mobile terminals. Specifically, as shown in Fig. 1, the image generation device 100 is applied to, but is not limited to, a PC used by a factory manager, a server on the cloud, a PC used by an operator of a machine manufacturer or a control machine manufacturer, and a mobile terminal used by a factory manager. The image generated by the image generation device 100 is used for managing industrial machinery (maintenance, production management, etc.).

[0014] 2 is a block diagram of the image generation device 100. The image generation device 100 includes an image acquisition unit 1, a data acquisition unit 2, a screen determination unit 3, an extraction condition storage unit 4, a data extraction unit 5, and an image generation unit 6.

[0015] The video acquisition unit 1 acquires video of machines and equipment in a factory from a data logger, internal and external servers, a PLC, a numerical control device, etc. In this disclosure, an example of acquiring video of a machining center in which a machine tool and a numerical control device are integrated will be described. FIG. 3 shows the exterior of the machining center. The machining center of this disclosure is equipped with two cameras. One camera captures an image of a workpiece attached to a table, and the other camera captures an image of an operation panel. It is also possible to capture only the buttons and dials of the operation panel, and not the display screen. The video captured by the video capture unit 1 has time information added to it, and can be merged with data (described later) representing the machine state. In the present disclosure, video includes both moving images and still images, and also includes an image in which a plurality of still images are displayed consecutively. In addition, in the present disclosure, an image of the interior of the machining center, the operation screen of the numerical control device, and the operation panel of the numerical control device are displayed side by side, but it is also possible to display only either the operation panel of the numerical control device or the operation screen of the numerical control device.

[0016] The data acquisition unit 2 acquires data related to the numerical control device and machine tool from data loggers, internal and external servers, PLCs, numerical control devices, etc. The data acquired by the data acquisition unit 2 includes status information of the numerical control device, operation input information, and control information of the machine tool. Status information of the numerical control device is set in the internal variables of the numerical control device, and includes the mode of the numerical control device, tool offset amount, alarms, time, number of machined parts, modal information, position information (coordinate system), and various signals. Operation input information includes input to a touch panel and input using physical buttons, dials, and handles. Control information of the machine tool includes physical quantities required for controlling the machine tool and status information of the machine tool. Control information of the machine tool includes power consumption, load on the servo motor, and data measured by sensors (temperature and pressure), etc.

[0017] The screen determination unit 3 determines the type of display screen based on the data acquired by the data acquisition unit 2 and the video acquired by the video acquisition unit 1. To determine the type of display screen based on the video, well-known image analysis is used. The type of screen being displayed may be determined by image analysis. Image analysis may also be used to determine information such as the button operated by the user, the type and position of the tool, the installation and position of the workpiece, and the start and end of processing. The data acquired by the data acquisition unit 2 includes status information of the numerical control device, operation input information of the numerical control device, control information of the machine tool, etc. The status information of the numerical control device includes mode information of the numerical control device. The screen discrimination unit 3 discriminates the type of display screen from the machine status and operation input of the numerical control device. The display screen can be discriminated by the screen ID.

[0018] The extraction condition storage unit 4 stores data extraction conditions. The data to be extracted is determined by the type of display screen and the state of the numerical control device (industrial machine).

[0019] An example of extraction conditions is shown in Figure 4. The table at the top of Figure 4 defines the extraction conditions for the type of display screen, and defines that when the type of display screen is the "Tool screen," "tool data change information" will not be extracted, but "data related to changes in the machine status other than tool data change information" will be extracted. Since "tool data change information" is displayed on the "Tool screen," only data that cannot be determined from the display screen is extracted. When the display screen type is "Work coordinate screen," it is defined that "information on changes to work coordinate data" is not extracted, but "data related to changes in the machine status other than information on changes to work coordinate data" is extracted. Since "information on changes to work coordinate data" is displayed on the "Work coordinate screen," data that cannot be determined from the display screen is extracted. It is defined that when the display screen is the "macro variable screen," "macro variable data change information" is not extracted, but "data related to changes in the machine status other than macro variable data change information" is extracted. Since the "macro variable screen" displays "macro variable data change information," data that cannot be determined from the display screen is extracted. The table at the bottom of Figure 4 defines the extraction conditions for status information. When the numerical control device is in "automatic operation," it defines that "coordinate value information" is not extracted, but rather "data related to changes in the machine's status other than coordinate value information" is extracted.

[0020] The data extraction unit 5 extracts data based on the extraction conditions stored in the extraction condition storage unit 4. The extracted data includes data that is always displayed in the generated video and data that is displayed only when the data value changes. In the case of data that is to be displayed at all times, the data is read out periodically. In the case of data that is displayed only when the data value changes, when a change (change) in the data is detected, the extraction conditions are referenced and it is determined whether or not to extract the data. The data extraction unit 5 adds a timestamp to the extracted data. The timestamp indicates the time when the data changed (changed).

[0021] As exemplified above, the extraction conditions are conditions for excluding information that overlaps with information provided on the display screen.

[0022] With reference to FIG. 5, data extraction when the tool length correction screen is displayed will be described. The data extraction unit 5 detects data instructing the start of setting the tool compensation amount, and determines that the tool compensation screen has been displayed in response to this instruction (step S1). FIG. 6 is an example of the tool compensation screen. The tool compensation screen in FIG. 6 displays a list of the tool number, tool shape compensation amount (tool length), wear compensation amount (tool length), tool shape compensation amount (tool diameter), and wear compensation amount (tool diameter), as well as relative coordinates, absolute coordinates, and machine coordinates. A compensation amount input button and a clear button are displayed at the bottom of the tool offset screen.

[0023] The data extraction unit 5 reads out the extraction conditions corresponding to the tool compensation screen (step S2).

[0024] The data extraction unit 5 searches for changes in the data included in the extraction conditions (step S3).

[0025] The data extraction unit 5 detects the setting (change) of the workpiece coordinate system (absolute coordinates) with the switch of the tool measuring device as the origin, based on the data acquired by the data acquisition unit 2 (step S4). When operating the control panel, keystrokes such as "↑ (up arrow)," "↓ (down arrow)," "← (left arrow)," and "→ (right arrow)" are used, but these changes in input signals can be determined from the video and are not extracted as display data. In the example of Figure 8, the machine coordinates "X: 300.0", "Y: 240.0", and "Z: -380.0" of the tool measuring device switch are set to the work coordinate origin "G55". The setting of the work coordinate origin "G55" can be determined from the image, so it is not extracted as display data.

[0026] The data extraction unit 5 detects the selection (change) of the tool to be measured (step S5). Data related to the tool cannot be determined from the tool correction screen in FIG. 6. Therefore, data related to the tool, such as "Tool change: T1 → T5", is extracted as display data. Note that if multiple tools are selected, the processes from step S4 to step S5 are repeated multiple times.

[0027] The data extraction unit 5 detects a change (switch) of the mode from the "memory mode" to the "handle mode" (step S6). "Memory mode" is a mode in which the numerical control device operates automatically according to a program, and "handle mode" is a mode in which the spindle is moved by operating the handle. The extraction condition for the mode is "always displayed." The data extraction unit 5 extracts data indicating the mode.

[0028] The data extraction unit 5 detects a change in the direction of operation of the handle (step S7). As a result of the manual operation, the tool is placed directly above the tool measuring device (see FIG. 9). Since the steering direction cannot be determined from the image, steering direction data, such as "HANDLE +Z", "HANDLE -Z", "HANDLE +X", and "HANDLE -X", is extracted as display data.

[0029] A tool is attached to the tip of the spindle. When the spindle is moved in the direction of the tool setter (see Figure 10), the tip of the tool comes into contact with the tool setter (step S8). Because the switch position of the tool setter is set to workpiece coordinate origin "G55," the value of the Z axis when the tool comes into contact with the tool setter, with the workpiece coordinate origin "G55" as the origin, indicates the tool length. In the example of Figure 10, the Z axis value is "110.000."

[0030] The data extraction unit 5 detects the setting (change) of the tool shape compensation amount (step S9). In the example of Fig. 10, the tool length compensation amount for tool number "002" is set to "110.000". The tool length compensation amount changes from "0.000" to "110.000", but this change can be determined from the image and is not extracted as display data.

[0031] The image generating unit 6 merges the image acquired by the image acquiring unit 1 and the data extracted by the data extracting unit 5 by time, and creates a screen that displays the changes in the state of the machine using data. Changes in the display screen will be described with reference to Figures 11 and 12. Figure 11 shows an example of an image generated by a conventional image generation device 100, and Figure 12 shows an example of an image generated by an image generation device 100 to which the present disclosure is applied.

[0032] In the conventional image (Fig. 11), the data "Tool length correction amount HG 0005 10.264->102.640," "MDI mode," and "Operation G0014.1" are displayed for the image at the time the tool offset was set.

[0033] In the image (Fig. 12) to which this disclosure is applied, the change in the tool length compensation amount is not displayed. Therefore, only "MDI mode" and "Operation G0014.1" are displayed. Because the amount of data displayed is small, the operation of the industrial machine and the display screen are not obscured by text.

[0034] When analyzing video, displaying all data on the screen results in a large amount of data, cluttering the screen. In the present disclosure, data extraction conditions are set at the time the video is captured, and data that meets the extraction conditions is extracted as display data. The display data is saved with a timestamp. The video generation device merges the video and data using the time as a key, and displays the data superimposed on the video. According to the image generating device of the present disclosure, the amount of display data can be reduced, and therefore a display screen with good visibility can be provided.

[0035] The hardware configuration of an image generation device to which the present disclosure is applied will be described below. Fig. 13 is a hardware configuration diagram of image generation device 100. As shown in Fig. 1, image generation device 100 includes a CPU 111 that controls the entire image generation device 100, a ROM 112 that records programs and data, and a RAM 113 for temporarily expanding data. CPU 111 reads a system program recorded in ROM 112 via a bus and controls the entire image generation device 100 in accordance with the system program.

[0036] The nonvolatile memory 114 is backed up by, for example, a battery (not shown), and the stored state is maintained even when the power to the image generation device 100 is turned off. The nonvolatile memory 114 stores various data such as programs read from external devices via interfaces 115, 118, and 119, and user operations input via the input unit 30. [Explanation of symbols]

[0037] 100 Image generation device 1. Video acquisition unit 2 Data Acquisition Section 3 Screen discrimination section 4 Extraction condition storage section 5 Data Extraction Section 6. Image generation unit 111 CPU 113 RAM 114 Non-volatile memory

Claims

1. an image acquisition unit that acquires an image of the industrial machine, including a display screen of the industrial machine; a data acquisition unit that acquires data indicating a machine state of the industrial machine; a screen discrimination unit that discriminates the type of the display screen; a data extraction unit that extracts a portion of the data based on the type of the display screen and the status information of the industrial machine; an image generating unit that generates an image that simultaneously displays the data extracted by the data extracting unit and the image acquired by the data acquiring unit; An image generating device comprising:

2. The video acquired by the video acquisition unit includes the time when the video was captured, The time when the data acquired by the data acquisition unit was output is added, The image generating device according to claim 1 , wherein the image generating unit merges the image and the data using the time as a key.

3. The image generating device according to claim 1 , wherein the screen determining unit determines the type of the display screen based on the display screen acquired from the image acquiring unit.

4. The image generating device according to claim 1 , wherein the screen determining unit determines the type of the display screen based on at least one of an operation mode or an operation signal of the industrial machine included in the data acquired by the data acquiring unit.

5. The image generating device according to claim 1 , wherein the data extracting section extracts data by excluding data that overlaps with information provided by the display screen determined by the screen determining section.

6. The image generating device according to claim 1 , wherein the data extracting unit extracts data based on at least one of an operation mode or a signal of the industrial machine included in the data acquired by the data acquiring unit.

7. The image generating device according to claim 1 , wherein, when a value of the data acquired by the data acquisition unit changes, the data extraction unit determines whether or not to extract the changed data.

8. 8. The image generating device according to claim 7, wherein the extraction conditions of said data extracting unit include data that is always extracted regardless of changes in the value of the data.

9. When executed by one or more processors, acquiring an image of the industrial machine, the image including a display screen of the industrial machine; acquiring data indicating a machine state of the industrial machine; Identifying the type of the display screen; extracting data to be used for generating an image based on the type of the display screen and the status information of the industrial machine; creating an image that simultaneously displays the extracted data and the acquired image; A storage medium that stores instructions readable by the processor.

Images