Operation management device
By using images captured by multiple cameras to detect and calculate feature points of industrial machinery, the problem of users manually collecting identification information is solved, enabling the automated creation of industrial machinery configuration diagrams in factories and improving efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FANUC LTD
- Filing Date
- 2021-05-07
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies are insufficient for the automated production and management of equipment in factories. Users need to manually collect identification and location information of multiple industrial machines in the factory using mobile phones or other terminal devices, which takes time and effort.
By using images captured by multiple cameras, feature points of industrial machinery are detected, camera parameters are calculated, configuration diagrams are created, and automated identification and management are achieved using operation management devices.
It enables the automated generation of configuration diagrams for industrial machinery in factories, reducing user operation time and effort and improving the efficiency of configuration diagram generation.
Smart Images
Figure CN117256005B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an operation management device. Background Technology
[0002] In factories where multiple industrial machines such as machine tools and robots operate, the following technology has been proposed: Control devices for controlling each piece of industrial machinery and management devices for collecting and monitoring data related to the operating status of each piece of industrial machinery are connected via a network to centrally manage the operating status of each piece of industrial machinery. For example, see Patent Document 1.
[0003] Existing technical documents
[0004] Patent documents
[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-10200 Summary of the Invention
[0006] The problem that the invention aims to solve
[0007] However, in creating the configuration diagram of industrial machinery on the management device, users need to use terminal devices such as smartphones to collect the identification and location information of each of the multiple industrial machines configured in the factory, which takes a lot of time and effort.
[0008] On the other hand, as the price of surveillance cameras decreases, multiple surveillance cameras capable of network connectivity are installed in factories.
[0009] Therefore, it is desirable to use images captured by multiple cameras to easily create configuration diagrams of multiple industrial machines deployed in a factory.
[0010] Methods for solving problems
[0011] One aspect of the operation management device disclosed herein is an operation management device that displays an icon representing at least one industrial machine on a configuration drawing representing the approximate shape of a facility where the industrial machine is installed. The operation management device includes: a machine tool detection unit that detects the industrial machine installed in the facility from images of multiple cameras installed within the facility based on multiple feature points of each industrial machine pre-registered in a machine tool shape storage unit; a change point detection unit that detects, from the images of the multiple cameras, a feature point exhibiting a time-series change and the time at which the feature point undergoes a time-series change; and an identical machine identification unit that identifies one or more identical machines from the images of the multiple cameras based on the time-series change feature point detected by the change point detection unit and the time at which the feature point undergoes a time-series change. The system includes: a camera parameter calculation unit that calculates camera parameters for each of the plurality of cameras based on the distances between at least one set of feature points pre-registered in the machine tool outline storage unit and the distances between the feature points on the image of the industrial machinery identified by the same machine identification unit; a machine distance calculation unit that calculates the distances between each feature point of the industrial machinery identified by the same machine identification unit and the positions of the camera lens centers of the plurality of cameras based on the camera parameters calculated by the camera parameter calculation unit; and a configuration drawing production unit that produces a configuration drawing containing icons of the identified industrial machinery based on the distances between each feature point of the industrial machinery and the positions of the camera lens centers of the plurality of cameras calculated by the machine distance calculation unit and the three-dimensional coordinates of the positions of the camera lens centers of the plurality of cameras pre-registered.
[0012] Invention Effects
[0013] According to one method, it is possible to easily create configuration diagrams of industrial machinery deployed in a factory using images captured by multiple cameras. Attached Figure Description
[0014] Figure 1 This is a functional block diagram illustrating an example of the functional structure of an operation and management system in one implementation method.
[0015] Figure 2A This is an example of an image captured by a camera.
[0016] Figure 2B This is an example of an image captured by a camera.
[0017] Figure 3 It is a block diagram representing the structure of the operation and management device.
[0018] Figure 4 This is a diagram showing an example of a machine tool's feature points.
[0019] Figure 5A This is a diagram showing an example of a plane representing the front of a machine tool.
[0020] Figure 5B This is a diagram showing an example of the plane and the front area of the machine tool after the front of the machine tool has been moved parallel to the ground.
[0021] Figure 6 This is a diagram illustrating an example of the relationship between the mechanical front region and the threshold in an image.
[0022] Figure 7 This diagram illustrates an example of the processing of a camera that automatically determines a positional relationship that can be detected with a specified precision.
[0023] Figure 8 This is a diagram representing an example of a configuration diagram.
[0024] Figure 9A This is an example of a person's action to request an augmented reality display.
[0025] Figure 9B This is an example of a person's action to request an augmented reality display.
[0026] Figure 10 This is a diagram representing an example of an AR image.
[0027] Figure 11 This is a flowchart illustrating the management and processing procedures of the operation and management device. Detailed Implementation
[0028] <One Implementation Method>
[0029] Figure 1 This is a functional block diagram illustrating an example of the functional structure of an operation management system according to one embodiment. Here, a machine tool is used as an example of industrial machinery. Furthermore, this invention can also be applied to industrial robots, service robots, forging machinery, and injection molding machines, among other industrial machinery.
[0030] like Figure 1 As shown, the operation management system 1 includes: an operation management device 10, n machine tools 20(1) to 20(n), and m cameras 30(1) to 30(m) that capture images (moving images) at a specified frame rate (n is an integer of 1 or more, and m is an integer of 2 or more).
[0031] The operation management device 10, machine tools 20(1) to 20(n), and cameras 30(1) to 30(m) can also be interconnected via a network such as a LAN (Local Area Network) or the Internet. In this case, the operation management device 10, machine tools 20(1) to 20(n), and cameras 30(1) to 30(m) have a communication unit (not shown) for communicating with each other via this connection. Furthermore, the operation management device 10, machine tools 20(1) to 20(n), and cameras 30(1) to 30(m) can also be directly interconnected via a wired or wireless connection interface (not shown).
[0032] Machine tools 20(1) to 20(n) are known to those skilled in the art and include a control device 210. Machine tools 20(1) to 20(n) operate according to action commands from the control device 210.
[0033] Furthermore, in the following, without needing to distinguish each of the machine tools 20(1) to 20(n) individually, they will be collectively referred to as "machine tool 20".
[0034] The control device 210 is, for example, a numerical control device known to those skilled in the art, which generates action commands based on control information and sends the generated action commands to the machine tool 20. Thus, the control device 210 controls the action of the machine tool 20.
[0035] Specifically, the control device 210 is a device that controls the machine tool 20 to perform prescribed machining operations. A machining program describing the actions of the machine tool 20 is provided to the control device 210. Based on the provided machining program, the control device 210 generates action commands including movement commands for each axis and rotation commands for the motor driving the spindle, and sends these action commands to the machine tool 20, thereby controlling the motor of the machine tool 20. Thus, the prescribed machining operations based on the machine tool 20 are executed.
[0036] In addition, the control device 210 sends the operating information of the machine tool 20, which includes the action commands, to the operation management device 10. Furthermore, the control device 210 can also append time information indicating the moment when the machine tool 20 performs an action according to the action commands to the operating data and output it to the operation management device 10, based on the clock signal of the clock (not shown) included in the control device 210.
[0037] Furthermore, if the machine tool 20 is a robot or the like, the control device 210 may also be a robot control device or the like.
[0038] Furthermore, the control device 210 can be used to control various types of machinery, including but not limited to machine tools 20 and robots. Industrial machinery includes machine tools, industrial robots, service robots, forging machines, and injection molding machines.
[0039] In this embodiment, a numerical control device is exemplified as the control device 210.
[0040] Cameras 30(1) to 30(m), such as digital cameras like surveillance cameras, are installed in the factory where the machine tool 20 is installed. Cameras 30(1) to 30(m) output frame images captured at a predetermined frame rate as images to the operation management device 10. In addition, based on the clock signals of the clocks (not shown) contained in each of the cameras 30(1) to 30(m), the cameras 30(1) to 30(m) obtain the time when each frame image was captured, append time information indicating the time of acquisition to the image, and output it to the operation management device 10.
[0041] Figure 2A and Figure 2B This is an example of an image captured by camera 30. In Figure 2A and Figure 2B In the image, two cameras 30 positioned at different locations capture images of the same three machine tools 20. That is, each camera 30 is configured to capture images of at least one part of the same machine tool 20, along with its adjacent camera 30.
[0042] Furthermore, in the following, without needing to distinguish each of the cameras 30(1) to 30(m) individually, they will be collectively referred to as "camera 30".
[0043] <Operation Management Device 10>
[0044] The operation management device 10 is, for example, a computer.
[0045] Figure 3 This is a block diagram showing the structure of the operation management device 10.
[0046] like Figure 3As shown, the operation management device 10 includes a control unit 110, a storage unit 130, an input unit 150, and a display unit 170. The control unit 110 further includes an image data acquisition unit 111, a machine tool detection unit 112, a human detection unit 113, a change point detection unit 114, a similar machine recognition unit 115, a camera parameter calculation unit 116, an operation information acquisition unit 117, an operation information machine determination unit 118, a machine distance calculation unit 119, a machine front area coordinate calculation unit 120, a machine front area human detection unit 121, a layout drawing generation unit 122, an augmented reality display request motion detection unit 123, and an augmented reality information transmission unit 124. The storage unit 130 stores image data 131, operation information 132, machine tool outline data 133, and camera three-dimensional coordinate data 134.
[0047] In addition, the operation management device 10 may also have the following function: checking and synchronizing the time of the clocks (not shown) of the control device 210 and each camera 30 at predetermined time intervals.
[0048] <Storage Department 130>
[0049] The storage unit 130 is, for example, a ROM (Read Only Memory) or an HDD (Hard Disk Drive), which stores the system program executed by the control unit 110 (described later) and the running monitoring application. In addition, the storage unit 130 stores image data 131, operating information 132, machine tool outline data 133, and camera three-dimensional coordinate data.
[0050] Image data 131, for example, stores images captured by the camera 30, which are acquired by the image data acquisition unit 111 described later.
[0051] The operation information 132, for example, stores the respective operation information of each machine tool 20 obtained by the operation information acquisition unit 117 described later.
[0052] The machine tool outline data 133, for example, serves as a mechanical outline storage unit, storing multiple feature points (e.g., corners, text on markings, numerical control (NC) components, warning lights, doors, etc.) pre-registered according to the model of the machine tool 20, based on CAD data and other accompanying drawings. Additionally, the machine tool outline data 133 may also store the distances between at least one set of pre-registered feature points, based on the accompanying drawings.
[0053] Figure 4 This is a diagram showing an example of feature points of machine tool 20. In Figure 4 In the diagram, black dots represent feature points.
[0054] In addition, the storage unit 130 stores the three-dimensional coordinates of the location of the origin of the camera coordinate system (i.e., the center of the camera lens) of each pre-registered camera 30 in the world coordinate system. Furthermore, the origin of the camera coordinate system is set to the center of the camera lens, and the Z-axis of the camera coordinate system is a straight line passing through the center of the lens and orthogonal to the lens surface, i.e., the optical axis of the camera 30. Additionally, the world coordinate system may use a predetermined location within the factory (e.g., a corner of a floor in the factory) as its origin.
[0055] Hereinafter, the three-dimensional coordinate values of the camera lens center of each camera 30 in the world coordinate system will be referred to as camera three-dimensional coordinate data 134.
[0056] <Input Section 150>
[0057] The input unit 150 may be, for example, a keyboard or a touch panel located on the display unit 170 described later, and accepts user input. The input unit 150 may function as a camera parameter input unit that accepts camera parameters of the camera 30 based on input operations from users such as operators, or it may function as a camera coordinate input unit that accepts three-dimensional coordinates representing the positions of the camera lens centers of each camera 30 in the world coordinate system.
[0058] <Display Unit 170>
[0059] The display unit 170 may be, for example, an LCD display, or it may function as a configuration information display unit that displays the configuration diagram of the machine tool 20 produced by the configuration diagram production unit 122 (described later).
[0060] <Control Unit 110>
[0061] The control unit 110 is well known to those skilled in the art and includes a CPU, ROM, RAM, and CMOS (Complementary Metal-Oxide-Semiconductor).
[0062] ) Memory, etc., which are configured to communicate with each other via a bus.
[0063] The CPU is the processor that controls the overall operation management device 10. The CPU reads the system program and operation management application stored in the ROM via the bus, and controls the operation management device 10 as a whole according to the system program and operation management application. Thus, as shown in FIG2, the control unit 110 is configured to perform the functions of an image data acquisition unit 111, a machine tool detection unit 112, a human detection unit 113, a change point detection unit 114, a similar machine recognition unit 115, a camera parameter calculation unit 116, an operation information acquisition unit 117, an operation information machine determination unit 118, a machine distance calculation unit 119, a machine front area coordinate calculation unit 120, a machine front area human detection unit 121, a layout map creation unit 122, an augmented reality display request action detection unit 123, and an augmented reality information transmission unit 124. Temporary calculation data, display data, and other data are stored in RAM. Furthermore, the CMOS memory is configured as a non-volatile memory that is backed up by a battery (not shown) and maintains its storage state even when the power supply to the operation management device 10 is disconnected.
[0064] The image data acquisition unit 111 acquires frame images of images captured by the camera 30 and attached with time information. The image data acquisition unit 111 stores the acquired images in the image data 131 according to the camera 30.
[0065] The machine tool inspection unit 112 performs feature point extraction processing, which is known to those skilled in the art, on the image acquired by the image data acquisition unit 111, and compares the extracted feature points with multiple feature points of each machine tool 20 that are pre-registered in the machine tool shape data 133 to detect the captured machine tool 20.
[0066] The human detection unit 113 performs feature point extraction processing, which is known to those skilled in the art, on frame images of images acquired by the image data acquisition unit 111. The extracted feature points are compared with feature point data (not shown) representing human joints, etc., which are pre-registered in the storage unit 130, to detect workers and other people captured in the image.
[0067] The change point detection unit 114 detects the feature points of time-series change among multiple feature points of the machine tool 20 detected by the machine tool detection unit 112, and the time when the feature points undergo time-series change, based on the images captured by the camera 30 and the time information attached to the images.
[0068] Specifically, the change point detection unit 114, for example, identifies the feature point corresponding to the warning light disposed on the machine tool 20 among the multiple detected feature points of the machine tool 20 as the feature point undergoing a time-series change. The change point detection unit 114 uses the image captured by the camera 30 and the time information to detect the state of the warning light (e.g., any one of the following lights is lit, such as green indicating operation, yellow indicating stop, or red indicating alarm) and the moment when the state of the warning light changes (e.g., changes from green to yellow) in the image that captured the identified feature point (warning light).
[0069] Additionally, the change point detection unit 114 can also identify the feature point corresponding to the door disposed on the machine tool 20 among the multiple detected feature points of the machine tool 20 as the feature point undergoing time-series change. The change point detection unit 114 can also use the image and time information captured by the camera 30 to detect the opening and closing state of the door and the timing of the door being opened or closed in the image captured of the identified feature point (door).
[0070] The change point detection unit 114 can also store the detected feature points of the time series change and the time of the time series change in the RAM (not shown) included in the control unit 110 according to the machine tool 20.
[0071] The identical machine identification unit 115 identifies one or more identical machine tools 20 in the images of each camera 30 based on the feature points of time-series changes detected by the change point detection unit 114 and the time when the feature points underwent time-series changes.
[0072] Specifically, the same mechanical identification unit 115, for example, in Figure 2A as well as Figure 2B In the image shown, based on the feature points of time-series changes detected by the change point detection unit 114, namely the state of the warning light and the door, and the time when the state of the warning light and the door changed, it is identified as being in... Figure 2A as well as Figure 2B The three machine tools 20 captured in the video are all the same machine tools.
[0073] The camera parameter calculation unit 116 calculates the camera parameters of each camera 30 based on the distance between at least one set of feature points in the machine tool 20 registered in the machine tool outline data 133 in advance and the distance between corresponding feature points on the image of the machine tool 20 identified by the same machine recognition unit 115.
[0074] Specifically, the camera parameter calculation unit 116 uses a known method (e.g., Z. Zhang: A Flexible New Technique for Camera Calibration, IEEE Trans. PAMI, 22, 11 (2000) 1330) to calculate the camera parameters of each camera 30 together with the parameters of distortion aberration in the image of the machine tool 20 containing two feature points with a known distance between feature points, based on the image of the machine tool 20 captured by each camera 30.
[0075] Alternatively, a checkerboard pattern with known distances between feature points can be configured on each machine tool 20, and the camera parameter calculation unit 116 calculates the camera parameters of each camera 30 based on the images of the machine tool 20 containing the checkerboard pattern captured by each camera 30.
[0076] The operation information acquisition unit 117 acquires operation information of the machine tool 20, including action commands, from the control device 210 of each machine tool 20. The operation information acquisition unit 117 stores the acquired operation information in the operation information 132 according to the machine tool 20.
[0077] The operation information machine determination unit 118 determines which machine tool 20 among the machine tools 20 identified by the same machine identification unit 115 is the operation information obtained by the operation information acquisition unit 117 based on the time series change represented by the operation information obtained by the operation information acquisition unit 117, the feature point of the time series change detected by the change point detection unit 114, and the time when the feature point undergoes a time series change.
[0078] Specifically, the operation information mechanical determination unit 118 determines, for example, the state of the machine tool 20, such as running, stopped, alarming, and door opening / closing, from the operation information obtained by the operation information acquisition unit 117, as well as the time when the state of the machine tool 20 changes, and the machine tool 20 identified by the same mechanical identification unit 115 that is consistent with the state of the warning light and door detected by the change point detection unit 114, and the time when the state of the warning light and door changes.
[0079] The mechanical distance calculation unit 119 calculates the distance between the feature points of the machine tool 20 identified by the same mechanical recognition unit 115 and each camera 30 based on the camera parameters of each camera 30 calculated by the camera parameter calculation unit 116.
[0080] Specifically, the mechanical distance calculation unit 119 uses, for example, the camera parameters of each of the cameras 30 calculated by the camera parameter calculation unit 116, and a known method (e.g., SfM (Structure for Motion)) to calculate the distance between the position of each feature point of the machine tool 20 identified by the same mechanical recognition unit 115 in the world coordinate system and the position of the camera lens center of each camera 30 based on the images captured by each camera 30.
[0081] The machine front area coordinate calculation unit 120 calculates the area of the front of the machine tool 20 based on the distance between the feature points of the machine tool 20 and each camera 30 calculated by the machine distance calculation unit 119, and the camera parameters of each camera 30 calculated by the camera parameter calculation unit 116, and calculates the coordinates of the image coordinate system of the images captured by each camera 30.
[0082] Specifically, for example, such as Figure 5A As shown, the machine front area coordinate calculation unit 120 calculates the formula of the plane shown by the thick line on the front of the machine tool 20 in the world coordinate system based on the distances from the position of the center of the camera lens of the camera 30 to the corner representing the front of the machine tool 20.
[0083] Mechanical Front Area Coordinate Calculation Unit 120 Figure 5B As shown, the calculation is a formula for the plane (shown as a thick line) after the calculated plane of the front of the machine tool 20 in the world coordinate system is paralleled in the vertical direction of the front of the machine tool 20 in the world coordinate system by a specified distance (e.g., 2m). The machine front area coordinate calculation unit 120 calculates the area enclosed by the calculated plane of the front of the machine tool 20 and the plane of the front of the machine tool 20 after parallel translation as the machine front area.
[0084] The machine front area coordinate calculation unit 120 uses the camera parameters of the camera 30 to convert the calculated formulas for the plane of the front of the machine tool 20 and the plane of the front of the machine tool 20 after parallel movement from the world coordinate system to the image coordinate system of the image from the camera 30. Thus, as... Figure 5B As shown, the machine front region coordinate calculation unit 120 can calculate the machine front region on the image.
[0085] The machine front area human detection unit 121 detects from the image of the camera 30, which is in a positional relationship that can be detected with a specified accuracy, whether a person is present in the area (machine front area) of the front of the machine tool 20 calculated by the machine front area coordinate calculation unit 120.
[0086] Specifically, the machine front area human detection unit 121 sets the minimum values x1, y1 and maximum values x2, y2 of the front area (machine front area) of the machine tool 20 in the image coordinate system of the image calculated by the machine front area coordinate calculation unit 120 as thresholds.
[0087] Figure 6 This is a diagram illustrating an example of the relationship between the mechanical front region and the threshold in an image.
[0088] Furthermore, when determining whether a person is present in the area in front of the machine using a threshold, the accuracy can sometimes vary depending on the configuration of the camera 30. Therefore, the person detection unit 121 in the area in front of the machine automatically determines which of the cameras 30 can be accurately determined to a certain extent, i.e., which camera 30 is in a positional relationship that can be detected with a specified precision.
[0089] Figure 7 This diagram illustrates an example of the processing of a camera 30 that automatically determines a positional relationship that can be detected with a specified precision. Figure 7 In the diagram, XZ plane of machine tool 20(1) is viewed from the +Y axis direction of the world coordinate system, and the image of machine tool 20(1) is captured by two cameras 30(1) and 30(2). The same applies to the cases of machine tools 20(2) to 20(n) and cameras 30(3) to 30(m), and the explanation is omitted.
[0090] First, the human detection unit 121 in the front area of the machine is located in the XZ plane of the world coordinate system, as follows: Figure 6 As shown, the formulas for the four straight lines represented by dashed lines corresponding to the minimum value x1 and maximum value x2 of the machine front region in the image coordinate system of the images captured by cameras 30(1) and 30(2) are calculated using the external parameters of cameras 30(1) and 30(2) respectively.
[0091] Next, the machine front area human detection unit 121 calculates the area of the region surrounded by the four calculated straight lines, excluding the shaded area in front of the machine.
[0092] Furthermore, if the calculated area is smaller than a certain proportion of the machine front area (for example, 20% of the machine front area can ensure that it is approximately the proportion before the machine tool 20(1)), the machine front area human detection unit 121 will determine that the cameras 30(1) and 30(2) are cameras that can be accurately determined to a certain extent, that is, cameras that are in a positional relationship that can be detected with a specified accuracy.
[0093] If the front-of-machine area human detection unit 121 determines that there are pixels representing the joints of a person's whole body, feet, arms, etc., detected by the human detection unit 113 within the range of thresholds x1 and x2 and thresholds y1 and y2 in the images of cameras 30(1), 30(2), etc., which are capable of being detected with a specified precision, the front-of-machine area human detection unit 121 determines that there is a person in the front-of-machine area.
[0094] The configuration drawing production unit 122 creates a configuration drawing that includes icons of the machine tool 20 recognized by the same mechanical recognition unit 115, based on the distance between each feature point of the machine tool 20 and the position of the camera lens center of each camera 30 calculated by the mechanical distance calculation unit 119, and the three-dimensional coordinates of the position of the camera lens center of each camera 30 pre-registered in the camera three-dimensional coordinate data 134.
[0095] Figure 8 This is a diagram representing an example of a configuration diagram.
[0096] like Figure 8 As shown, the configuration drawing production unit 122 calculates the coordinates of the machine tool 20 in the world coordinate system based on the distances between each feature point of the machine tool 20 and the center positions of the camera lenses of each camera 30, calculated by the mechanical distance calculation unit 119. Based on the calculated coordinates of the machine tool 20 and the three-dimensional coordinates of the center positions of the camera lenses of each camera 30 in the camera three-dimensional coordinate data 134, the configuration drawing production unit 122 creates a configuration drawing 300 that displays a top view (or bird's-eye view) of the icon 310 representing the machine tool 20. Furthermore, the configuration drawing production unit 122 can also display the icon 310 in green when the machine tool 20 is in operation, in yellow when the machine tool 20 is stopped, and in red when the machine tool 20 is under alarm. Additionally, the configuration drawing production unit 122 can also overlay the icon 320 of the person and the icon 310 of the machine tool 20 that detected the person when the person is detected in the machine front area area of the machine tool 20 by the machine front area human detection unit 121.
[0097] Furthermore, the configuration diagram production unit 122 displays the produced configuration diagram 300 in real time on the display unit 170, which serves as the configuration diagram information display unit.
[0098] Furthermore, when the configuration diagram production unit 122 produces the configuration diagram 300, it uses the coordinate values registered in the camera three-dimensional coordinate data 134 as the three-dimensional coordinates of the world coordinate system for the position of the camera lens center of each camera 30, but it may also use the coordinate values input by the operator or others via the input unit 150.
[0099] The augmented reality display request motion detection unit 123 detects the motion of the person in the image of the person detected by the human detection unit 121 in the front area of the machine, and requests the motion of the person in the augmented reality display that overlays the virtual reality onto the real reality.
[0100] Figure 9A as well as Figure 9B This is an example of a person's action to request an augmented reality display.
[0101] like Figure 9A and Figure 9B As shown, when the camera 30 takes pictures of a person who raises a smartphone, tablet, or other device toward the machine tool 20, or a person wearing an augmented reality (AR) device or other wearable device, the augmented reality display request motion detection unit 123 detects the posture of raising the smartphone or other device toward the machine tool 20 or the wearing of the wearable device in the captured image.
[0102] The augmented reality information transmission unit 124 sequentially generates AR image data including the operating information of the machine tool 20, and sends the generated AR image data to a smartphone, tablet terminal, wearable terminal, etc., which is an augmented reality display unit for augmented reality display. The mechanical front area of the machine tool 20 is the area where a person performing the action detected by the augmented reality display request action detection unit 123 exists.
[0103] Figure 10 This is a diagram representing an example of an AR image.
[0104] like Figure 10 As shown, a smartphone or similar device adjusts the position and orientation of the received AR image data according to the camera coordinate system of a camera (not shown) included in the smartphone or similar device, and displays a real-world image captured by the camera (not shown) of the smartphone or similar device and the received AR image.
[0105] Furthermore, the AR image may include, for example, the machine tool model name "ROBODRILL-1" and the operating information "Running" and "To be completed: 5 minutes". Additionally, the AR image may also include "Spindle temperature sensor value: XXX" and a chart of time-series data of the spindle temperature sensor values as operating information.
[0106] <Management and processing of operation management device 10>
[0107] Next, the management and processing operations of the operation management device 10 will be explained.
[0108] Figure 11 This is a flowchart illustrating the management processes of the operation management device 10. The process shown here is executed repeatedly during the management process.
[0109] In step S11, the image data acquisition unit 111 acquires the images captured by each camera 30 from each camera 30.
[0110] In step S12, the machine tool detection unit 112 performs feature point extraction processing on the image obtained in step S11, compares the extracted feature points with multiple feature points of each machine tool 20 registered in the machine tool shape data 133 in advance, and detects the captured machine tool 20.
[0111] In step S13, the human detection unit 113 performs feature point extraction processing on the frame image of the image obtained in step S11, compares the extracted feature points with the human joint and other feature point data (not shown) that are registered in the storage unit 130 in advance, and detects the human captured in the image.
[0112] In step S14, the change point detection unit 114 detects the feature points of time-series change among the multiple feature points of the machine tool 20 detected in step S12 and the time when the feature points underwent time-series change, based on the images captured by the camera 30 and the time information attached to the images.
[0113] In step S15, the same machine identification unit 115 identifies the same machine tool 20 in the images of the respective cameras 30 based on the feature points of time-series changes detected in step S14 and the time when the feature points underwent time-series changes.
[0114] In step S16, the camera parameter calculation unit 116 calculates the camera parameters of each camera 30 based on the distance between feature points of the machine tool 20 registered in the machine tool outline data 133 in advance and the distance between corresponding feature points on the image of the machine tool 20 identified in step S15.
[0115] In step S17, the operation information acquisition unit 117 acquires the operation information of the machine tool 20, which includes the action commands, from the control device 210 of each machine tool 20.
[0116] In step S18, the operation information mechanical determination unit 118 determines which machine tool 20's operation information obtained in step S17 is the operation information of the machine tool 20 identified in step S15, based on the time series change represented by the operation information obtained in step S17, the feature point of the time series change detected in step S14, and the time when the feature point underwent a time series change.
[0117] In step S19, the mechanical distance calculation unit 119 calculates the distance between each feature point of the machine tool 20 identified in step S15 and the position of the center of the camera lens of each camera 30 based on the camera parameters of each camera 30 calculated in step S16.
[0118] In step S20, the machine front area coordinate calculation unit 120 calculates the machine front area of the machine tool 20 based on the distance between each feature point of the machine tool 20 and the position of the camera lens center of each camera 30 calculated in step S19, and the camera parameters of each camera 30 calculated in step S16, by using the coordinates of the image coordinate system of the images of each camera 30.
[0119] In step S21, the machine front area human detection unit 121 detects from the image of the camera 30, which is in a positional relationship that can be detected with a specified accuracy, whether a person is present in the machine front area of the machine tool 20 as calculated in step S20.
[0120] In step S22, the configuration drawing production unit 122 creates a configuration drawing 300 that includes the icon 310 of the machine tool 20 identified in step S15, based on the distance between each feature point of the machine tool 20 and the position of the camera lens center of each camera 30 calculated in step S19, and the three-dimensional coordinates of the position of the camera lens center of each camera 30 registered in the camera three-dimensional coordinate data 134 in advance.
[0121] In step S23, the augmented reality display request motion detection unit 123 determines whether the motion of the person requesting augmented reality display is detected in the image of the person detected by the machine front area person detection unit 121. If the motion of the person requesting augmented reality display is detected, the process proceeds to step S24. If the motion of the person requesting augmented reality display is not detected, the process returns to step S11.
[0122] In step S24, the augmented reality information sending unit 124 sequentially generates AR image data including the operating information of the machine tool 20, and sends the generated AR image data to a smartphone or the like. The machine tool 20's front mechanical area is the area where the person performing the action detected in step S23 exists.
[0123] Based on the above, the operation management device 10 of one embodiment can easily create a configuration diagram of multiple machine tools 20 arranged in a factory using images captured by multiple cameras 30.
[0124] In addition, the operation management device 10 can easily and quickly determine whether there is a person in the area in front of the machine tool 20 without calculating the three-dimensional coordinates of the person's standing position.
[0125] In addition, the operation management device 10 does not obtain the location information of devices such as smartphones or the location information of the machine tool 20 that wants to display the operation information in augmented reality. It can determine which machine tool 20 to display the operation information in augmented reality based on the actions of the person requesting the augmented reality display.
[0126] The above describes one embodiment, but the operation management device 10 is not limited to the embodiment described above, and includes variations and improvements within the scope of achieving the purpose.
[0127] <Variation Example>
[0128] In one embodiment, the operation management device 10 is a computer, but it is not limited to this. For example, the server may also include part or all of the following: an image data acquisition unit 111, a machine tool detection unit 112, a human detection unit 113, a change point detection unit 114, a similar machine recognition unit 115, a camera parameter calculation unit 116, an operation information acquisition unit 117, an operation information machine determination unit 118, a machine distance calculation unit 119, a machine front area coordinate calculation unit 120, a machine front area human detection unit 121, a layout map creation unit 122, an augmented reality display request action detection unit 123, and an augmented reality information transmission unit 124 of the operation management device 10. Alternatively, the various functions of the operation management device 10 may be implemented in the cloud using virtual server functions or the like.
[0129] Furthermore, the operation management device 10 can also be a distributed processing system that appropriately distributes the various functions of the operation management device 10 to multiple servers.
[0130] Furthermore, the various functions included in the operation management device 10 of one embodiment can be implemented separately by hardware, software, or a combination thereof. Here, implementation by software means implementation by reading and executing a program by a computer.
[0131] Programs can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disks), optical-magnetic recording media (e.g., optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash memory ROMs, and RAM). Additionally, programs can also be provided to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can provide programs to a computer via wired communication paths such as wires and optical fibers, or via wireless communication paths.
[0132] Furthermore, the steps describing a program recorded in a recording medium naturally include processes performed in that order in a time sequence, as well as processes that are not necessarily performed in a time sequence, and processes that are performed in parallel or individually.
[0133] In other words, the operation management device disclosed herein can be implemented in various ways having the following structure.
[0134] (1) The operation management device 10 of this disclosure displays an icon representing at least one machine tool 20 on a configuration drawing representing the approximate shape of a facility where the machine tool 20 is installed. The operation management device 10 includes: a machine tool detection unit 112, which detects the machine tool 20 installed in the facility from the images of multiple cameras 30 installed in the facility based on multiple feature points of each machine tool 20 pre-registered in machine tool shape data 133; a change point detection unit 114, which detects the feature points of time-series changes among the multiple feature points of the detected machine tool 20 and the time when the feature points of time-series changes occur from the images of the multiple cameras 30; an identical machine identification unit 115, which identifies one or more identical machine tools 20 from the images of the multiple cameras 30 based on the feature points of time-series changes detected by the change point detection unit 114 and the time when the feature points of time-series changes occur; and a camera reference unit 115. The data calculation unit 116 calculates the camera parameters of each of the multiple cameras 30 based on the distance between at least one set of feature points pre-registered in the machine tool outline data 133 and the distance between feature points on the image of the machine tool 20 identified by the same machine tool recognition unit 115; the machine distance calculation unit 119 calculates the distance between each feature point of the machine tool 20 identified by the same machine tool recognition unit 115 and the position of the center of the camera lens of each of the multiple cameras 30 based on the camera parameters of each of the multiple cameras 30 calculated by the camera parameter calculation unit 116; the configuration drawing production unit 122 produces a configuration drawing 300 that configures the icon 310 of the identified machine tool 20 based on the distance between each feature point of the machine tool 20 and the position of the center of the camera lens of each of the multiple cameras 30 calculated by the machine distance calculation unit 119 and the three-dimensional coordinates of the position of the center of the camera lens of each of the multiple cameras pre-registered.
[0135] According to the operation management device 10, it is possible to easily create a configuration diagram 300 of the machine tool 20 configured in the factory using images captured by multiple cameras 30.
[0136] (2) In the operation management device 10 described in (1), it may also include: an operation information acquisition unit 117, which acquires the operation information of the machine tool 20 from the control device 210 controlling the machine tool 20; and an operation information mechanical determination unit 118, which determines which machine tool 20 among the machine tools 20 identified by the same mechanical identification unit 115 is the operation information of the operation information acquired by the operation information acquisition unit 117, based on the time series change represented by the operation information acquired by the operation information acquisition unit 117, the feature point of the time series change detected by the change point detection unit 114, and the time when the feature point undergoes a time series change.
[0137] Therefore, by automatically matching the machine tool 20 on the image with the acquired operating information, the operation management device 10 can reduce the burden on operators and others.
[0138] (3) The operation management device 10 described in (2) may also include: a machine front area coordinate calculation unit 120, which calculates the area in front of the machine tool 20 based on the distance between each feature point of the machine tool 20 and the position of the camera lens center of each of the plurality of cameras 30 calculated by the machine distance calculation unit 119, and the camera parameters of each of the plurality of cameras 30 calculated by the camera parameter calculation unit 116, and calculates the area in front of the machine tool 20 based on the coordinates of the image coordinate system of the images captured by the plurality of cameras 30; and a machine front area human detection unit 121, which detects the human from the plurality of cameras 30. The machine tool 20's front area, calculated by the machine front area coordinate calculation unit 120, detects whether a person is present in the image of a camera capable of detecting positional relationships with a specified accuracy. The configuration drawing production unit 122, based on the distances between the feature points of the machine tool 20 and the respective distances of the multiple cameras 30, calculates by the machine distance calculation unit 119, and in real time configures the machine tool icon 310, the machine tool 20's operating information determined by the operating information machine determination unit 118, and the presence or absence of a person in the front area of the machine tool 20 as detected by the machine front area person detection unit 121 in the configuration drawing 300.
[0139] Therefore, the operation management device 10 can easily and quickly determine whether there is a person in the area in front of the machine tool 20 without calculating the three-dimensional coordinates of the person's standing position.
[0140] (4) The operation management device 10 described in (3) may also include: an augmented reality display request motion detection unit 123, which detects the motion of a person requesting to display an augmented reality image that will be virtually overlaid on the real image from the image of a person detected by the machine front area human detection unit 121; and an augmented reality information transmission unit 124, which transmits augmented reality image data containing operation information of the machine tool 20 to an external device, wherein the area of the front of the machine tool 20 is the area where the person performing the motion detected by the augmented reality display request motion detection unit 123 exists.
[0141] Therefore, the operation management device 10 does not need to obtain the location information of devices such as smartphones or the location information of the machine tool 20 that wants to display the operation information in augmented reality. It can know which machine tool 20 to display the operation information in augmented reality based on the actions of the person who requests the augmented reality display.
[0142] Explanation of reference numerals in the attached figures
[0143] 1. Operation Management System
[0144] 10. Operation Management Device
[0145] 110 Control Department
[0146] 111 Image Data Acquisition Department
[0147] 112 Machine Tool Inspection Department
[0148] 113 people testing department
[0149] 114 Change Point Detection Department
[0150] 115 Identification of the Same Machine
[0151] 116 Camera Parameter Calculation Department
[0152] 117 Operational Information Acquisition Department
[0153] 118 Operation Information Mechanical Determination Department
[0154] 119 Mechanical Distance Calculation Department
[0155] 120 Machine Front Area Coordinate Calculation Department
[0156] 121 Machine Pre-Extraction Area Personnel Inspection Department
[0157] 122 Configuration Diagram Production Department
[0158] 123 Augmented Reality Display Request Motion Detection Department
[0159] 124 Augmented Reality Information Transmission Department
[0160] 130 Storage Department
[0161] 131 Image Data
[0162] 132 Operation Information
[0163] 133 Machine Tool Outline Data
[0164] 134 Camera 3D Coordinate Data
[0165] 150 Input Section
[0166] 170 Display Section
[0167] 20(1)~20(n) machine tools
[0168] 210 Control device
[0169] 30(1)~30(m) camera.
Claims
1. An operation management device that displays icons representing at least one piece of industrial machinery on a configuration diagram showing the approximate shape of a facility where said industrial machinery is installed, characterized in that, The operation management device has: The machine tool inspection department detects the industrial machine installed in the facility from images of multiple cameras installed in the facility based on multiple feature points of each industrial machine that are pre-registered in the machine tool shape storage department. The change point detection unit detects, from the images of the plurality of cameras, the feature points of time-series change among the plurality of feature points of the industrial machinery and the time when the feature points of time-series change; The identical machine identification unit identifies one or more identical industrial machines in the images of the plurality of cameras based on the feature points that change in a time series detected by the change point detection unit and the time when the feature points change in a time series. The camera parameter calculation unit calculates the camera parameters of each of the plurality of cameras based on the distance between at least one set of feature points pre-registered in the machine tool outline storage unit and the distance between the feature points on the image of the industrial machine identified by the same machine recognition unit. The mechanical distance calculation unit calculates the distance between each feature point of the industrial machinery identified by the same mechanical identification unit and the position of the center of the camera lens of each of the plurality of cameras, based on the camera parameters of each of the plurality of cameras calculated by the camera parameter calculation unit. The configuration diagram production unit creates a configuration diagram containing icons of the identified industrial machinery based on the distances between each feature point of the industrial machinery and the center of the camera lens of each of the plurality of cameras calculated by the machinery distance calculation unit, and the three-dimensional coordinates of the center of the camera lens of each of the plurality of cameras registered in advance.
2. The operation management device according to claim 1, characterized in that, The operation management device has: The operation information acquisition unit acquires the operation information of the industrial machinery from the control device that controls the industrial machinery; The operation information machine determination unit determines which of the industrial machines identified by the same machine identification unit the operation information obtained by the operation information acquisition unit belongs to based on the time series change represented by the operation information obtained by the operation information acquisition unit, the feature point of the time series change detected by the change point detection unit, and the time when the feature point underwent the time series change.
3. The operation management device according to claim 2, characterized in that, The operation management device has: The machine front area coordinate calculation unit calculates the area in front of the industrial machine based on the distance between the feature points of the industrial machine and each of the plurality of cameras calculated by the machine distance calculation unit, and the camera parameters of each of the plurality of cameras calculated by the camera parameter calculation unit, and calculates the area in front of the industrial machine using the coordinates of the image coordinate system of the images captured by the plurality of cameras respectively. The machine front area human detection unit detects whether a person is present in the area in front of the industrial machine, as calculated by the machine front area coordinate calculation unit, from images of cameras positioned at a predetermined accuracy among the plurality of cameras. The configuration map production unit, based on the distance between each feature point of the industrial machine calculated by the machine distance calculation unit and the position of the center of the respective camera lens of the plurality of cameras, configures in real time on the configuration map the icon of the industrial machine, the operating information of the industrial machine determined by the operating information machine determination unit, and the presence or absence of people in the area in front of the industrial machine detected by the machine front area human detection unit.
4. The operation management device according to claim 3, characterized in that, The operation management device has: The augmented reality display request motion detection unit detects the motion of the person requesting display from the image of the person detected by the human detection unit in the front area of the machine, and then overlays the virtual image onto the real image in the augmented reality. An augmented reality information transmitting unit transmits augmented reality image data containing operational information of industrial machinery to an external device, wherein the area on the front of the industrial machinery is the area where a person performing the action detected by the augmented reality display request motion detection unit exists.