Specific Point Detection System, Specific Point Detection Method, and Specific Point Detection Program

Abstract

To facilitate processing in detecting a specific point from three-dimensional information of an object.SOLUTION: A specific point detection system 200 comprises: an imaging device 71 that acquires an image of an object W; a first detection unit 83 that receives the image acquired by the imaging device 71 as input and detects a burr B included on the object W using a first detection model 86 that has been trained by machine learning; a three-dimensional scanner 72 that acquires three-dimensional information about the object W including the burr B detected by the first detection unit 83; and a second detection unit 85 that receives the three-dimensional information acquired by the three-dimensional scanner 72 as input and re-detects the burr B using a second detection model 87 that has been trained by machine learning.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a specific point detection system, a specific point detection method, and a specific point detection program.

Background Art

[0002] Conventionally, an apparatus for detecting specific points included in a workpiece has been known. For example, Patent Document 1 discloses an apparatus that detects burrs as specific points by comparing a three-dimensional shape expressed using voxels from measurement data of a workpiece with a 3D-CAD model of the workpiece.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, when measuring, that is, acquiring three-dimensional information of an object and detecting specific points of the object as in Patent Document 1, the detection accuracy of the specific points improves as the three-dimensional information becomes more detailed. In order to acquire detailed three-dimensional information, it is conceivable to improve the resolution of the three-dimensional information representing the object by acquiring three-dimensional information of an enlarged object. However, when acquiring three-dimensional information of an enlarged object, only partial three-dimensional information of the object can be acquired in one acquisition of three-dimensional information. Therefore, the man-hours for acquiring three-dimensional information increase, and the man-hours for detecting specific points from the three-dimensional information also increase.

[0005] The present disclosure has been made in view of such points, and an object thereof is to simplify the process of detecting specific points from three-dimensional information of an object.

Means for Solving the Problems

[0006] The specific point detection system of the present disclosure includes an imaging device that acquires an image of an object, a first detection unit that uses a first detection model learned by machine learning to detect a specific point included in the object by inputting the image acquired by the imaging device, a three-dimensional information acquisition device that acquires three-dimensional information of the object including the specific point detected by the first detection unit, and a second detection unit that uses a second detection model learned by machine learning to re-detect the specific point by inputting the three-dimensional information acquired by the three-dimensional information acquisition device.

[0007] The specific point detection method of the present disclosure includes acquiring an image of an object, detecting a specific point included in the object by using a first detection model learned by machine learning with the image as an input, acquiring three-dimensional information of the object including the specific point detected by the first detection model, and re-detecting the specific point by using a second detection model learned by machine learning with the three-dimensional information as an input.

[0008] The specific point detection program of the present disclosure causes a computer to acquire an image of an object, detect a specific point included in the object by using a first detection model learned by machine learning with the image as an input, acquire three-dimensional information of the object including the specific point detected by the first detection model, and re-detect the specific point by using a second detection model learned by machine learning with the three-dimensional information as an input.

Advantages of the Invention

[0009] According to the specific point detection system, the process of detecting a specific point from the three-dimensional information of the object can be simplified.

[0010] According to the specific point detection method, the process of detecting a specific point from the three-dimensional information of the object can be simplified.

[0011] According to the specific inspection program, the process of detecting specific points from the three-dimensional information of the object can be simplified.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0013] Hereinafter, exemplary embodiments will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram showing the configuration of a processing system 100 according to an embodiment. Note that the dashed line in FIG. 1 represents wireless communication.

[0014] The processing system 100 includes a specific point detection system 200 (hereinafter also referred to as the "detection system 200") that detects specific points of the object W. In this example, the specific point is a burr on the object W. Burrs include casting burrs, cutting burrs, grinding burrs, shearing burrs, plastic deformation burrs, sprue burrs, and welding burrs, etc.

[0015] The detection system 200 includes an imaging device 71 that acquires an image of the object W, a first detection unit 83 (see FIG. 6) that detects a burr B included in the object W based on the image acquired by the imaging device 71, a three-dimensional scanner 72 that acquires three-dimensional information of the object W including the burr B detected by the first detection unit 83, and a second detection unit 85 (see FIG. 6) that re-detects the burr B based on the three-dimensional information acquired by the three-dimensional scanner 72. The three-dimensional scanner 72 is an example of a three-dimensional information acquisition device.

[0016] The detection system 200 further includes a robot arm 12. The imaging device 71 and the three-dimensional scanner 72 are arranged on the robot arm 12. More specifically, the detection system 200 further includes a robot 1 and a control device 3 that controls the robot 1. The robot arm 12 is included in the robot 1.

[0017] The processing system 100 further includes an operating device 2 operated by a user. The robot 1 applies an action to the object W according to the operation of the operating device 2. The control device 3 also controls the operating device 2. The robot 1 and the operating device 2 are remotely arranged. The processing system 100 can perform manual processing and automatic processing of the object W.

[0018] The processing system 100 realizes remote control between the operating device 2 and the robot 1. In remote control, the operating device 2 functions as a master device, and the robot 1 functions as a slave device.

[0019] In the present disclosure, the operations performed by the robot 1 do not include teaching operations and confirmation and correction operations of instructions. Therefore, the operating device 2 does not include a teach pendant.

[0020] The robot 1 and the operating device 2 are communicably connected. Specifically, the robot 1 is communicably connected to the control device 3. The operating device 2 is communicably connected to the control device 3. That is, the operating device 2 communicates with the robot 1 via the control device 3.

[0021] In this example, the robot 1 is an industrial robot. The robot 1 acts on the object W. The action is specifically processing, and more specifically grinding. The action is not limited to grinding, and may be cutting or polishing, etc.

[0022] The robot 1 has a sensor that detects the operating state of the robot 1. In this example, the sensor further has a contact force sensor 13 that detects the reaction force (hereinafter referred to as "contact force") received by the robot 1 from the object W.

[0023] The control device 3 receives the detection result of the contact force sensor 13 via the robot 1. The control device 3 executes operation control of at least one of the robot 1 and the operating device 2 based on the detection result of the contact force sensor 13. In this example, the control device 3 controls the operation of the robot 1 and controls the operation of the operating device 2 so as to present the reaction force acting on the robot 1 to the user according to the operation of the operating device 2 by the user and the detection result of the contact force sensor 13.

[0024] [Robot] The robot 1 may have an end effector 11 that applies an action to the object W, and a robot arm 12 that operates the end effector 11. The robot 1 operates, i.e., moves, the end effector 11 by the robot arm 12, and applies an action to the object W by the end effector 11. The robot 1 may further have a base 10 that supports the robot arm 12 and a robot control device 14 that controls the entire robot 1.

[0025] A three-axis orthogonal robot coordinate system is defined for the robot 1. For example, the Z-axis is set in the vertical direction, and the X-axis and Y-axis that are orthogonal to each other in the horizontal direction are set.

[0026] The end effector 11 has a grinding device 11a and applies grinding as an action to the object W. For example, the grinding device 11a may be a grinder, an orbital sander, a random orbital sander, a delta sander, a belt sander, or the like. The grinder may be of a type that rotates a disk-shaped grinding wheel, a type that rotates a conical or cylindrical grinding wheel, or the like. Here, the grinding device 11a is a grinder.

[0027] The robot arm 12 changes the position of the grinding device 11a. Further, the robot arm 12 may change the posture of the grinding device 11a. The robot arm 12 is a vertically articulated robot arm. The robot arm 12 has a plurality of links 12a, joints 12b that connect the plurality of links 12a, and servo motors 15 (see FIG. 2) that rotationally drive the plurality of joints 12b.

[0028] Incidentally, the robot arm 12 may be a horizontally articulated type, a parallel link type, a rectangular coordinate type, a polar coordinate type robot arm, or the like.

[0029] In this example, the contact force sensor 13 is provided between the robot arm 12 and the end effector 11 (specifically, at the connection between the robot arm 12 and the end effector 11). The contact force sensor 13 detects the contact force that the end effector 11 receives from the object W. The contact force sensor 13 detects forces in three orthogonal axial directions and moments about the three axes.

[0030] Note that the force sensor is not limited to the contact force sensor 13. For example, the contact force sensor 13 may detect only forces in one, two, or three axial directions. Alternatively, the force sensor may be a current sensor that detects the current of the servo motor 15 of the robot arm 12 or a torque sensor that detects the torque of the servo motor 15, etc.

[0031] The imaging device 71 is attached to the robot arm 12. Specifically, the imaging device 71 is attached to the link 12a at the most distal end of the robot arm 12. The imaging device 71 captures an RGB image. The captured image of the imaging device 71 is input from the robot control device 14 to the control device 3 as an image signal.

[0032] The three-dimensional scanner 72 is attached to the robot arm 12. Specifically, the three-dimensional scanner 72 is attached to the link 12a at the most distal end of the robot arm 12. The three-dimensional scanner 72 acquires point cloud data of the object W as three-dimensional information. That is, the three-dimensional scanner 72 outputs the three-dimensional coordinates of a large number of points on the surface of the object W as a point cloud. The point cloud data of the three-dimensional scanner 72 is input from the robot control device 14 to the control device 3.

[0033] When acquiring an image of the object W, the robot arm 12 moves the imaging device 71 to a predetermined imaging position, and when acquiring the point cloud data of the object W, the robot arm 12 moves the three-dimensional scanner 72 to a position corresponding to the bar B detected by the first detection unit 83.

[0034] FIG. 2 is a diagram showing a schematic hardware configuration of the robot control device 14. The robot control device 14 controls the servo motor 15 of the robot arm 12 and the grinding device 11a. The robot control device 14 receives the detection signal of the contact force sensor 13. The robot control device 14 transmits and receives information, commands, data, etc. to and from the control device 3. The robot control device 14 includes a control unit 16, a storage unit 17, and a memory 18.

[0035] The control unit 16 controls the entire robot control device 14. The control unit 16 performs various arithmetic processes. For example, the control unit 16 is formed of a processor such as a CPU (Central Processing Unit). The control unit 16 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0036] The storage unit 17 stores programs and various data executed by the control unit 16. The storage unit 17 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0037] The memory 18 temporarily stores data and the like. For example, the memory 18 is formed of a volatile memory.

[0038] [Operating device] As shown in FIG. 1, the operating device 2 includes an operation unit 21 operated by the user and an operation force sensor 23 that detects the operation force applied by the user to the operation unit 21. The operating device 2 receives an input for manually operating the robot 1 and outputs operation information, which is the input information, to the control device 3. Specifically, the user holds and operates the operation unit 21 to operate the operating device 2. At that time, the operation force sensor 23 detects the force applied to the operation unit 21. The operation force detected by the operation force sensor 23 is output to the control device 3 as operation information.

[0039] The operating device 2 may further include a base 20, a support mechanism 22 provided on the base 20 for supporting the operation unit 21, and an operation control device 24 for controlling the entire operating device 2. The operating device 2 presents a reaction force against the operating force to the user under the control from the control device 3. Specifically, the operation control device 24 receives a command from the control device 3 and controls the support mechanism 22 to make the user sense the reaction force.

[0040] An orthogonal three-axis operation coordinate system is defined for the operating device 2. The operation coordinate system corresponds to the robot coordinate system. That is, the Z-axis is set in the vertical direction, and the X-axis and the Y-axis perpendicular to each other are set in the horizontal direction.

[0041] The support mechanism 22 includes a plurality of links 22a, joints 22b connecting the plurality of links 22a, and servo motors 25 (see FIG. 3) for rotationally driving the plurality of joints 22b. The support mechanism 22 supports the operation unit 21 so that the operation unit 21 can take an arbitrary position and posture in the three-dimensional space. The servo motor 25 rotates corresponding to the position and posture of the operation unit 21. The rotation amount of the servo motor 25, that is, the rotation angle, is uniquely determined.

[0042] In this example, the operation force sensor 23 is provided between the operation unit 21 and the support mechanism 22 (specifically, at the connecting portion between the operation unit 21 and the support mechanism 22). The operation force sensor 23 detects forces in three orthogonal axial directions and moments about the three axes.

[0043] Note that the detection unit for the operation force is not limited to the operation force sensor 23. For example, the operation force sensor 23 may detect only the force in one-axis, two-axis, or three-axis directions. Alternatively, the detection unit may be a current sensor that detects the current of the servo motor 25 of the support mechanism 22 or a torque sensor that detects the torque of the servo motor 25, etc.

[0044] FIG. 3 is a diagram showing a schematic hardware configuration of the operation control device 24. The operation control device 24 operates the support mechanism 22 by controlling the servo motor 25. The operation control device 24 receives a detection signal from the operation force sensor 23. The operation control device 24 transmits and receives information, commands, data, etc. to and from the control device 3. The operation control device 24 includes a control unit 26, a storage unit 27, and a memory 28.

[0045] The control unit 26 controls the entire operation control device 24. The control unit 26 performs various arithmetic processes. For example, the control unit 26 is formed of a processor such as a CPU (Central Processing Unit). The control unit 26 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0046] The storage unit 27 stores programs and various data executed by the control unit 26. The storage unit 27 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

[0047] The memory 28 temporarily stores data and the like. For example, the memory 28 is formed of a volatile memory.

[0048] [Control Device] The control device 3 controls the entire processing system 100 and detection system 200, and controls the operations of the robot 1 and the operating device 2. Specifically, the control device 3 performs manual control of the processing system 100 according to the user's operation and automatic control of the processing system 100. In manual control, the control device 3 performs master-slave control, specifically bilateral control, between the robot 1 and the operating device 2. The control device 3 controls the operation of the robot 1 according to the operation of the operating device 2 by the user's operation, and controls the operation of the operating device 2 so as to present the reaction force according to the detection result of the contact force sensor 13 to the user. That is, the end effector 11 processes the object W according to the user's operation, and the reaction force during processing is presented to the user via the operating device 2. In addition, in automatic control, the control device 3 detects the burr B of the object W. Further, the control device 3 automatically grinds the burr B after detecting the burr B.

[0049] FIG. 4 is a diagram showing a schematic hardware configuration of the control device 3. The control device 3 transmits and receives information, commands, data, etc. with the robot control device 14 and the operation control device 24. The control device 3 includes a control unit 31, a storage unit 32, and a memory 33. Although not shown, the control device 3 may further include an input operation unit operated by the user to set the operation control of the robot 1 and the operating device 2, and a display for displaying the set content.

[0050] The control unit 31 controls the entire control device 3. The control unit 31 performs various arithmetic processes. For example, the control unit 31 is formed of a processor such as a CPU (Central Processing Unit). The control unit 31 may be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0051] The storage unit 32 stores the programs and various data executed by the control unit 31. For example, the storage unit 32 stores the programs for controlling the processing system 100 and the detection system 200. The storage unit 32 is formed of a non-volatile memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like. For example, the program stored in the storage unit 32 is a specific point detection program 32a that causes a computer to execute a predetermined procedure to detect the burr B of the object W.

[0052] The memory 33 temporarily stores data and the like. For example, the memory 33 is formed of a volatile memory.

[0053] <Control of the processing system> In the processing system 100 configured as described above, when manual processing is executed, the control device 3 controls the operation of the robot 1 according to the operation of the operating device 2 by the user, and controls the operation of the operating device 2 so as to present a reaction force according to the detection result of the contact force sensor 13 to the user. Execute manual control. When automatic processing is executed, the control device 3 detects the burr B based on the image and three-dimensional information of the object W by the imaging device 71 and the three-dimensional scanner 72, and performs automatic control in which the robot 1 performs processing on the detected burr B.

[0054] First, the manual control of the processing system 100 will be described. FIG. 5 is a block diagram showing the configuration of the control system of the manual control of the processing system 100.

[0055] The control unit 16 of the robot control device 14 realizes various functions by reading the program from the storage unit 17 and expanding it into the memory 18. Specifically, the control unit 16 functions as an input processing unit 41 and an operation control unit 42.

[0056] The input processing unit 41 outputs information, data, commands, etc. received from the contact force sensor 13 and the servo motor 15 to the control device 3. Specifically, the input processing unit 41 receives a six-axis force detection signal (hereinafter referred to as the "sensor signal") from the contact force sensor 13 and outputs the sensor signal to the control device 3. Further, the input processing unit 41 receives detection signals from the rotation sensor (e.g., encoder) and the current sensor of the servo motor 15. The input processing unit 41 outputs the detection signals to the motion control unit 42 for feedback control of the robot arm 12 by the motion control unit 42. In addition, the input processing unit 41 outputs the detection signals to the control device 3 as the position information of the robot arm 12.

[0057] The motion control unit 42 receives the command position xds from the control device 3 and generates a control command for operating the robot arm 12 according to the command position xds. The motion control unit 42 outputs the control command to the servo motor 15, operates the robot arm 12, and moves the grinding device 11a to the position corresponding to the command position. At this time, the motion control unit 42 performs feedback control of the operation of the robot arm 12 based on the detection signals of the rotation sensor and / or the current sensor of the servo motor 15 from the input processing unit 41. In addition, the motion control unit 42 outputs a control command to the grinding device 11a and operates the grinding device 11a. Thereby, the grinding device 11a grinds the object W.

[0058] The control unit 26 of the operation control device 24 realizes various functions by reading and expanding the program from the storage unit 27 into the memory 28. Specifically, the control unit 26 functions as an input processing unit 51 and a motion control unit 52.

[0059] The input processing unit 51 outputs information, data, commands, etc. received from the operation force sensor 23 to the control device 3. Specifically, the input processing unit 51 receives the detection signals of the six-axis force from the operation force sensor 23 and outputs the detection signals to the control device 3. Further, the input processing unit 51 receives the detection signals of the rotation sensor (e.g., encoder) and the current sensor from the servo motor 25. The input processing unit 51 outputs the detection signals to the operation control unit 52 for the feedback control of the support mechanism 22 by the operation control unit 52.

[0060] The operation control unit 52 receives the command position xdm from the control device 3 and generates a control command for operating the support mechanism 22 according to the command position xdm. The operation control unit 52 outputs the control command to the servo motor 25, operates the support mechanism 22, and moves the operation unit 21 to the position corresponding to the command position. At this time, the operation control unit 52 performs feedback control on the operation of the support mechanism 22 based on the detection signals of the rotation sensor and / or the current sensor of the servo motor 25 from the input processing unit 51. As a result, a reaction force is applied to the operation force applied by the user to the operation unit 21. As a result, the user can operate the operation unit 21 while pseudo-feeling the reaction force from the object W from the operation unit 21.

[0061] The control unit 31 of the control device 3 realizes various functions by reading out the program from the storage unit 32 and expanding it in the memory 33. Specifically, the control unit 31 functions as an operation force acquisition unit 61, a contact force acquisition unit 62, an addition unit 63, a force / velocity conversion unit 64, a first velocity / position conversion unit 65, and a second velocity / position conversion unit 66.

[0062] The operation force acquisition unit 61 receives the detection signal of the operation force sensor 23 via the input processing unit 51 and acquires the operation force fm based on the detection signal. The operation force acquisition unit 61 inputs the operation force fm to the addition unit 63.

[0063] The contact force acquisition unit 62 receives the sensor signal of the contact force sensor 13 via the input processing unit 41 and acquires the contact force fs based on the sensor signal. The contact force acquisition unit 62 inputs the contact force fs to the addition unit 63.

[0064] The addition unit 63 calculates the sum of the operating force fm input from the operating force acquisition unit 61 and the contact force fs input from the contact force acquisition unit 62. Here, since the operating force fm and the contact force fs are forces in opposite directions, the operating force fm and the contact force fs have different positive and negative signs. That is, by adding the operating force fm and the contact force fs, the absolute value of the resultant force fm + fs, which is the sum of the operating force fm and the contact force fs, becomes smaller than the absolute value of the operating force fm. The addition unit 63 outputs the resultant force fm + fs.

[0065] The force / velocity conversion unit 64 converts the input resultant force fm + fs into the commanded velocity xd'. The force / velocity conversion unit 64 calculates the commanded velocity xd' using a motion model based on the equation of motion including the inertia coefficient, the viscous coefficient (damping coefficient), and the stiffness coefficient (spring coefficient). Specifically, the force / velocity conversion unit 64 calculates the commanded velocity xd' based on the following equation of motion.

[0066]

Equation

[0067] Equation (1) is a linear differential equation. Solving Equation (1) for xd' results in Equation (2).

[0068]

Equation

[0069] Equation (2) is stored in the storage unit 32. The force / velocity conversion unit 64 reads Equation (2) from the storage unit 32 to obtain the commanded velocity xd', and outputs the obtained commanded velocity xd' to the first velocity / position conversion unit 65 and the second velocity / position conversion unit 66.

[0070] The first speed / position conversion unit 65 converts the command speed xd' that has been coordinate-converted into a command position xds for robot 1 with reference to the robot coordinate system. For example, when the ratio of the movement amount of robot 1 to the movement amount of the operating device 2 is set, the first speed / position conversion unit 65 multiplies the command position xd obtained from the command speed xd' by the movement ratio according to the movement ratio to obtain the command position xds. The first speed / position conversion unit 65 outputs the obtained command position xds to the robot control device 14, specifically, the motion control unit 42. The motion control unit 42 operates the robot arm 12 based on the command position xds as described above.

[0071] The second speed / position conversion unit 66 converts the command speed xd' into a command position xdm for the operating device 2 with reference to the operation coordinate system. The second speed / position conversion unit 66 outputs the obtained command position xdm to the operation control device 24, specifically, the motion control unit 52. The motion control unit 52 operates the support mechanism 22 based on the command position xdm as described above.

[0072] Next, the automatic control of the processing system 100 will be described. FIG. 6 is a block diagram showing the configuration of the control system for the automatic control of the processing system 100.

[0073] The control unit 31 of the control device 3 realizes various functions by reading a program (for example, the specific point detection program 32a) from the storage unit 32 into the memory 33 and expanding it. Specifically, the control unit 31 functions as an operation command unit 81, an imaging unit 82, a first detection unit 83, a three-dimensional information acquisition unit 84, and a second detection unit 85.

[0074] The first detection unit 83 uses the first detection model 86 that has been learned by machine learning, takes the image acquired by the imaging device 71 as input, and detects the burr B included in the object W (hereinafter, this detection is also referred to as "first detection"). The second detection unit 85 uses the second detection model 87 that has been learned by machine learning, takes the point cloud data acquired by the three-dimensional scanner 72 as input, and re-detects the burr B (hereinafter, this detection is also referred to as "second detection"). The first detection model 86 and the second detection model 87 are stored in the storage unit 32.

[0075] The motion command unit 81 creates the commanded position of the robot arm 12 and obtains the rotation angles of the respective joints 12b of the robot arm 12 corresponding to the created commanded position. Further, the motion command unit 81 creates a command value corresponding to the obtained rotation angles of the respective joints 12b and outputs the created command value to the robot control device 14.

[0076] The robot control device 14 drives the servo motor 15 based on the command value from the motion command unit 81. At this time, the robot control device 14 performs feedback control on the supply current to the servo motor 15 based on the detection result of the encoder.

[0077] The imaging unit 82 controls the imaging device 71 to cause the imaging device 71 to image the object W.

[0078] The first detection unit 83 detects the burr B of the object W based on the image acquired by the imaging device 71. The first detection unit 83 detects the burr B using the first detection model 86 that has been learned by machine learning. The first detection model 86 takes the image of the object W as input and outputs the position of the burr B of the object W.

[0079] The three-dimensional information acquisition unit 84 controls the three-dimensional scanner 72 to cause the three-dimensional scanner 72 to acquire the point cloud data of the object W.

[0080] The second detection unit 85 detects the burr B of the object W based on the point cloud data acquired by the three-dimensional scanner 72. The second detection unit 85 detects the burr B using a second detection model 87 that has been learned by machine learning. The second detection model 87 takes the point cloud data of the object W as input and outputs the position of the burr B of the object W.

[0081] The first detection model 86 and the second detection model 87 will be further described.

[0082] The first detection model 86 is created by machine learning such as deep learning or a neural network. For example, images of the object W before and after grinding the burr B are acquired, and training data in which the classes and regions (i.e., positions) are annotated with the attributes of the burr B and parts other than the burr B are created from the acquired images. Using the created training data as input, the first detection model 86 is created using a deep learning algorithm such as a neural network. The first detection model 86 takes an image of the object W as input and outputs the presence or absence of the burr B of the object W and the position of the burr B.

[0083] More specifically, the first detection model 86 can be a machine learning model having an extraction block that extracts feature amounts of the burr from the image, a position prediction block that regression-predicts the position coordinates of the burr from the feature amounts extracted by the extraction block, and a prediction block that predicts the class of the burr from the feature amounts extracted by the extraction block. A convolutional neural network is used for the extraction of the feature amounts in the extraction block. Note that a convolutional neural network including all the blocks may be used. Also, each block does not necessarily exist independently, and one block may have the functions of a plurality of blocks.

[0084] The training data for the first detection model 86 is an image that includes a relatively large range of the object W, such as an image that includes the whole or most of the object W. For each type of object W, images of the object W having various burrs B are adopted as the training data. Further, the training data for the first detection model 86 is preferably an image of the object W before and after being actually processed by the manual control processing system 100. Since the object W is actually manually processed using the processing system 100, the training data can be easily obtained by acquiring images of the object W before and after grinding at this time. Furthermore, since an image reflecting the environment of the actual site where the robot 1 is installed can be obtained, the first detection model 86 with high detection accuracy of the burr B is created.

[0085] The second detection model 87 is created by machine learning that is more explanatory or readable than the first detection model 86. For example, the second detection model 87 is created using a decision tree-based machine learning algorithm. For example, training data in which classes and regions (i.e., positions) are annotated with attributes of the burr B and parts other than the burr B is created from point cloud data obtained from a sample of the burr B. Using the created training data as input, the second detection model 87 is created using a decision tree-based machine learning algorithm. The second detection model 87 takes the point cloud data of the object W as input and outputs the presence or absence of the burr B of the object W and the position of the burr B.

[0086] More specifically, the second detection model 87 may be a machine learning model having a calculation block that calculates local feature amounts for each point of the point cloud data or each voxel region generated from the point cloud data, a prediction block that predicts the burr class from the feature amounts calculated by the calculation block, and a position calculation block that calculates the position coordinates of the burr from the feature amounts calculated by the calculation block, the class predicted by the prediction block, and geometric processing on the point cloud data. A decision tree-based machine learning algorithm is used for the prediction block. Also, each block does not necessarily exist independently, and one block may have the functions of a plurality of blocks.

[0087] The range of the object W represented by the point cloud data only needs to be a local part including the burr B, and may be smaller than the range of the object W represented by the image input to the first detection model 86. Further, the second detection model 87 is a model created by machine learning that can be additionally trained. That is, the learned second detection model 87 is updated by additionally performing machine learning using new training data.

[0088] The training data of the second detection model 87 is an image including a relatively small range of the object W, such as a local image including the burr B. The training data does not need to be the point cloud data of the entire sample, and may be at least local point cloud data including the burr B. Therefore, the sample may be a part of the same type as the object W or a part of a different type from the object W.

[0089] The range of the object W input to the second detection model 87 as point cloud data is narrower than the range of the object W input to the first detection model 86 as an image. In other words, the object W represented by the point cloud data input to the second detection model 87 is more local than the object W represented by the image input to the first detection model 86.

[0090] In this way, the first detection model 86 globally and preliminarily detects the burr B from the whole of the object W based on the image of the object W. Thereafter, the second detection model 87 locally examines the burr B detected by the first detection unit 83 based on the point cloud data of the object W and re-detects it with higher accuracy than the first detection model 86.

[0091] [Operation of the processing system] Next, the operation of the processing system 100 configured as described above will be described.

[0092] <Manual control> In manual control, the user operates the operating device 2 to cause the robot 1 to perform an actual operation on the object W. For example, the user operates the operating device 2 to cause the robot 1 to grind the object W. As an operation via the user's operating device 2, the operating force applied by the user to the operation unit 21 is detected by the operating force sensor 23. The robot arm 12 is controlled according to the operating force.

[0093] Specifically, when the user operates the operating device 2, the operating force sensor 23 detects the operating force applied by the user via the operation unit 21. At this time, the contact force sensor 13 of the robot 1 detects the contact force.

[0094] The operating force detected by the operating force sensor 23 is input to the control device 3 as a detection signal by the input processing unit 51. In the control device 3, the operating force acquisition unit 61 inputs the operating force fm based on the detection signal to the addition unit 63.

[0095] At this time, the contact force detected by the contact force sensor 13 is input to the input processing unit 41 as a sensor signal. The sensor signal input to the input processing unit 41 is input to the contact force acquisition unit 62. The contact force acquisition unit 62 inputs the contact force fs based on the sensor signal to the addition unit 63.

[0096] The addition unit 63 inputs the combined force fm + fs to the force / velocity conversion unit 64. The force / velocity conversion unit 64 obtains the command velocity xd' based on Equation (2) using the combined force fm + fs.

[0097] Regarding the robot 1, the first velocity / position conversion unit 65 obtains the command position xds from the command velocity xd'. The operation control unit 42 of the robot control device 14 operates the robot arm 12 according to the command position xds and controls the position of the grinding device 11a. Thereby, while the pressing force corresponding to the operating force fm is applied to the object W, the object W is ground by the grinding device 11a.

[0098] On the other hand, regarding the operating device 2, the second speed / position conversion unit 66 obtains a commanded position xdm from the commanded speed xd'. The operation control unit 52 of the operation control device 24 operates the support mechanism 22 according to the commanded position xdm to control the position of the operation unit 21. Thereby, the user senses a reaction force corresponding to the contact force fs.

[0099] When the user operates such an operating device 2, the robot 1 executes machining of the object W.

[0100] Incidentally, it is preferable that an image of the object W is acquired before and after machining of the object W by such manual operation of the user as described above. The acquired image can be used as training data to create the first detection model 86.

[0101] <Automatic Control> Subsequently, the operation of the automatic control of the machining system 100 will be described. FIG. 7 is a flowchart of the automatic control of the machining system 100.

[0102] First, in step S1, the control device 3 executes imaging of the object W. FIG. 8 is a schematic diagram showing the state of the robot arm 12 at the time of imaging the object W. FIG. 9 is a schematic diagram showing an example of an image acquired by the imaging device 71.

[0103] Specifically, as shown in FIG. 8, the operation command unit 81 moves the robot arm 12 so that the imaging device 71 is located at a predetermined imaging position. Thereafter, the imaging unit 82 causes the imaging device 71 to image the object W. For example, as shown in FIG. 9, the imaging device 71 images the object W so as to acquire an image of the entire object W.

[0104] Incidentally, the imaging of the object W may be performed not once but a plurality of times. That is, the imaging device 71 moves to a plurality of imaging positions and images the object W at each imaging position.

[0105] Furthermore, the control device 3 may execute acquisition of point cloud data of the object W in order to detect the position of the object W in the robot coordinate system. At this time, the three-dimensional information acquisition unit 84 causes the three-dimensional scanner 72 to acquire point cloud data of a relatively large range (preferably, the entire range) of the object W. For example, at the position of the robot arm 12 when imaging the object W described above, the three-dimensional scanner 72 acquires point cloud data of the object W. If the position of the robot arm 12 at the time of imaging the object W (i.e., the position of the three-dimensional scanner) is not suitable for acquiring point cloud data of a wide range of the object W, the operation command unit 81 may move the robot arm 12 to an appropriate position and then the three-dimensional scanner 72 may acquire point cloud data of an appropriate range of the object W.

[0106] In addition, when the position of the object W in the robot coordinate system is known, or when the position of the object W is detected using the acquired image of the object W in step S1, etc., acquisition of the point cloud data of the object W described above can be omitted.

[0107] In step S2, the control device 3 executes first detection of the burr B from the acquired image by the imaging device 71. Specifically, the first detection unit 83 inputs the acquired image into the first detection model 86. The first detection model 86 outputs the position of the burr B. If the object W does not contain the burr B, the first detection model 86 outputs that the object W does not contain the burr B. The first detection is executed for the number of acquired images.

[0108] In step S3, the control device 3 executes acquisition of point cloud data of the object W. FIG. 10 is a schematic diagram showing the state of the robot arm 12 when acquiring the point cloud data of the object W. FIG. 11 is a schematic diagram showing an example of the point cloud data acquired by the three-dimensional scanner 72.

[0109] Specifically, the operation instruction unit 81 moves the robot arm 12 so that the three-dimensional scanner 72 is positioned at a location where the point cloud data of the portion of the object W that includes the burr B detected by the first detection can be acquired. The operation instruction unit 81 moves the robot arm 12 based on the position of the burr B detected by the first detection. Thereafter, the three-dimensional information acquisition unit 84 causes the three-dimensional scanner 72 to acquire the point cloud data of the object W. For example, the three-dimensional scanner 72 acquires the point cloud data of a local portion of the object W that includes at least the burr B, rather than the point cloud data of the entire object W.

[0110] Note that the number of times the point cloud data of the object W is acquired corresponds to the number of burrs B detected in the first detection. That is, when a plurality of burrs B are detected in the first detection, a plurality of point cloud data are acquired. However, when a plurality of burrs B can be included in one point cloud data, the acquisition of the point cloud data is executed once for those burrs B.

[0111] In step S4, the control device 3 performs a second detection of the burr B from the point cloud data acquired by the three-dimensional scanner 72. Specifically, the second detection unit 85 inputs the point cloud data into the second detection model 87. The second detection model 87 outputs the position of the burr B. Note that when the object W does not include the burr B, the second detection model 87 outputs that the object W does not include the burr B. The detection of the burr B by the second detection model 87 is performed the same number of times as the number of times the point cloud data is acquired.

[0112] The point cloud data of the burr B preliminarily detected by the first detection model 86 is input to the second detection model 87. That is, the detection of the burr B by this second detection model 87 is a re-determination, that is, a re-evaluation of the burr B detected by the first detection model 86. The first detection model 86 detects the burr B from a two-dimensional image, while the second detection model 87 detects the burr B based on the point cloud data which is three-dimensional information. Therefore, the second detection model 87 can detect the burr B with higher accuracy than the first detection model 86. Furthermore, the point cloud data input to the second detection model 87 is the local point cloud data of the object W. Also in this regard, the second detection model 87 can detect the burr B with high accuracy.

[0113] In step S5, the control device 3 performs grinding of the burr B. FIG. 12 is a schematic diagram showing the state of the robot arm 12 when grinding the burr B. Specifically, the operation command unit 81 outputs a control command to the grinding device 11a to operate the grinding device 11a. Then, the operation command unit 81 moves the robot arm 12 so that the grinding device 11a grinds the burr B detected by the second detection. The operation command unit 81 moves the robot arm 12 based on the position of the burr B detected by the second detection. Thereby, the grinding device 11a grinds the object W. The grinding is performed on all the burrs B detected by the second detection.

[0114] Thus, according to the automatic control of the processing system 100, the processing system 100 detects the burr B of the object W and grinds the detected burr B with the grinding device 11a. At this time, first, using the first detection model 86 with the image of the object W as the input, the burr B contained in the object W is preliminarily detected. Then, the point cloud data of the object W including the burr B detected by the first detection model 86 is acquired by the three-dimensional scanner 72. Using the second detection model 87 with the point cloud data of the object W as the input, the burr B contained in the object W is finally detected.

[0115] Finally, since burr B is detected based on the point cloud data, the detection accuracy of burr B is improved. At this time, it is not necessary to obtain the entire point cloud data of the object W, and it is sufficient to obtain the point cloud data including burr B preliminarily detected from the image of the object W. Therefore, it is possible to obtain local point cloud data of the object W in which burr B is enlarged, so that detailed point cloud data of burr B can be obtained. This also improves the detection accuracy of burr B.

[0116] Furthermore, when obtaining detailed point cloud data for the entire object W, it is necessary to divide the object W into a plurality of parts and obtain the point cloud data a plurality of times. However, by preliminarily detecting burr B using the first detection model 86, the range of the object W for which the point cloud data is obtained can be limited. Therefore, the number of times of obtaining the point cloud data can be reduced. That is, the man-hour for obtaining the point cloud data can be reduced, and as a result, the man-hour for analyzing the point cloud data (i.e., the man-hour for detecting burr B using the second detection model 87) can also be reduced.

[0117] According to the specific inspection of the processing system 100, even without three-dimensional data such as design data, burr B can be accurately detected, and the processing from obtaining the point cloud data of the object W to detecting burr B can be simplified.

[0118] In particular, the resolution of three-dimensional information such as point cloud data is finite. When obtaining the point cloud data of a relatively wide range of the object W by the three-dimensional scanner 72, there is a risk that the data corresponding to burr B is missing. On the contrary, as described above, by preliminarily detecting burr B by the first detection model 86, it is possible to obtain the point cloud data of the enlarged burr B close to burr B when obtaining the point cloud data of the object W by the three-dimensional scanner 72. Thereby, it is possible to prevent the loss of data corresponding to burr B in the point cloud data.

[0119] Further, since the second detection model 87 detects the burr B by using the local point cloud data of the object W including the burr B as input, the training data of the second detection model 87 is point cloud data including the burr B and does not depend on the type of the object W. That is, the point cloud data of the burr B of objects W of different types can also be training data. Therefore, the learned second detection model 87 can be applied to the detection of the burr B of various types of objects W without depending on the type of the object W.

[0120] As described above, the detection system 200 includes an imaging device 71 that acquires an image of the object W, a first detection unit 83 that uses the first detection model 86 learned by machine learning and takes the image acquired by the imaging device 71 as input to detect the burr B (specific point) included in the object W, a three-dimensional scanner 72 (three-dimensional information acquisition device) that acquires three-dimensional information of the object W including the burr B detected by the first detection unit 83, and a second detection unit 85 that uses the second detection model 87 learned by machine learning and takes the three-dimensional information acquired by the three-dimensional scanner 72 as input to re-detect the burr B.

[0121] In other words, the specific point detection method of the specific point detection system includes acquiring an image of the object W, using the first detection model 86 learned by machine learning and taking the image as input to detect the burr B (specific point) included in the object W, acquiring three-dimensional information of the object W including the burr B detected by the first detection model 86, and using the second detection model 87 learned by machine learning and taking the three-dimensional information as input to re-detect the burr B.

[0122] Further, the specific point detection program causes a computer to acquire an image of the object W, use the first detection model 86 learned by machine learning and take the image as input to detect the burr B (specific point) included in the object W, acquire three-dimensional information of the object W including the burr B detected by the first detection model 86, and use the second detection model 87 learned by machine learning and take the three-dimensional information as input to re-detect the burr B in order to detect the burr B of the object W.

[0123] According to these configurations, using the first detection model 86 with the image of the object W acquired by the imaging device 71 as an input, the burr B included in the object W is detected. Three-dimensional information of the object W including the burr B detected in this way is acquired by the three-dimensional scanner 72. Using the second detection model 87 with the acquired three-dimensional information as an input, the burr B included in the object W is detected. That is, the detection of the burr B based on the image of the object W is preliminary detection. Three-dimensional information of the burr B whose position is specified by the preliminary detection of the burr B is acquired by the three-dimensional scanner 72. Therefore, local three-dimensional information of the burr B, that is, enlarged three-dimensional information of the burr B can be acquired. By detecting the burr B using the second detection model 87 with such three-dimensional information as an input, the burr B can be detected accurately. That is, the detection of the burr B based on the three-dimensional information of the object W is final or definitive detection. By performing the detection of the burr B by the first detection unit 83 and the detection of the burr B by the second detection unit 85 in order, the range of the object W for which three-dimensional information is acquired is limited, so the man-hours for acquiring three-dimensional information are reduced and detailed three-dimensional information of the burr B can be acquired. As a result, the process for detecting the burr B from the three-dimensional information of the object W can be simplified and the detection accuracy of the burr B can be improved.

[0124] In addition, the range of the object W input to the second detection model 87 as three-dimensional information is narrower than the range of the object W input to the first detection model 86 as an image.

[0125] According to this configuration, while detecting the burr B by the first detection model 86 using an image of the object W in a relatively wide range as an input, the burr B is detected by the second detection model 87 using three-dimensional information of the object W in a relatively narrow range as an input. Since the detection of the burr B by the first detection model 86 is a preliminary detection, the burr B can be detected from a wide range of the object W by inputting an image of the object W in a relatively wide range. Since the detection of the burr B by the second detection model 87 is for re-detecting the burr B detected by the first detection model 86, fine three-dimensional information can be utilized by inputting three-dimensional information of the object W in a relatively narrow range, and the detection accuracy of the burr B can be improved.

[0126] Specifically, the three-dimensional information is point cloud data.

[0127] According to this configuration, after preliminarily specifying the position of the burr B in the object W by the detection of the burr B by the first detection model 86, the point cloud data of the burr B is acquired by the three-dimensional scanner 72. Therefore, it is possible to acquire the enlarged point cloud data of the burr B and prevent the loss of the point cloud data corresponding to the burr B.

[0128] In addition, the machine learning of the second detection model 87 has higher interpretability than the machine learning of the first detection model 86.

[0129] According to this configuration, since the machine learning of the second detection model 87 has high interpretability, it is easy to verify the reason for the false detection when a false detection of the burr B occurs. In addition, since the machine learning with high interpretability is easy to perform additional learning, additional learning can be performed using the misdetected three-dimensional information as training data, and the update of the second detection model 87 becomes easy.

[0130] In addition, the detection system 200 further includes a robotic arm 12, and the imaging device 71 and the three-dimensional scanner 72 are disposed on the robotic arm 12. When acquiring an image of the object W, the robotic arm 12 moves the imaging device 71 to a predetermined imaging position. When acquiring three-dimensional information of the object W, the robotic arm 12 moves the three-dimensional scanner 72 to a position corresponding to the burr B detected by the first detection unit 83.

[0131] According to this configuration, the movement of the imaging device 71 and the three-dimensional scanner 72 is realized by the robotic arm 12. Therefore, the imaging device 71 can be flexibly moved to an imaging position suitable for acquiring an image of the object W, and the three-dimensional scanner 72 can be flexibly moved to a position suitable for acquiring three-dimensional information of the object W. For example, the robotic arm 12 can move the imaging device 71 to a plurality of imaging positions to acquire an image of the object W multiple times so that there is no portion that is not reflected due to a blind spot. Further, the robotic arm 12 can move the three-dimensional scanner 72 to a position optimal for acquiring three-dimensional information of the burr B detected by the first detection model 86.

[0132] 《Other Embodiments》 As described above, the embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited thereto, and is also applicable to embodiments in which appropriate changes, replacements, additions, omissions, etc. are made. Further, it is also possible to combine the respective components described in the above embodiments to form a new embodiment. In addition, among the components described in the accompanying drawings and the detailed description, there may be included not only the components essential for solving the problem, but also the components not essential for solving the problem for exemplifying the technology. Therefore, just because those non-essential components are described in the accompanying drawings and the detailed description, it should not be immediately determined that those non-essential components are essential.

[0133] For example, although the detection system 200 is incorporated into the processing system 100, it is not limited thereto. The robot 1 is not limited to one capable of realizing bilateral control. For example, the operating device 2 may be omitted.

[0134] The detection system 200 does not necessarily process the burr B until the burr B is detected.

[0135] The specific point of the object is not limited to the burr B. Any part can be a specific point as long as it can be detected by using the learned model with the image and the three-dimensional information as inputs respectively. For example, the specific point may be a painting or welding target point (i.e., the point where painting or welding is performed) of the object. When the painting or welding target point can be distinguished from other parts of the object, the first detection model 86 and the second detection model 87 are created so that the painting or welding target point can be detected. In that case, the processing of the processing system 100 is painting or welding instead of grinding.

[0136] The imaging device 71 does not have to be provided on the robot arm 12. For example, the imaging device 71 may be fixed at a location away from the robot 1. For example, the imaging device 71 may be disposed above the robot 1 and the object W.

[0137] The device for moving the three-dimensional scanner 72 is not limited to the robot arm 12. The three-dimensional scanner 72 is disposed on any device as long as it is a device capable of moving to a position suitable for acquiring the three-dimensional information of the specific point detected by the first detection.

[0138] The three-dimensional information of the object is not limited to point cloud data. The three-dimensional information may be any information representing the three-dimensional shape of the object. For example, the three-dimensional information may be a depth image.

[0139] The training data for the first detection model 86 is not limited to images of the object W before and after being actually processed by the manual control processing system 100. The training data for the first detection model 86 is not limited to images that reflect the actual on-site environment where the robot 1 is installed, and any image can be adopted.

[0140] The first detection model 86 can be a model that has been trained by machine learning using deep learning. Deep learning can be, for example, a neural network, and more specifically, a convolutional neural network.

[0141] The second detection model 87 is not limited to a model that has been trained by decision tree-based machine learning. The second detection model 87 can be any model as long as it takes three-dimensional information such as point cloud data as input and detects specific points. For example, the second detection model 87 can be a regression model using logistic regression or a support vector machine, etc., or a tree structure model using a regression tree, gradient boosting tree, or random forest, etc. Alternatively, the second detection model 87 can be a rule-based model. The models exemplified as these second detection models 87 have relatively high interpretability.

Explanation of Signs

[0142] 200 Specific Point Detection System 12 Robot Arm 71 Imaging Device 72 Three-Dimensional Scanner (Three-Dimensional Information Acquisition Device) 83 First Detection Unit 85 Second Detection Unit 86 First Detection Model 87 Second Detection Model B Burr (Specific Point) W Object

Claims

1. An imaging device that acquires an image of an object, a first detection unit that uses a first detection model learned by machine learning and takes the image acquired by the imaging device as an input to detect a specific point included in the object, a three-dimensional information acquisition device that acquires three-dimensional information of the object including the specific point detected by the first detection unit, a second detection unit that uses a second detection model learned by machine learning and takes the three-dimensional information acquired by the three-dimensional information acquisition device as an input to re-detect the specific point, a specific point detection system in which the range of the object input to the second detection model as the three-dimensional information is narrower than the range of the object input to the first detection model as the image.

2. The specific point detection system according to claim 1, wherein the three-dimensional information is point cloud data.

3. The specific point detection system according to claim 1 or 2, wherein the machine learning of the second detection model is more explanatory than the machine learning of the first detection model.

4. An imaging device that acquires an image of an object, a first detection unit that uses a first detection model learned by machine learning and takes the image acquired by the imaging device as an input to detect a specific point included in the object, a three-dimensional information acquisition device that acquires three-dimensional information of the object including the specific point detected by the first detection unit, a second detection unit that uses a second detection model learned by machine learning and takes the three-dimensional information acquired by the three-dimensional information acquisition device as an input to re-detect the specific point, a robotic arm, wherein the imaging device and the three-dimensional information acquisition device are arranged on the robotic arm, the robotic arm moves the imaging device to a predetermined imaging position when acquiring the image of the object, and moves the three-dimensional information acquisition device to a position corresponding to the specific point detected by the first detection unit when acquiring the three-dimensional information of the object.

5. acquiring an image of an object; using a first detection model learned by machine learning and taking the image as an input to detect a specific point included in the object; acquiring three-dimensional information of the object including the specific point detected by the first detection model; Using the second detection model learned by machine learning, using the three-dimensional information as input to redetect the specific point, including: A specific point detection method in which the range of the object input into the second detection model as the three-dimensional information is narrower than the range of the object input into the first detection model as the image.

6. Obtaining an image of an object by an imaging device arranged on a robotic arm; Using a first detection model learned by machine learning, using the image as input to detect a specific point included in the object; Obtaining three-dimensional information of the object including the specific point detected by the first detection model by a three-dimensional information acquisition device arranged on the robotic arm; Using the second detection model learned by machine learning, using the three-dimensional information as input to redetect the specific point, including: When obtaining the image of the object, moving the imaging device to a predetermined imaging position by the robotic arm; A specific point detection method in which when obtaining the three-dimensional information of the object, moving the three-dimensional information acquisition device by the robotic arm to a position corresponding to the specific point detected by the first detection model.

7. Causing a computer to: Obtain an image of an object; Using a first detection model learned by machine learning, using the image as input to detect a specific point included in the object; Obtain three-dimensional information of the object including the specific point detected by the first detection model; Using a second detection model learned by machine learning, using the three-dimensional information as input to redetect the specific point, A specific point detection program in which the range of the object input into the second detection model as the three-dimensional information is narrower than the range of the object input into the first detection model as the image.

8. Causing a computer to: Obtain an image of an object by an imaging device arranged on a robotic arm; Using a first detection model learned by machine learning, using the image as input to detect a specific point included in the object; Obtaining three-dimensional information of the object including the specific point detected by the first detection model by a three-dimensional information acquisition device arranged on the robotic arm; Execute redetecting the specific point using the second detection model learned by machine learning with the three-dimensional information as input. When acquiring the image of the object, move the imaging device to a predetermined imaging position by the robot arm. A specific point detection program for moving the three-dimensional information acquisition device by the robot arm to a position corresponding to the specific point detected by the first detection model when acquiring the three-dimensional information of the object.

Images