Data acquisition method, data acquisition device, computer equipment, and storage medium

Abstract

The embodiment of the present application provides a data acquisition method, a data acquisition device, a computer device, and a storage medium, wherein the data acquisition method includes: acquiring first position data; acquiring posture data; acquiring a calibration plate image captured by a 3D camera based on the first position data and posture data, and obtaining second position data based on the calibration plate image; obtaining first relationship data based on the second position data; obtaining second relationship data based on the first position data and posture data; obtaining hand-eye calibration parameters based on the first relationship data and the second relationship data; and obtaining three-dimensional data of a target object based on the hand-eye calibration parameters. The embodiment of the present application installs a 3D camera on a moving mechanism of a robot, controls the 3D camera to shoot a target object at different positions, and obtains three-dimensional shape data of the target object based on the hand-eye calibration parameters, thereby being able to obtain complete three-dimensional shape data of the target object while keeping the configuration of the 3D camera unchanged.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data acquisition method, a data acquisition device, a computer device, and a storage medium. Background Art

[0002] In the related art, a 3D camera is used to shoot the target object. However, due to the field of view of the 3D camera, a single shot cannot cover the entire target object. At the same time, there will be mutual occlusion in the morphology of the target object, which makes it difficult to obtain complete three-dimensional morphological data of the target object. While keeping the 3D camera configuration unchanged, the field of view of the 3D camera is expanded to enable a single shot to cover the entire object. However, as the field of view expands, the scanning distance of the 3D camera becomes farther, resulting in reduced accuracy and missing data of the scanned target object. Improving the configuration of the 3D camera will significantly increase the application cost. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a data acquisition method, a data acquisition device, a computer device, and a storage medium, which can obtain complete three-dimensional shape data of a target object while keeping the 3D camera configuration unchanged.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a data acquisition method, which is applied to a data acquisition device, wherein the data acquisition device includes a robot and a 3D camera, wherein the 3D camera is installed on a moving mechanism of the robot, and the method includes:

[0005] Acquire first position data, where the first position data is the coordinates of the mobile mechanism relative to the robot base coordinate system;

[0006] Acquiring posture data, wherein the posture data is an angle of the mobile mechanism relative to a base coordinate system of the robot;

[0007] Acquire a calibration plate image captured by the 3D camera according to the first position data and the posture data, and obtain second position data according to the calibration plate image, where the second position data is the coordinates of the target position on the calibration plate in the 3D camera visual coordinate system;

[0008] Obtaining first relationship data according to the second position data, where the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system;

[0009] Obtain second relationship data according to the first position data and the posture data, wherein the second relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system;

[0010] Obtaining hand-eye calibration parameters according to the first relationship data and the second relationship data, wherein the hand-eye calibration parameters are used to characterize the relationship between the 3D camera visual coordinate system and the moving mechanism coordinate system;

[0011] The three-dimensional data of the target object is acquired according to the hand-eye calibration parameters.

[0012] In some embodiments, obtaining the first relationship data according to the second position data includes:

[0013] Acquire target position data of the target position in the calibration plate coordinate system;

[0014] Construct a perspective projection model based on the PNP algorithm;

[0015] The second position data and the target position data are input into the perspective projection model, and the first relationship data is obtained according to the perspective projection model.

[0016] In some embodiments, the calibration plate is a checkerboard calibration plate, and obtaining target position data of the target position in the calibration plate coordinate system includes:

[0017] Acquire the chessboard position data of the target position in the chessboard calibration plate;

[0018] The target position data of the target position in the calibration plate coordinate system is obtained according to the chessboard position data.

[0019] In some embodiments, acquiring three-dimensional data of the target object according to the hand-eye calibration parameters includes:

[0020] Acquire first point cloud position data collected by a 3D camera, where the first point cloud position data is the coordinates of the point cloud data of the target object in the visual coordinate system of the 3D camera;

[0021] Acquire third relationship data, where the third relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system when the 3D camera collects the first point cloud position data;

[0022] Obtain second point cloud position data according to the first point cloud position data, the third relationship data and the hand-eye calibration parameter, wherein the second point cloud position data is the coordinate of the point cloud data in the robot base coordinate system;

[0023] The point cloud data are spliced ​​according to the second point cloud position data to obtain three-dimensional data of the target object in the robot base coordinate system.

[0024] In some embodiments, obtaining the second relationship data according to the first position data and the posture data includes:

[0025] Obtaining a translation matrix of the mobile mechanism relative to the robot base coordinate system according to the first position data;

[0026] Obtaining the Euler rotation matrix of the mobile mechanism relative to the robot base coordinate system according to the posture data;

[0027] The second relationship data is obtained according to the translation matrix and the Euler rotation matrix.

[0028] In some embodiments, acquiring the posture data includes:

[0029] Get the preset Euler angle rotation mode data;

[0030] Controlling the movement of the mobile mechanism according to the Euler angle rotation mode data to obtain the posture data of the mobile mechanism;

[0031] storing the posture data in the teaching pendant of the robot;

[0032] The posture data is obtained from the teaching pendant.

[0033] In some embodiments, after obtaining the hand-eye calibration parameters according to the first relationship data and the second relationship data, the method further includes:

[0034] Storing the hand-eye calibration parameters in a preset saving directory;

[0035] The step of acquiring three-dimensional data of the target object according to the hand-eye calibration parameters includes:

[0036] In response to acquiring the three-dimensional data of the target object in the robot base coordinate system, the hand-eye calibration parameters are read from the saved directory, and the point cloud data of the target object is converted to the robot base coordinate system according to the hand-eye calibration parameters to obtain the three-dimensional data of the target object.

[0037] A second aspect of an embodiment of the present application provides a data acquisition device, the data acquisition device comprising a robot and a 3D camera, the 3D camera being installed on a moving mechanism of the robot, the device comprising:

[0038] A first acquisition module, used for acquiring first position data, where the first position data is the coordinates of the mobile mechanism relative to the robot base coordinate system;

[0039] A second acquisition module is used to acquire posture data, where the posture data is the angle of the mobile mechanism relative to the robot base coordinate system;

[0040] a third acquisition module, configured to acquire a calibration plate image captured by the 3D camera according to the first position data and the posture data, and obtain second position data according to the calibration plate image, wherein the second position data is the coordinates of the target position on the calibration plate in the 3D camera visual coordinate system;

[0041] A first relationship module, used for obtaining first relationship data according to the second position data, wherein the first relationship data is used for representing the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system;

[0042] A second relationship module, used for obtaining second relationship data according to the first position data and the posture data, wherein the second relationship data is used for representing the relationship between the mobile mechanism coordinate system and the robot base coordinate system;

[0043] A hand-eye calibration module, used for obtaining hand-eye calibration parameters according to the first relationship data and the second relationship data, wherein the hand-eye calibration parameters are used for characterizing the relationship between the 3D camera visual coordinate system and the mobile mechanism coordinate system;

[0044] The data acquisition module is used to acquire three-dimensional data of the target object according to the hand-eye calibration parameters.

[0045] A third aspect of an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor is used to execute the method described in any one of the embodiments of the first aspect of the present application.

[0046] A fourth aspect of the embodiments of the present application proposes a storage medium, which is a computer-readable storage medium. The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a method as described in any one of the embodiments of the first aspect of the present application.

[0047] The data acquisition method, data acquisition device, computer equipment, and storage medium proposed in the embodiments of the present application obtain first position data, where the first position data is the coordinate of the moving mechanism relative to the robot base coordinate system, obtain posture data, where the posture data is the angle of the moving mechanism relative to the robot base coordinate system, obtain a calibration plate image captured by a 3D camera based on the first position data and the posture data, obtain second position data based on the calibration plate image, where the second position data is the coordinate of the target position on the calibration plate in the 3D camera visual coordinate system, obtain first relationship data based on the second position data, where the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system, and obtain the first position data based on the posture data. The second relationship data is obtained according to the state data, and the second relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system. The hand-eye calibration parameters are obtained according to the first relationship data and the second relationship data, and the hand-eye calibration parameters are used to characterize the relationship between the 3D camera vision coordinate system and the mobile mechanism coordinate system. The three-dimensional data of the target object is obtained according to the hand-eye calibration parameters. In the embodiment of the present application, by installing a 3D camera on the mobile mechanism of the robot, the robot can control the 3D camera to move, shoot the target object at different positions, and obtain the three-dimensional morphology data of the target object according to the hand-eye calibration parameters. The complete three-dimensional morphology data of the target object can be obtained while keeping the 3D camera configuration unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a data acquisition method provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of the specific method of step S120;

[0050] Figure 3 yes Figure 1 Flow chart of the specific method of step S140;

[0051] Figure 4 yes Figure 3 Flow chart of the specific method of step S310;

[0052] Figure 5 yes Figure 1 Flow chart of the specific method of step S150;

[0053] Figure 6 yes Figure 1 A second flow chart of the specific method of step S170;

[0054] Figure 7 is a first schematic diagram of a data acquisition method provided in an embodiment of the present application;

[0055] Figure 8is a second schematic diagram of the data acquisition method provided in an embodiment of the present application;

[0056] Fig. 9 is a third schematic diagram of the data acquisition method provided in an embodiment of the present application;

[0057] Fig.10 is a fourth schematic diagram of the data acquisition method provided in an embodiment of the present application;

[0058] Fig.11 It is a module structure diagram of the data acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0062] In the related art, a 3D camera is used to shoot the target object. However, due to the field of view of the 3D camera, a single shot cannot cover the entire target object. At the same time, there will be mutual occlusion in the morphology of the target object, which makes it difficult to obtain complete three-dimensional morphological data of the target object. While keeping the 3D camera configuration unchanged, the field of view of the 3D camera is expanded to enable a single shot to cover the entire object. However, as the field of view expands, the scanning distance of the 3D camera becomes farther, resulting in reduced accuracy and missing data of the scanned target object. Improving the configuration of the 3D camera will significantly increase the application cost.

[0063] Based on this, the main purpose of the embodiment of the present application is to propose a data acquisition method, by installing a 3D camera on the moving mechanism of the robot, so that the robot can control the 3D camera to move to different positions to shoot the target object, and convert the point cloud data of the target object collected by the 3D camera into the robot base coordinates according to the hand-eye calibration parameters, and complete the three-dimensional modeling of the point cloud data in the robot base coordinates, so as to obtain the three-dimensional morphology data of the target object. There is no need to improve the configuration of the 3D camera. While maintaining the original configuration of the 3D camera, the complete three-dimensional morphology data of the target object can be obtained.

[0064] The data acquisition method, data acquisition device, computer equipment, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data acquisition method in the embodiments of the present application is described.

[0065] Reference Figure 1 According to the data acquisition method of the first aspect of the embodiment of the present application, the data acquisition method is applied to a data acquisition device, the data acquisition device includes a robot and a 3D camera, the 3D camera is installed on the moving mechanism of the robot, and the data acquisition method includes but is not limited to steps S110 to S170.

[0066] S110, obtaining first position data;

[0067] S120, acquiring posture data;

[0068] S130, acquiring a calibration plate image captured by the 3D camera according to the first position data and the posture data, and obtaining second position data according to the calibration plate image;

[0069] S140, obtaining first relationship data according to the second position data;

[0070] S150, obtaining second relationship data according to the first position data and the posture data;

[0071] S160, obtaining a hand-eye calibration parameter according to the first relationship data and the second relationship data;

[0072] S170, acquiring three-dimensional data of the target object according to the hand-eye calibration parameters.

[0073] In step S110 to step S120, the robot includes a base and a mobile mechanism, a three-dimensional rectangular coordinate system established with the geometric center position of the base as the origin is the robot base coordinate system, a three-dimensional rectangular coordinate system established with the center position of the mobile mechanism as the origin is the mobile mechanism coordinate system, the coordinates of the mobile mechanism relative to the robot base coordinate system are taken as the first position data, the first position data includes an x-coordinate, a y-coordinate and a z-coordinate, the angle of the mobile mechanism relative to the robot base coordinate system is taken as the posture data, the posture data includes an angle in the x-axis direction, an angle in the y-axis direction and an angle in the z-axis direction. It can be understood that the position data and the posture data are collectively referred to as posture data, and the posture data includes three-dimensional position coordinates and angles in three directions, that is, the posture data includes six parameters: an x-coordinate, a y-coordinate, a z-coordinate, an angle in the x-axis direction, an angle in the y-axis direction and an angle in the z-axis direction.

[0074] In step S130, since the 3D camera is installed on the mobile mechanism of the robot, by controlling the position and posture of the mobile mechanism, the 3D camera can be moved to different positions to shoot the calibration plate to obtain multiple groups of calibration plate pictures. According to the first position data and posture data, the calibration plate picture collected by the 3D camera at the corresponding position and posture is obtained, the calibration plate picture is named using a preset naming rule, a point or an area on the calibration plate is selected as the target position, and the second position data is obtained according to the calibration plate picture. The second position data is the coordinate of the target position on the calibration plate in the 3D camera visual coordinate system, wherein the 3D camera visual coordinate system is a pixel coordinate system established with the pixel point in the upper left corner of the calibration plate picture as the origin, which is a two-dimensional rectangular coordinate system, the unit is pixel, and the 3D camera visual coordinate system is represented as uv, u represents the row where the picture pixel point is located, and v represents the column where the picture pixel point is located. The picture format of the calibration plate picture can be bmp, jpg, png, etc., and the embodiment of the present application does not specifically limit the picture format of the calibration plate picture.

[0075] In step S140, the first relationship data is obtained according to the second position data, that is, the first relationship data is obtained according to the coordinates of the target position on the calibration plate in the 3D camera visual coordinate system, wherein the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system, and the calibration plate coordinate system is the world coordinate system, which is a three-dimensional rectangular coordinate system, and is used to describe the spatial position of the target position on the calibration plate in the physical world.

[0076] In step S150, second relationship data is obtained based on the first position data and posture data, that is, the second relationship data is obtained based on the coordinates of the mobile mechanism relative to the robot base coordinate system and the angle of the mobile mechanism relative to the robot base coordinate system, wherein the second relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system.

[0077] In step S160, the hand-eye calibration parameters are obtained according to the first relationship data and the second relationship data, wherein the hand-eye calibration parameters are used to characterize the relationship between the 3D camera visual coordinate system and the mobile mechanism coordinate system. The robot base coordinate system and the calibration plate coordinate system are kept unchanged, the target position on the calibration plate is P, and the 3D camera collects the calibration plate image at the first position. At this time, the position data conversion relationship of the target position in each coordinate system is shown in formula (1).

[0078]

[0079] Among them, P base is the coordinate of the target position in the robot base coordinate system; The relationship between the moving mechanism coordinate system and the robot base coordinate system when the 3D camera is at the first position, that is, the second relationship data of the first position; is the coordinate of the target position in the moving mechanism coordinate system when the 3D camera is in the first position; cam2end is the relationship between the 3D camera visual coordinate system and the mobile mechanism coordinate system, i.e., the hand-eye calibration parameters; is the coordinates of the target position in the 3D camera visual coordinate system when the 3D camera is at the first position, that is, the second position data of the first position; is the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system when the 3D camera is at the first position, that is, the first relationship data of the first position; world is the coordinate of the target position in the calibration plate coordinate system. Formula (1) is simplified as formula (2), which is as follows.

[0080]

[0081] When the 3D camera collects the calibration plate image at the second position, the coordinates of the target position in the robot base coordinates are as shown in formula (3).

[0082]

[0083] Since the robot base coordinate system and the calibration plate coordinate system remain unchanged, the target position selected on the calibration plate remains unchanged, so:

[0084]

[0085] make H cam2end =X, According to formula (4), we can get A2×X×B2=A1×X×B1, that is The hand-eye calibration parameter X is obtained according to the first relationship data B1 of the first position, the second relationship data A1 of the first position, the first relationship data B2 of the second position and the second relationship data A2 of the second position.

[0086] In step S170, the coordinates of the target object in the world coordinate system are obtained. According to the relationship between the world coordinate system and the 3D camera visual coordinate system, the hand-eye calibration parameters, and the relationship between the mobile mechanism coordinate system and the robot base coordinate system, the coordinates of the target object in the world coordinate system are transformed into the robot base coordinate system to obtain the target coordinates. The target object is modeled according to the target coordinates to obtain the three-dimensional data of the target object. The three-dimensional data can be used to achieve the purpose of fully automated defect detection, size detection, positioning and clamping, guiding robots to grab in an unordered manner, guiding automated equipment to process parts, and guiding robots to autonomously produce and assemble on the automated production line, thus realizing intelligent manufacturing of industrial production lines.

[0087] After step S160, the hand-eye calibration parameters are stored in a preset saving directory. In response to obtaining the three-dimensional data of the target object in the robot base coordinate system, the hand-eye calibration parameters are read from the saving directory, and the coordinate system of the point cloud data of the target object is converted to the robot base coordinate system according to the hand-eye calibration parameters to obtain the three-dimensional data of the target object, wherein the preset saving directory can be an SDK directory.

[0088] The data acquisition method proposed in the embodiment of the present application obtains first position data, the first position data is the coordinate of the moving mechanism relative to the robot base coordinate system, obtains posture data, the posture data is the angle of the moving mechanism relative to the robot base coordinate system, obtains a calibration plate image captured by a 3D camera according to the first position data and the posture data, obtains second position data according to the calibration plate image, the second position data is the coordinate of the target position on the calibration plate in the 3D camera visual coordinate system, obtains first relationship data according to the second position data, the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system, and obtains second relationship data according to the first position data and the posture data The first and second relationship data are used to characterize the relationship between the coordinate system of the moving mechanism and the base coordinate system of the robot. The hand-eye calibration parameters are obtained according to the first and second relationship data. The hand-eye calibration parameters are used to characterize the relationship between the 3D camera vision coordinate system and the moving mechanism coordinate system. The three-dimensional data of the target object are obtained according to the hand-eye calibration parameters. In the embodiment of the present application, a 3D camera is installed on the moving mechanism of the robot so that the robot can control the movement of the 3D camera, shoot the target object at different positions, and obtain the three-dimensional morphology data of the target object according to the hand-eye calibration parameters. The complete three-dimensional morphology data of the target object can be obtained while keeping the configuration of the 3D camera unchanged.

[0089] In some embodiments, Figure 2 As shown, step S120 specifically includes but is not limited to steps S210 to S240.

[0090] S210, obtaining preset Euler angle rotation mode data;

[0091] S220, controlling the movement of the mobile mechanism according to the Euler angle rotation mode data to obtain posture data of the mobile mechanism;

[0092] S230, storing the posture data in the teaching pendant of the robot;

[0093] S240, obtaining posture data from the teaching pendant.

[0094] In step S210, preset robot Euler angle rotation mode data is obtained, wherein the Euler angle rotation mode includes rotation modes around three axes and rotation modes around two axes, the rotation modes around three axes include XYZ rotation mode, XZY rotation mode, YXZ rotation mode, YZX rotation mode, ZXY rotation mode, and ZYX rotation mode, and the rotation modes around two axes include XYX rotation mode, YXY rotation mode, XZX rotation mode, ZXZ rotation mode, YZY rotation mode, and ZYZ rotation mode.

[0095] In step S220, the movement of the mobile mechanism is controlled according to the Euler angle rotation mode, so that the mobile mechanism rotates according to the Euler angle rotation mode, and the posture data of the mobile mechanism relative to the robot base coordinate system is obtained. For example, when the robot Euler angle rotation mode is the ZYX rotation mode, the mobile mechanism first rotates around the Z axis of the robot base coordinate system by an angle of α, then rotates around the Y axis of the robot base coordinate system by an angle of β, and finally rotates around the X axis of the robot base coordinate system by an angle of γ, and the posture data of the mobile mechanism relative to the robot base coordinate system is αβγ. When the robot Euler angle rotation mode is the XYZ rotation mode, the mobile mechanism first rotates around the X axis of the robot base coordinate system by an angle of γ, then rotates around the Y axis of the robot base coordinate system by an angle of β, and finally rotates around the Z axis of the robot base coordinate system by an angle of α, and the posture data of the mobile mechanism relative to the robot base coordinate system is γβα. The method of obtaining posture data according to other Euler angle rotation modes is the same as the method of obtaining posture data according to the ZYX rotation mode and the XYZ rotation mode, and will not be repeated here.

[0096] In step S230 to step S240, the posture data is stored in the teaching pendant of the robot, and the posture data can be visually read from the display screen of the teaching pendant of the robot. It is understandable that the first position data can also be stored in the teaching pendant of the robot, so that the first position data can be directly and visually read from the display screen of the teaching pendant of the robot.

[0097] In some embodiments, Figure 3As shown, step S140 specifically includes but is not limited to steps S310 to S330.

[0098] S310, obtaining target position data of the target position in the calibration plate coordinate system;

[0099] S320, constructing a perspective projection model according to the PNP algorithm;

[0100] S330: Input the second position data and the target position data into a perspective projection model, and obtain first relationship data according to the perspective projection model.

[0101] In step S310 , if the target position is P, the target position data of the target position P in the calibration plate coordinate system is obtained, wherein the target position data is the three-dimensional space coordinate of the target position P.

[0102] In step S320, if the coordinates of the target object in the world coordinate system are represented by P world =[X w ,Y w ,Z w ] T , the coordinates in the 3D camera visual coordinate system are expressed as P cam =[X c ,Y c ,Z c ] T , the relationship between the world coordinate system and the camera coordinate system is expressed as H world2cam , then the perspective projection model constructed according to the PNP algorithm is P cam =H world2cam ×P world ,Right now:

[0103] [X c ,Y c ,Z c ] T =[R t][X w ,Y w ,Z w ] T (5)

[0104] Where R is the rotation matrix and t is the translation matrix.

[0105] In step S330, the second position data corresponds to P in the perspective projection model. cam , the target position data corresponds to P in the perspective projection model world , the second position data and the target position data are input into the perspective projection model, and the first relationship data, namely the rotation matrix R and the translation matrix t, are obtained according to the perspective projection model.

[0106] In some embodiments, Figure 4 As shown, the calibration board is a checkerboard calibration board, and step S310 specifically includes but is not limited to steps S410 to S420.

[0107] S410, obtaining chessboard position data of the target position in the chessboard calibration plate;

[0108] S420, obtaining target position data of the target position in the calibration plate coordinate system according to the chessboard position data.

[0109] In step S410 to step S420, since the distance between the squares in the checkerboard calibration plate is fixed and known, the three-dimensional spatial coordinate information of each point on the checkerboard calibration plate in the calibration plate coordinate system is known. The checkerboard position data of the target position in the checkerboard calibration plate is obtained. The checkerboard position data can be the square number of the target position, and the target position data of the target position is obtained according to the square number, the length of a square, and the origin of the calibration plate coordinate system. The checkerboard position data can also be the length of the square where the target position is located from each axis of the calibration plate coordinate system, and the target position data is obtained according to the length.

[0110] In some embodiments, Figure 5 As shown, step S150 specifically includes but is not limited to steps S510 to S530.

[0111] S510, obtaining a translation matrix of the mobile mechanism relative to the robot base coordinate system according to the first position data;

[0112] S520, obtaining the Euler rotation matrix of the mobile mechanism relative to the robot base coordinate system according to the posture data;

[0113] S530: Obtain second relationship data according to the translation matrix and the Euler rotation matrix.

[0114] In step S510, the first position data is the coordinates of the mobile mechanism relative to the robot base coordinate system, and the translation matrix of the mobile mechanism relative to the robot base coordinate system is obtained according to the coordinates of the mobile mechanism relative to the robot base coordinate system. For example, if the first position data is (x1, y1, z1), then the translation matrix t = [x1, y1, z1] T .

[0115] In step S520, when the robot Euler angle rotation mode is the ZYX rotation mode, the posture data of the mobile mechanism relative to the robot base coordinate system is αβγ, then the basic rotation matrix in the X-axis direction is R x (γ), the basic rotation matrix in the Y-axis direction is R y (β), the basic rotation matrix in the Z-axis direction is R z (α), according to the basic rotation matrix in three directions, the Euler rotation matrix R is Rz (α)R y (β)R x (γ).

[0116] In step S530, the translation matrix t and the Euler rotation matrix R are used as the second relationship data, that is, the relationship between the mobile mechanism coordinate system and the robot base coordinate system. If the coordinates of a point in the mobile mechanism coordinate system are known to be (x end ,y end ,z end ), the coordinates of this point in the robot base coordinate system are expressed as (x base ,y base ,z base ), according to the translation matrix t and the Euler rotation matrix R, the coordinates of the point in the robot base coordinate system can be obtained as (x base ,y base ,z base ) T =R(x end ,y end ,z end ) T +t.

[0117] In some embodiments, Figure 6 As shown, step S170 specifically includes but is not limited to steps S610 to S640.

[0118] S610, obtaining first point cloud position data collected by the 3D camera;

[0119] S620, obtaining third relationship data;

[0120] S630, obtaining second point cloud position data according to the first point cloud position data, the third relationship data and the hand-eye calibration parameter;

[0121] S640, splicing the point cloud data according to the second point cloud position data to obtain three-dimensional data of the target object in the robot base coordinate system.

[0122] In step S610, the first point cloud position data is the coordinates of the point cloud data of the target object in the 3D camera visual coordinate system. The point cloud data of the target object is collected by the 3D camera, and after the hand-eye calibration parameters are obtained, the coordinates of the point cloud data of the target object collected by the 3D camera in the 3D camera visual coordinate system are obtained, and the coordinates are used as the first point cloud position data.

[0123] In step S620, the third relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system when the 3D camera collects the first point cloud position data. When the 3D camera collects the target object point cloud data, the position data and posture data of the mobile mechanism relative to the robot base coordinate system at this time are stored in the robot teaching pendant, and the position data and the posture data are read from the teaching pendant. The position data is the coordinate of the mobile mechanism relative to the robot base coordinate system, and the posture data is the angle of the mobile mechanism relative to the robot base coordinate system. The translation matrix of the mobile mechanism relative to the robot base coordinate system is obtained according to the position data, and the Euler rotation matrix of the mobile mechanism relative to the robot base coordinate system is obtained according to the posture data. The translation matrix and the Euler rotation matrix are used as the third relationship data.

[0124] In step S630, the second point cloud position data is the coordinates of the point cloud data in the robot base coordinate system. The first point cloud position data is represented as P cam , the third relation data is represented by H end2base , the hand-eye calibration parameter is expressed as H cam2end , the second point cloud position data is represented as P base , a calculation method for obtaining the second point cloud position data according to the first point cloud position data, the third relationship data and the hand-eye calibration parameters is shown in formula (6).

[0125] P base =H end2base ×H cam2end ×P cam (6)

[0126] In step S640, the coordinates of the point cloud data in the 3D camera visual coordinate system are converted to the robot base coordinate system according to the second point cloud position data, and the point cloud data in the robot base coordinate system is automatically stitched according to a preset point cloud stitching algorithm to perform three-dimensional modeling of the target object and obtain three-dimensional data of the target object in the robot base coordinate system.

[0127] Reference below Figures 7 to 10 The data acquisition method according to the embodiment of the present invention is described in detail with a specific embodiment. It is worth noting that the following description is only an exemplary description and is not a specific limitation of the invention.

[0128] Reference Figure 7 , Figure 77 is the first schematic diagram of the data acquisition method of the embodiment of the present application, and the data acquisition method is applied to a data acquisition device, which includes a machine vision 3D camera 710, a robot 720, a host computer, and a checkerboard calibration plate 730, wherein the robot 720 includes a base 721 and a mobile mechanism 722, and the machine vision 3D camera 710 is installed on the end flange 7221 of the mobile mechanism 722. The three-dimensional rectangular coordinate system established with the base 721 is the robot base coordinate system, expressed as Base; the three-dimensional rectangular coordinate system established with the end flange 7221 is the end flange rectangular coordinate system, expressed as End; the three-dimensional rectangular coordinate system established with the machine vision 3D camera 710 is the 3D camera visual coordinate system, expressed as Cam; the three-dimensional rectangular coordinate system established with the checkerboard calibration plate 730 is the world coordinate system, expressed as World.

[0129] In order to obtain the complete three-dimensional morphological data of the target object, it is first necessary to use the checkerboard calibration plate to perform hand-eye calibration and solve the hand-eye calibration parameters, that is, the relationship between the 3D camera visual coordinate system and the flange end coordinate system. The embodiment of the present application sets the robot Euler angle rotation mode through the robot teaching pendant, controls the mobile mechanism to perform rotational motion, and enables the end flange to move the machine vision 3D camera to the corresponding shooting position to take pictures of the checkerboard calibration plate, and obtain multiple sets of calibration plate pictures. When the end flange moves the machine vision 3D camera to the corresponding shooting position to take pictures of the checkerboard calibration plate, the calibration plate picture obtained by the current shooting and the corresponding position and posture of the end flange relative to the robot base coordinate system are recorded in the data file of the host computer. For example, the calibration plate picture A01.bmp, the position xyz of the end flange relative to the robot base coordinate system, and the posture RxRyRz are recorded in the data file xyzoat.txt. It should be noted that the calibration plate images are named according to the preset naming rules. If the number of calibration plate images is 10 groups and the image format is bmp, the first group of calibration plate images is named A01.bmp, the second group of calibration plate images is named A02.bmp, and so on. The tenth group of calibration plate images is named A10.bmp. It should be further noted that the recording order is set according to the robot Euler angle rotation mode. If the Euler angle rotation mode is ZYX rotation mode, the position data and posture data recorded in the data file are xyzRzRyRx. If the Euler angle rotation mode is XYZ rotation mode, the position data and posture data recorded in the data file are xyzRxRyRz.

[0130] If 10 sets of calibration plate images need to be collected, one set of calibration plate images can be collected by the machine vision 3D camera shooting the calibration plate from the front, i.e., at the 0° position, and the remaining nine sets can be collected by the robot carrying the machine vision 3D camera and rotating the Z axis of the Cartesian coordinate system around the center of the calibration plate at 40°, 80°, 120°, 160°, 200°, 240°, 280°, 320°, 360°, etc. The position and posture of the robot with the machine vision 3D camera are as follows: Figures 8 to 10 shown.

[0131] Reference Figure 8 , the machine vision 3D camera 710 shoots the checkerboard calibration plate 730 from the front, that is, the machine vision 3D camera 710 is at the 0° position, the checkerboard calibration plate 730 is placed directly below the machine vision 3D camera 710 and fixed on the workbench 810, the lens of the machine vision 3D camera 710 is 600mm away from the calibration plate surface 820 of the checkerboard calibration plate 730, the logo surface 830 of the camera housing is facing forward, and the side opposite to the logo surface 830 of the camera housing is the bottom surface 840 of the camera. The logo surface 830 of the camera housing is represented as side A, the bottom surface 840 of the camera is represented as side B, and the calibration plate surface 820 is represented as side C. It should be noted that 600mm is the object distance of the machine vision 3D camera, which can be changed according to the optimal imaging distance of the machine vision 3D camera to ensure that the camera can clearly scan the checkerboard calibration plate. Refer to Fig. 9 and Fig.10 , the robot with the machine vision 3D camera 710 rotates a certain angle around the Z axis of the Cartesian coordinate system of the chessboard calibration plate 730, the logo surface 830 of the camera housing faces forward, and the angle between it and the calibration plate surface 820 is 45°, the lens of the machine vision 3D camera 710 is still 600mm away from the calibration plate surface 820, and since the bottom surface 840 of the camera and the logo surface 830 of the camera housing are parallel to each other, the angle between the bottom surface 840 of the camera and the calibration plate surface 820 is also 45°. It should be noted that when the camera collects the image of the calibration plate, the position of the calibration plate cannot be moved at all, and the angle between the A surface and the C surface is always maintained at 45°, and the A surface is parallel to the edge of the calibration plate in the y-axis direction.

[0132] The collected 10 sets of calibration plate images and their corresponding positions and postures are uploaded to the host computer. The host computer reads the uploaded data and constructs 9 sets of equations as shown in formula (4) based on the uploaded data. The hand-eye calibration parameters are solved by the 9 sets of equations, so that the error of the hand-eye calibration parameters is small. It should be noted that since the uploaded data at the 0° position is consistent with the uploaded data at the 360° position, the uploaded data at the 360° position does not participate in the calculation. It should be further noted that the embodiment of the present application does not specifically limit the number of calibration plate images and the rotation angle of the camera, which can be set according to actual conditions.

[0133] The present application also provides a data acquisition device, such as Fig.11 As shown, the above-mentioned data acquisition method can be implemented. The device includes a robot and a 3D camera. The 3D camera is installed on the moving mechanism of the robot. The device also includes a first acquisition module 1110, a second acquisition module 1120, a third acquisition module 1130, a first relationship module 1140, a second relationship module 1150, a hand-eye calibration module 1160 and a data acquisition module 1170. The first acquisition module 1110 is used to acquire first position data, which is the coordinate of the mobile mechanism relative to the robot base coordinate system; the second acquisition module 1120 is used to acquire posture data, which is the angle of the mobile mechanism relative to the robot base coordinate system; the third acquisition module 1130 is used to acquire the calibration plate image collected by the 3D camera according to the first position data and the posture data, and obtain the second position data according to the calibration plate image, and the second position data is the coordinate of the target position on the calibration plate in the 3D camera visual coordinate system; the first relationship module 1140 is used to obtain the first relationship data according to the second position data, and the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system; the second relationship module 1150 is used to obtain the second relationship data according to the first position data and the posture data, and the second relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system; the hand-eye calibration module 1160 is used to obtain the hand-eye calibration parameters according to the first relationship data and the second relationship data, and the hand-eye calibration parameters are used to characterize the relationship between the 3D camera visual coordinate system and the mobile mechanism coordinate system; the data acquisition module 1170 is used to obtain the three-dimensional data of the target object according to the hand-eye calibration parameters.

[0134] The data acquisition device of the embodiment of the present application is used to execute the data acquisition method in the above embodiment. Its specific processing process is the same as the data acquisition method in the above embodiment, and will not be repeated here.

[0135] The data acquisition device proposed in the embodiment of the present application obtains first position data, the first position data is the coordinate of the moving mechanism relative to the robot base coordinate system, obtains posture data, the posture data is the angle of the moving mechanism relative to the robot base coordinate system, obtains a calibration plate image captured by a 3D camera according to the first position data and the posture data, obtains second position data according to the calibration plate image, the second position data is the coordinate of the target position on the calibration plate in the 3D camera visual coordinate system, obtains first relationship data according to the second position data, the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system, and obtains second relationship data according to the first position data and the posture data The first and second relationship data are used to characterize the relationship between the coordinate system of the moving mechanism and the base coordinate system of the robot. The hand-eye calibration parameters are obtained according to the first and second relationship data. The hand-eye calibration parameters are used to characterize the relationship between the 3D camera vision coordinate system and the moving mechanism coordinate system. The three-dimensional data of the target object are obtained according to the hand-eye calibration parameters. In the embodiment of the present application, a 3D camera is installed on the moving mechanism of the robot so that the robot can control the movement of the 3D camera, shoot the target object at different positions, and obtain the three-dimensional morphology data of the target object according to the hand-eye calibration parameters. The complete three-dimensional morphology data of the target object can be obtained while keeping the configuration of the 3D camera unchanged.

[0136] The present application also provides a computer device, including:

[0137] at least one processor, and

[0138] a memory communicatively connected to at least one processor; wherein,

[0139] The memory stores instructions, and the instructions are executed by at least one processor so that when the at least one processor executes the instructions, a method as described in any one of the embodiments of the first aspect of the present application is implemented.

[0140] The computer device includes: a processor, a memory, an input / output interface, a communication interface and a bus.

[0141] The processor may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0142] The memory can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device or RAM (Random Access Memory). The memory can store operating systems and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory, and the processor is called to execute the data acquisition method of the embodiments of this application;

[0143] Input / output interface, used to realize information input and output;

[0144] Communication interface, used to realize communication interaction between this device and other devices, which can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0145] The bus, which transfers information between the various components of the device (such as the processor, memory, input / output interfaces, and communication interfaces);

[0146] The processor, memory, input / output interface and communication interface are connected to each other through a bus inside the device.

[0147] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the data acquisition method of the embodiment of the present application.

[0148] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0150] It can be understood by those skilled in the art that Figures 1 to 10The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0151] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0153] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0154] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0159] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A data acquisition method, characterized in that: The method is applied to a data acquisition device, the data acquisition device includes a robot and a 3D camera, the 3D camera is installed on a moving mechanism of the robot, and the method includes: Acquire first position data, where the first position data is the coordinates of the mobile mechanism relative to the robot base coordinate system; Acquiring posture data, wherein the posture data is an angle of the mobile mechanism relative to a base coordinate system of the robot; Acquire a calibration plate image captured by the 3D camera according to the first position data and the posture data, and obtain second position data according to the calibration plate image, where the second position data is the coordinates of the target position on the calibration plate in the 3D camera visual coordinate system; Obtaining first relationship data according to the second position data, where the first relationship data is used to characterize the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system; Obtain second relationship data according to the first position data and the posture data, wherein the second relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system; Obtaining hand-eye calibration parameters according to the first relationship data and the second relationship data, wherein the hand-eye calibration parameters are used to characterize the relationship between the 3D camera visual coordinate system and the moving mechanism coordinate system; The three-dimensional data of the target object is acquired according to the hand-eye calibration parameters.

2. The data acquisition method according to claim 1, characterized in that: The obtaining the first relationship data according to the second position data comprises: Acquire target position data of the target position in the calibration plate coordinate system; Construct a perspective projection model based on the PNP algorithm; The second position data and the target position data are input into the perspective projection model, and the first relationship data is obtained according to the perspective projection model.

3. The data acquisition method according to claim 2, characterized in that: The calibration plate is a checkerboard calibration plate, and the step of obtaining target position data of the target position in a coordinate system of the calibration plate includes: Acquire the chessboard position data of the target position in the chessboard calibration plate; The target position data of the target position in the calibration plate coordinate system is obtained according to the chessboard position data.

4. The data acquisition method according to claim 1, characterized in that: The step of acquiring three-dimensional data of the target object according to the hand-eye calibration parameters includes: Acquire first point cloud position data collected by a 3D camera, where the first point cloud position data is the coordinates of the point cloud data of the target object in the visual coordinate system of the 3D camera; Acquire third relationship data, where the third relationship data is used to characterize the relationship between the mobile mechanism coordinate system and the robot base coordinate system when the 3D camera collects the first point cloud position data; Obtain second point cloud position data according to the first point cloud position data, the third relationship data and the hand-eye calibration parameter, wherein the second point cloud position data is the coordinate of the point cloud data in the robot base coordinate system; The point cloud data are spliced ​​according to the second point cloud position data to obtain three-dimensional data of the target object in the robot base coordinate system.

5. The data acquisition method according to claim 1, characterized in that: The obtaining of second relationship data according to the first position data and the posture data comprises: Obtaining a translation matrix of the mobile mechanism relative to the robot base coordinate system according to the first position data; Obtaining the Euler rotation matrix of the mobile mechanism relative to the robot base coordinate system according to the posture data; The second relationship data is obtained according to the translation matrix and the Euler rotation matrix.

6. The data acquisition method according to any one of claims 1 to 5, characterized in that: The acquiring of posture data comprises: Get the preset Euler angle rotation mode data; Controlling the movement of the mobile mechanism according to the Euler angle rotation mode data to obtain the posture data of the mobile mechanism; storing the posture data in the teaching pendant of the robot; The posture data is obtained from the teaching pendant.

7. The data acquisition method according to any one of claims 1 to 5, characterized in that: After obtaining the hand-eye calibration parameters according to the first relationship data and the second relationship data, the method further includes: Storing the hand-eye calibration parameters in a preset saving directory; The step of acquiring three-dimensional data of the target object according to the hand-eye calibration parameters includes: In response to acquiring the three-dimensional data of the target object in the robot base coordinate system, the hand-eye calibration parameters are read from the saved directory, and the point cloud data of the target object is converted to the robot base coordinate system according to the hand-eye calibration parameters to obtain the three-dimensional data of the target object.

8. A data acquisition device, characterized in that: The data acquisition device includes a robot and a 3D camera, wherein the 3D camera is installed on a moving mechanism of the robot, and the device includes: A first acquisition module, used to acquire first position data, where the first position data is the coordinates of the mobile mechanism relative to the robot base coordinate system; A second acquisition module is used to acquire posture data, where the posture data is the angle of the mobile mechanism relative to the robot base coordinate system; a third acquisition module, configured to acquire a calibration plate image captured by the 3D camera according to the first position data and the posture data, and obtain second position data according to the calibration plate image, wherein the second position data is the coordinates of the target position on the calibration plate in the 3D camera visual coordinate system; A first relationship module, used for obtaining first relationship data according to the second position data, wherein the first relationship data is used for representing the relationship between the calibration plate coordinate system and the 3D camera visual coordinate system; A second relationship module, used for obtaining second relationship data according to the first position data and the posture data, wherein the second relationship data is used for representing the relationship between the mobile mechanism coordinate system and the robot base coordinate system; A hand-eye calibration module, used for obtaining hand-eye calibration parameters according to the first relationship data and the second relationship data, wherein the hand-eye calibration parameters are used for characterizing the relationship between the 3D camera visual coordinate system and the mobile mechanism coordinate system; The data acquisition module is used to acquire three-dimensional data of the target object according to the hand-eye calibration parameters.

9. Computer device, characterized in that The computer device includes a memory and a processor, wherein the memory stores a program, and when the program is executed by the processor, the processor is used to execute: The method according to any one of claims 1 to 7.

10. A storage medium, wherein the storage medium is a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is configured to perform: The method according to any one of claims 1 to 7.

Images