Virtual-real mapping method based on visual feature matching, medium, equipment and device

Abstract

The invention relates to a virtual-real mapping method based on visual feature matching, a medium, equipment and a device, and the method comprises the steps: obtaining an internal reference matrix of collection equipment and a first conversion matrix between the collection equipment and a mechanical arm, and controlling the mechanical arm to move to each collection point to collect a 2D image of a calibration workpiece; the method comprises the following steps: acquiring a mechanical arm, recording current poses of the tail end of the mechanical arm when the tail end of the mechanical arm is positioned at each acquisition point, creating an initial virtual scene, and importing a virtual mechanical arm, virtual acquisition equipment and a workpiece digital model according to the current poses of the tail end of the mechanical arm when the tail end of the mechanical arm is positioned at each acquisition point, an internal reference matrix of the acquisition equipment and a first conversion matrix; and obtaining coordinates and corresponding numbers of image feature points in the 2D image of the calibrated workpiece according to the priori region of interest, constructing an objective function, and obtaining a second conversion matrix between the image feature points and the digital-analog feature points so as to obtain a final virtual scene. The problems that in the prior art, cost is high, operation complexity is high, and accuracy and stability are low are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a virtual-reality mapping method, medium, equipment and device based on visual feature matching. Background Art

[0002] There are three main existing virtual-reality mapping methods: physical measurement, laser measurement, and pointing. Among them, the physical measurement method uses tape measures, calipers, etc. to measure the size and distance in the real scene, and adjusts the position of the calibration workpiece in the virtual scene. It depends on the accuracy and stability of the operator, and there may be human measurement errors, especially in complex measurement tasks and narrow production environments. Some positions or angles may not be directly measured. At the same time, compared with the automated measurement system, the physical measurement method is more time-consuming and not real-time. The laser measurement method uses high-precision equipment such as laser trackers to measure the relative relationship between the robot arm and the calibration workpiece in the scene. However, high-precision equipment such as laser trackers are usually expensive, especially for small or medium-sized enterprises. It may be a financial burden, and the setup and calibration of the equipment may require professional technical support, the operation is more complex, and requires special training. The pointing method makes a pin tool at the end of the robot arm, performs TCP calibration, and points to multiple points for virtual-real mapping. However, the pointing method relies on the accuracy and stability of the end tool (TCP). If the tool itself has errors or instability, it will affect the final virtual-real mapping accuracy. On large and complex calibration workpieces, especially for unstructured or irregularly shaped calibration workpieces, the implementation of the pointing method requires a relatively complex calibration process, which may require more processing steps and algorithm support.

[0003] Therefore, how to improve it is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] Based on this, the purpose of this application is to provide a virtual-reality mapping method, medium, equipment and device based on visual feature matching to solve at least one technical problem mentioned in the above background technology.

[0005] In a first aspect, the present application provides a virtual-reality mapping method based on visual feature matching, comprising:

[0006] The acquisition device is installed at the end of the robotic arm, and the internal parameter calibration and hand-eye calibration of the acquisition device are performed to obtain the internal parameter matrix of the acquisition device and the first conversion matrix between the acquisition device and the robotic arm;

[0007] Set several acquisition points, control the robot arm to move to each acquisition point to acquire the 2D image of the calibration workpiece, and record the current position of the end of the robot arm when it is at each acquisition point;

[0008] Create an initial virtual scene, import the virtual robot arm, virtual acquisition device and workpiece digital model according to the current posture of the end of the robot arm at each acquisition point, the acquisition device internal parameter matrix and the first transformation matrix; the workpiece digital model includes the coordinates of several digital model feature points and the corresponding numbers;

[0009] According to the a priori region of interest, the coordinates and corresponding numbers of each image feature point in the 2D image of the calibration workpiece are obtained;

[0010] According to the internal parameter matrix of the acquisition device, as well as the coordinates and corresponding numbers of each image feature point and digital model feature point, the objective function is constructed to obtain the second conversion matrix between each image feature point and digital model feature point, as well as the current position and posture of the workpiece digital model under the acquisition device;

[0011] The current posture of the workpiece digital model is corrected according to the first conversion matrix and the second conversion matrix to obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

[0012] Furthermore, the steps of creating a virtual scene include:

[0013] Import the industrial robot arm model, acquisition equipment model and the design digital model of the calibrated workpiece of the same model as the actual scene into the virtual scene, obtain the virtual robot arm, virtual acquisition equipment and workpiece digital model, and obtain the coordinates and corresponding numbers of several digital model feature points on the workpiece digital model;

[0014] According to the current posture of the end of the robot arm when it is located at each acquisition point, the corresponding posture of the virtual robot arm in the virtual scene is corrected;

[0015] The virtual acquisition device parameters are configured according to the internal parameter matrix, and the relative position and posture of the virtual acquisition device in the virtual scene is obtained according to the first conversion matrix;

[0016] According to the position of the calibrated workpiece in the real scene, the corresponding workpiece digital model is placed in the virtual scene.

[0017] Furthermore, the step of obtaining a second conversion matrix between each image feature point and the digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device includes:

[0018] Convert each image feature point and digital model feature point into the same scene;

[0019] Construct the objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of the image feature points and the digital model feature points;

[0020] The objective function is solved to obtain the second transformation matrix between each image feature point and the digital model feature point, as well as the current position and posture of the workpiece digital model under the acquisition device.

[0021] Further, the steps of obtaining the second transformation matrix between each image feature point and the digital model feature point, and the current pose of the workpiece digital model under the acquisition device further include:

[0022] Determine whether the number of acquisition points is greater than 1. If not, construct a first objective function; if so, construct a second objective function;

[0023] Solve the objective function to obtain the rotation and translation matrix between each image feature point and the digital model feature point, which is the second transformation matrix, and the current pose of the workpiece digital model under the acquisition device.

[0024] Further, the first objective function is expressed as:

[0025]

[0026] where u i , v i are the coordinates of the i-th image feature point, X i , Y i , Z i are the coordinates of the i-th digital model feature point, 0 < i ≤ N, N is the number of image feature points and digital model feature points, R is the rotation matrix between each image feature point and the digital model feature point, and t is the translation matrix between each image feature point and the digital model feature point; the combination of R and t is the second transformation matrix;

[0027] The second objective function is expressed as Equation 5-2:

[0028]

[0029] where K is the internal parameter matrix of the acquisition device, i is the i-th acquisition point, N is the total number of acquisition points, 0 < i ≤ N, j is the j-th feature point of the image corresponding to the i-th acquisition point, M is the total number of feature points of the image corresponding to the i-th acquisition point, 0 < j ≤ M, is the pixel coordinate of the j-th image feature point in the i-th acquisition point, is the three-dimensional coordinate of the digital model feature point corresponding to the j-th image feature point in the i-th acquisition point, R is the rotation matrix between each image feature point and the digital model feature point, and t is the translation matrix between each image feature point and the digital model feature point; the combination of R and t is the second transformation matrix.

[0030] In a second aspect, the present application provides a virtual-real mapping method based on 3D point cloud feature matching, including:

[0031] Install the acquisition device at the end of the robotic arm, and perform internal parameter calibration and hand-eye calibration of the acquisition device to obtain the internal parameter matrix of the acquisition device and the first transformation matrix between the acquisition device and the robotic arm;

[0032] Set several acquisition points, control the robot arm to move to each acquisition point to collect and calibrate the workpiece surface point cloud, obtain several local calibration workpiece point clouds, and record the current posture of the robot arm when it is located at each acquisition point;

[0033] According to the first transformation matrix, each local calibration workpiece point cloud is transformed into the robot base coordinate system to obtain a global calibration workpiece point cloud;

[0034] Create an initial virtual scene, import the virtual robot arm, virtual acquisition device and workpiece digital model according to the current posture of the end of the robot arm when it is located at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix;

[0035] Control the virtual robot arm to move to each acquisition point to scan the surface of the workpiece digital model, obtain several partial point clouds of the workpiece digital model, and transform each partial point cloud of the workpiece digital model into the robot base coordinate system according to the first transformation matrix to obtain the global workpiece digital model point cloud;

[0036] Performing point cloud registration on the global calibration workpiece point cloud and the global workpiece digital model point cloud to obtain a second transformation matrix between the global calibration workpiece point cloud and the global workpiece digital model point cloud, as well as the current position and posture of the workpiece digital model;

[0037] The current posture of the workpiece digital model is corrected according to the first conversion matrix and the second conversion matrix to obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

[0038] Furthermore, the step of obtaining the second conversion matrix includes:

[0039] Calculate the feature descriptors of each point in the calibration workpiece point cloud and the workpiece digital model point cloud to construct a number of feature point pairs;

[0040] Construct the least squares equation based on the feature point pairs to obtain the second transformation matrix;

[0041] The second transformation matrix is ​​precisely aligned to obtain an optimized second transformation matrix.

[0042] In the third aspect, the present application also provides a computer storage medium storing an executable program code; the executable program code is used to execute the virtual-to-real mapping method based on visual feature matching described in any one of the first aspects or the virtual-to-real mapping method based on 3D point cloud feature matching described in the second aspect.

[0043] In a fourth aspect, the present application also provides a terminal device comprising a memory and a processor; the memory stores a program code that can be executed by the processor; the program code is used to execute the virtual-to-real mapping method based on visual feature matching described in any one of the first aspects or the virtual-to-real mapping method based on 3D point cloud feature matching described in the second aspect.

[0044] In a fifth aspect, the present invention provides a device for a virtual-real mapping method based on visual feature matching, comprising:

[0045] A calibration module, used for acquisition device internal parameter calibration and hand-eye calibration, to obtain the acquisition device internal parameter matrix and the first conversion matrix between the acquisition device and the robotic arm;

[0046] The image acquisition module is used to set a number of acquisition points, control the robot arm to move to each acquisition point to acquire a 2D image of the calibration workpiece, and record the current position of the end of the robot arm when it is located at each acquisition point;

[0047] The virtual scene module is used to create an initial virtual scene, import the virtual robot arm, the virtual acquisition device and the workpiece digital model according to the current posture of the end of the robot arm at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix; the workpiece digital model includes the coordinates of several digital model feature points and the corresponding numbers;

[0048] Computing module, including:

[0049] A feature point acquisition unit is used to obtain the coordinates and corresponding numbers of each image feature point in the 2D image of the calibration workpiece according to the prior region of interest;

[0050] A posture calculation unit is used to construct an objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of each image feature point and the digital model feature point, and obtain a second conversion matrix between each image feature point and the digital model feature point, as well as the current posture of the workpiece digital model under the acquisition device;

[0051] The posture correction unit is used to correct the current posture of the workpiece digital model according to the first transformation matrix and the second transformation matrix, and obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

[0052] The present invention provides a virtual-reality mapping method, medium, equipment and device based on visual feature matching. The method comprises installing a collection device at the end of a mechanical arm, and performing internal parameter calibration and hand-eye calibration of the collection device, so as to obtain an internal parameter matrix of the collection device and a first conversion matrix between the collection device and the mechanical arm, so as to facilitate the subsequent steps to configure the virtual collection device parameters and improve the operation accuracy of the virtual scene. Then, a number of collection points are set, and the mechanical arm is controlled to move to each collection point to collect and calibrate the 2D image of the workpiece, and the current posture of the end of the mechanical arm when it is located at each collection point is recorded, and then an initial virtual scene is created. According to the current posture of the end of the mechanical arm when it is located at each collection point, the internal parameter matrix of the collection device and the first conversion matrix, the virtual mechanical arm, the virtual collection device and the workpiece digital model are imported; the workpiece digital model includes the coordinates and corresponding numbers of a number of digital model feature points, and the image feature points in the 2D image of the calibrated workpiece are obtained according to the prior region of interest. The coordinates and corresponding numbers are constructed according to the internal parameter matrix of the acquisition device, as well as the coordinates and corresponding numbers of each image feature point and digital model feature point, and the objective function is constructed to obtain the second conversion matrix between each image feature point and digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device. Finally, the current position and posture of the workpiece digital model is corrected according to the first conversion matrix and the second conversion matrix, and the corrected position and posture of the workpiece digital model after correction in the robot base coordinate system are obtained to obtain the final virtual scene. By importing all the equipment and workpieces in the real scene into the virtual scene, and initializing the corresponding equipment in the virtual scene according to the relative position and equipment parameters of each equipment in the real scene, the technicians in this field can realize the operations and effects that can be achieved by the same equipment in the real scene by operating various equipment in the virtual scene, which greatly reduces the use cost, reduces the complexity of operation, and can improve the accuracy and stability of the operation through multiple simulations. The problems of high cost, high complexity of operation, low accuracy and stability of the existing technology are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flow chart of a virtual-real mapping method based on visual feature matching according to an embodiment of the present invention;

[0054] Figure 2 It is a flow chart of a virtual-real mapping method based on 3D point cloud feature matching according to an embodiment of the present invention;

[0055] Figure 3 The figure is a schematic diagram of the structure of a device for a virtual-reality mapping method based on visual feature matching according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., then the directional indication is only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly. In addition, if the embodiments of the present invention involve descriptions of "first, second", "S1, S2", "step one, step two", etc., then such descriptions are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features or indicating the execution order of the method, etc. Those skilled in the art can understand that anything that does not violate the gist of the invention under the technical concept of the invention should be included in the protection scope of the present invention.

[0058] like Figure 1 As shown, the present invention provides a virtual-real mapping method based on visual feature matching, which can be optionally used for virtual-real mapping of 2D image feature matching:

[0059] S1: Install the acquisition device at the end of the robotic arm, and perform internal parameter calibration and hand-eye calibration of the acquisition device to obtain the internal parameter matrix of the acquisition device and the first conversion matrix between the acquisition device and the robotic arm;

[0060] Specifically, it is optional but not limited to installing the acquisition device at the end of the robotic arm, and then calibrating the acquisition device using the internal parameter calibration method to obtain the acquisition device internal parameter matrix, and then calibrating the acquisition device and the end of the robotic arm using the hand-eye calibration method to obtain the conversion matrix between the acquisition device and the robotic arm, which is the first conversion matrix; the acquisition device, optionally but not limited to 3D line scan cameras, 3D structured light cameras, TOF cameras, laser scanners, and other common scanning devices that can be used to obtain three-dimensional data of the surface of objects and mobile phones, cameras and other common scanning devices that can be used to obtain 2D data of the surface of objects; the hand-eye calibration method, optionally including common methods such as the checkerboard method, the moving camera method and the feature matching method.

[0061] Preferably, the acquisition device is installed at the tool center point at the end of the robotic arm.

[0062] S2: Set a number of acquisition points, control the robot arm to move to each acquisition point to acquire the 2D image of the calibration workpiece, and record the current posture of the end of the robot arm when it is located at each acquisition point;

[0063] Specifically, it is optional but not limited to controlling the robotic arm to move to several predetermined positions respectively, and then collecting 2D image data of the calibration workpiece at each predetermined position, and at the same time obtaining the current posture of the robotic arm when the collection device collects the 2D image data of the calibration workpiece through the feedback end such as the internal controller of the robotic arm; the predetermined position is arbitrarily set by technical personnel in this field.

[0064] It is worth noting that each predetermined position needs to make the calibration workpiece located within the acquisition range of the acquisition device so that the acquisition device can acquire complete 2D image data of the calibration workpiece.

[0065] S3: Create an initial virtual scene, import the virtual robot arm, the virtual acquisition device and the workpiece digital model according to the current posture of the end of the robot arm at each acquisition point, the acquisition device internal parameter matrix and the first transformation matrix; the workpiece digital model includes the coordinates of several digital model feature points and the corresponding numbers;

[0066] Specifically, it is optional but not limited to that technical personnel in this field may place corresponding virtual acquisition devices, virtual robotic arms and workpiece digital models in the virtual scene according to the positions of the acquisition devices, the end of the robotic arm and the calibration workpiece in the real scene, and correct the current postures of the corresponding virtual acquisition devices and virtual robotic arms according to the postures of the acquisition devices, the end of the robotic arm and the calibration workpiece in the real scene, so as to ensure that the operation of each device in the virtual scene is the same as that in the real scene.

[0067] As an example, it is optional but not limited to that a technician in this field places a virtual robotic arm, a virtual acquisition device and a workpiece digital model at a virtual position corresponding to a predetermined position in a virtual scene, and then calibrates the virtual robotic arm and the virtual acquisition device according to the postures of the robotic arm and the acquisition device at each predetermined position, and configures the acquisition device parameters according to the internal parameter calibration results, so that the image information collected by the acquisition device in the virtual scene is the same as that in the real scene.

[0068] Preferably, the steps of creating a virtual scene and importing a virtual robotic arm, a virtual acquisition device and a workpiece digital model may optionally include:

[0069] S31: Importing the industrial robot arm model, acquisition equipment model and design digital model of the calibration workpiece of the same model as the actual scene into the virtual scene, obtaining the virtual robot arm, virtual acquisition equipment and workpiece digital model, and obtaining the coordinates and corresponding numbers of several digital model feature points on the workpiece digital model;

[0070] Specifically, it is optional but not limited to importing the design digital model of an industrial robot arm model, an acquisition equipment model and a calibrated workpiece of the same model as the actual scene from the virtual software platform library into the virtual scene to obtain a virtual robot arm, a virtual acquisition equipment and a workpiece digital model, and at the same time obtain the coordinates of all set digital model feature points on the workpiece digital model, and number each digital model feature point according to the set order to obtain the coordinates of several digital model feature points and the corresponding numbers; the virtual software platform library may optionally include commonly used virtualization platforms such as VMware, Oracle VirtualBox, KVM, etc., and the setting order may be arbitrarily set by technical personnel in this field.

[0071] S32: Correcting the corresponding posture of the virtual robotic arm in the virtual scene according to the current posture of the robotic arm end when it is located at each acquisition point;

[0072] S33: configuring the virtual acquisition device parameters according to the internal parameter matrix, and obtaining the relative position and posture of the virtual acquisition device in the virtual scene according to the first conversion matrix;

[0073] Specifically, it is optional but not limited to placing the virtual robotic arm at the corresponding position of the virtual scene according to the current posture of the end of the robotic arm when it is located at each acquisition point, and adjusting the posture of the virtual robotic arm when it is located at each acquisition point, and then setting the virtual acquisition device parameters according to the acquisition device internal parameter matrix calibrated in step S1, so that when the virtual acquisition device shoots in the virtual scene, it can achieve the same shooting effect as the acquisition device in the real scene, so as to simulate various operations of the acquisition device in the real scene and reduce costs. At the same time, since the first conversion matrix between the acquisition device and the end of the robotic arm is also obtained in step S1, the corresponding posture of the virtual acquisition device when it is located at each acquisition point can be obtained according to the posture of the virtual robotic arm when it is located at each acquisition point.

[0074] S34: According to the position of the calibrated workpiece in the real scene, a corresponding workpiece digital model is placed in the virtual scene.

[0075] Specifically, it is optional but not limited to that the technicians in this field place the workpiece digital model in the virtual scene according to the position of the calibrated workpiece in the real scene. It is worth noting that since the subsequent steps also include the posture correction of the workpiece digital model in the virtual scene, when the technicians in this field place the workpiece digital model, the position of the workpiece digital model does not deviate too much from the position of the calibrated workpiece in the real scene.

[0076] S4: Obtain the coordinates and corresponding numbers of each image feature point in the 2D image of the calibration workpiece according to the prior region of interest;

[0077] Specifically, it is optional but not limited to extracting various feature areas in the 2D image of the calibration workpiece according to the prior region of interest, such as feature images of circular holes, rectangles, grooves, etc., so as to obtain the coordinates of the center point or corner point of the feature image according to each feature image as the image feature point; in the example, taking the circular hole feature image as an example, the coordinates of the center point of the circle can be used as the image feature point. At the same time, the corresponding numbers of each image feature point are obtained according to the preset number of the prior region of interest; more specifically, the prior region of interest can be optionally but not limited to be obtained according to the calibration workpiece template, structure diagram, etc. of the calibration workpiece.

[0078] S5: construct an objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of each image feature point and digital model feature point, obtain the second conversion matrix between each image feature point and digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device;

[0079] Specifically, it is optional but not limited to constructing an objective function based on the internal parameter matrix of the acquisition device and the coordinates and corresponding numbers of each image feature point and digital model feature point, and preferably solving the objective function according to the LM algorithm, so that when each image feature point is mapped to each digital model feature point or when each digital model feature point is mapped to each image feature point, the overlap between the two is the highest, and the second transformation matrix between each image feature point and the digital model feature point, as well as the current posture of the workpiece digital model under the acquisition device, can be obtained.

[0080] Preferably, the specific steps of obtaining the second transformation matrix between each image feature point and the digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device may include but are not limited to:

[0081] S51: converting each image feature point and digital model feature point into the same scene;

[0082] Specifically, it is optional but not limited to converting each image feature point to the acquisition device coordinate system in the virtual scene, or converting the digital-analog feature point to the calibration workpiece coordinate system in the real scene, thereby converting each image feature point and the digital-analog feature point to the same spatial coordinate system to calculate the rotation and translation matrix between the two, which is the second transformation matrix.

[0083] S52: constructing an objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of the image feature points and the digital-analog feature points;

[0084] S53: Solve the objective function to obtain a second conversion matrix between each image feature point and the digital model feature point, as well as the current position and posture of the workpiece digital model under the acquisition device.

[0085] Specifically, due to process conditions or environmental factors in the real scenario, when collecting image data of the calibration workpiece surface through the acquisition device, the number of set acquisition points varies. Therefore, it is optional but not limited to setting different calculation methods for the transformation matrix according to the number of acquisition points.

[0086] Preferably, the step of constructing the objective function based on the internal parameter matrix of the acquisition device, image feature points, and digital model feature points to obtain the second transformation matrix between each image feature point and digital model feature point may include:

[0087] S521: Determine whether the number of acquisition points is greater than 1. If not, construct the first objective function; if so, construct the second objective function;

[0088] S522: Solve the objective function to obtain the rotation and translation matrix between each image feature point and digital model feature point, which is the second transformation matrix, and the current pose of the workpiece digital model under the acquisition device.

[0089] Preferably, the first objective function can be optionally expressed as Equation 5-1:

[0090]

[0091] where u i , v i are the coordinates of the i-th image feature point, X i , Y i , Z i are the coordinates of the i-th digital model feature point, 0 < i ≤ N, N is the number of image feature points and digital model feature points, R is the rotation matrix between each image feature point and digital model feature point, and t is the translation matrix between each image feature point and digital model feature point. The combination of R and t is the second transformation matrix.

[0092] Further preferably, the second objective function can be optionally expressed as Equation 5-2:

[0093]

[0094] where K is the internal parameter matrix of the acquisition device, i is the i-th acquisition point, N is the total number of acquisition points, 0 < i ≤ N, j is the j-th feature point of the image corresponding to the i-th acquisition point, M is the total number of feature points of the image corresponding to the i-th acquisition point, 0 < j ≤ M, is the pixel coordinate of the j-th image feature point in the i-th acquisition point, is the three-dimensional coordinate of the digital model feature point corresponding to the j-th image feature point in the i-th acquisition point, R is the rotation matrix between each image feature point and digital model feature point, and t is the translation matrix between each image feature point and digital model feature point. The combination of R and t is the second transformation matrix.

[0095] Further preferably, by solving formula 5-2 according to a commonly used nonlinear numerical optimization method, the second transformation matrix can be obtained, and further converted to obtain the current posture of the workpiece digital model under the acquisition device; the nonlinear numerical optimization method can optionally include commonly used algorithms such as LM algorithm and Gauss-Newton algorithm.

[0096] S6: Correct the current posture of the workpiece digital model according to the first transformation matrix and the second transformation matrix to obtain a corrected posture of the workpiece digital model to obtain a final virtual scene.

[0097] Specifically, it is optional but not limited to converting the current pose of the workpiece digital model to the robot base coordinate system according to the first transformation matrix between the acquisition device and the robotic arm, and then converting the current pose of the workpiece digital model according to the second transformation matrix, so that each image feature point and the digital model feature point are aligned, and the corrected pose of the workpiece digital model in the robot base coordinate system can be obtained, so that the relative pose between each device in the virtual scene is the same as the relative pose between each device in the real scene, so that when performing simulation operations in the virtual scene, the same results as in the real scene can be obtained, which greatly saves operating costs, improves operating accuracy, and improves operating efficiency.

[0098] In this embodiment, a virtual-to-real mapping method based on visual feature matching of the present invention is provided. The acquisition device is installed at the end of the robot arm, and the acquisition device internal parameter calibration and hand-eye calibration are performed to obtain the acquisition device internal parameter matrix and the first conversion matrix between the acquisition device and the robot arm, so as to configure the virtual acquisition device parameters in the subsequent steps and improve the operation accuracy of the virtual scene. Then, a number of acquisition points are set, and the robot arm is controlled to move to each acquisition point to acquire and calibrate the 2D image of the workpiece, and the current posture of the end of the robot arm when it is located at each acquisition point is recorded, and then an initial virtual scene is created. According to the current posture of the end of the robot arm when it is located at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix, the virtual robot arm, the virtual acquisition device and the workpiece digital model are imported; the workpiece digital model includes the coordinates and corresponding numbers of a number of digital model feature points, and the image feature points in the 2D image of the calibrated workpiece are obtained according to the prior region of interest. The coordinates and corresponding numbers are constructed according to the internal parameter matrix of the acquisition device, as well as the coordinates and corresponding numbers of each image feature point and digital model feature point, and the objective function is constructed to obtain the second conversion matrix between each image feature point and digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device. Finally, the current position and posture of the workpiece digital model is corrected according to the first conversion matrix and the second conversion matrix, and the corrected position and posture of the workpiece digital model after correction in the robot base coordinate system are obtained to obtain the final virtual scene. By importing all the equipment and workpieces in the real scene into the virtual scene, and initializing the corresponding equipment in the virtual scene according to the relative position and equipment parameters of each equipment in the real scene, the technicians in this field can realize the operations and effects that can be achieved by the same equipment in the real scene by operating various equipment in the virtual scene, which greatly reduces the use cost, reduces the complexity of operation, and can improve the accuracy and stability of the operation through multiple simulations. The problems of high cost, high complexity of operation, low accuracy and stability of the existing technology are solved.

[0099] On the other hand, the present invention also provides a virtual-real mapping method based on point cloud registration:

[0100] P1: Install the acquisition device at the end of the robotic arm, and perform the acquisition device internal parameter calibration and hand-eye calibration to obtain the acquisition device internal parameter matrix and the first conversion matrix between the acquisition device and the robotic arm;

[0101] Specifically, it is optional but not limited to installing the acquisition device at the end of the robotic arm, then calibrating the acquisition device using the intrinsic parameter calibration method to obtain the acquisition device intrinsic parameter matrix, and then calibrating the acquisition device and the end of the robotic arm using the hand-eye calibration method to obtain the first conversion matrix between the acquisition device and the robotic arm; the acquisition device may be optional but not limited to 3D line scan cameras, 3D structured light cameras, TOF cameras, laser scanners, and other commonly used scanning devices that can be used to obtain three-dimensional data of the surface of objects; the hand-eye calibration method may optionally include common methods such as Zhang Zhengyou method, multi-view geometry method, direct linear transformation method and nonlinear least squares optimization method.

[0102] Preferably, the acquisition device is installed at the tool center point at the end of the robotic arm.

[0103] P2: Set several acquisition points, control the robot arm to move to each acquisition point to collect and calibrate the workpiece surface point cloud, obtain several local calibration workpiece point clouds, and record the current posture of the robot arm at each acquisition point;

[0104] Specifically, it is optional but not limited to controlling the robotic arm to move to several predetermined positions respectively, and then collecting surface point cloud data of the calibration workpiece at each predetermined position to obtain several local calibration workpiece point clouds, and at the same time, through the feedback end such as the internal controller of the robotic arm, the current posture of the robotic arm when the collection device collects the surface point cloud data of the calibration workpiece is obtained; the predetermined position is arbitrarily set by technical personnel in this field.

[0105] It is worth noting that each predetermined position needs to make the calibration workpiece located within the acquisition range of the acquisition device so that the acquisition device can acquire complete surface point cloud data of the calibration workpiece.

[0106] P3: According to the first transformation matrix, each local calibration workpiece point cloud is transformed into the robot base coordinate system to obtain the global calibration workpiece point cloud;

[0107] Specifically, it is optional but not limited to converting some partial surface point cloud data of the calibrated workpiece in the acquisition device coordinate system to the robot end coordinate system according to the current posture of the robot when it is located at each predetermined position, and then obtaining partial surface point cloud data of each calibrated workpiece in the robot base coordinate system according to the prior transformation matrix between the robot end and the robot base, and finally splicing several incomplete surface point clouds of the calibrated workpiece into one point cloud to obtain a complete point cloud of the calibrated workpiece in the robot base coordinate system.

[0108] Preferably, since there may be many repeated superimposed points in the complete calibration workpiece point cloud, it is necessary to optimize it to reduce the amount of data processing in subsequent steps, thereby saving resources and improving efficiency. Therefore, step P3 also includes:

[0109] P31: Filter and / or downsample the calibration workpiece point cloud to obtain optimized calibration workpiece point cloud data.

[0110] Specifically, since there may be repeated point clouds in several partial point clouds of the calibration workpiece surface acquired by the acquisition device, in order to reduce the amount of data processing in subsequent steps, after obtaining the complete calibration workpiece point cloud, the calibration workpiece point cloud can be filtered according to the straight-through filtering method to remove outliers and noise points in the point cloud, and then the point cloud is uniformly down-sampled to simplify the point cloud data structure, so as to obtain the processed point cloud data of the calibration workpiece, which is the optimized point cloud data of the calibration workpiece.

[0111] It is worth noting that parameters such as the filtering range in the straight-through filtering method and the downsampling rate in the uniform downsampling method are arbitrarily set by those skilled in the art.

[0112] P4: Create the initial virtual scene, import the virtual robot arm, virtual acquisition device and workpiece digital model according to the current posture of the end of the robot arm at each acquisition point, the internal parameter matrix of the acquisition device and the first transformation matrix;

[0113] Specifically, it is optional but not limited to creating an initial virtual scene according to a virtual-reality mapping method based on visual feature matching.

[0114] P5: Control the virtual robot arm to move to each acquisition point to scan the surface of the workpiece digital model, obtain several local point clouds of the workpiece digital model, and transform each local point cloud of the workpiece digital model to the robot base coordinate system according to the first transformation matrix to obtain the global workpiece digital model point cloud;

[0115] Specifically, it is optional but not limited to controlling the virtual robotic arm to move to each collection point to collect the workpiece digital model surface point cloud, obtain several workpiece digital model local point clouds, and convert each workpiece digital model local point cloud to the robot base coordinate system according to the first transformation matrix, and splice them to obtain the global workpiece digital model point cloud.

[0116] P6: Perform point cloud registration on the global calibration workpiece point cloud and the global workpiece digital model point cloud to obtain the second transformation matrix between the global calibration workpiece point cloud and the global workpiece digital model point cloud, as well as the current position and posture of the workpiece digital model;

[0117] Specifically, it is optional but not limited to calculating the rotation and translation matrix between the calibration workpiece point cloud and the workpiece digital model point cloud through the global feature point cloud registration algorithm, which is the second transformation matrix, and the current position and posture of the workpiece digital model in the robot base coordinate system; the global feature point cloud registration algorithm can optionally include the RANSAC algorithm, Teaser++ algorithm, global optimization iterative closest point algorithm, fast global registration algorithm, super four-point coplanar algorithm and other commonly used algorithms. More specifically, the global point cloud registration algorithm preferably adopts the RANSAC algorithm.

[0118] Preferably, the step of obtaining the second conversion matrix may optionally include but is not limited to:

[0119] P61: Calculate the feature descriptors of each point in the calibration workpiece point cloud and the workpiece digital model point cloud to construct several feature point pairs;

[0120] P62: Construct the least squares equation based on the feature point pairs to obtain the second transformation matrix;

[0121] Specifically, it is optional but not limited to calculating the feature descriptors of each point in the calibration workpiece point cloud and the workpiece digital model point cloud through a point cloud feature extraction algorithm, and then forming a pair of feature point pairs with the points in the calibration workpiece point cloud and the workpiece digital model point cloud that are closest to the feature descriptors. By traversing the calibration workpiece point cloud and the workpiece digital model point cloud, a number of feature point pairs can be obtained, and a least squares formula is constructed based on the feature point pairs to obtain a second transformation matrix; the point cloud feature extraction algorithm may optionally include common algorithms such as fast point feature histogram algorithm, normal vector calculation method, super point graph algorithm, curvature descriptor algorithm, etc.

[0122] Preferably, since the global feature point cloud registration algorithm is mainly based on the global features of the point cloud, it may not be able to capture local details. Therefore, the preliminary transformation matrix may have large errors in some areas, which may lead to local optimality rather than global optimality, especially when the point cloud has complex structures or repeated features. At the same time, it may be affected by noise and outliers, resulting in inaccurate transformation matrix. Therefore, after determining the second transformation matrix between the calibration workpiece point cloud and the current point cloud, it may also optionally include:

[0123] P63: Perform precise alignment on the second transformation matrix to obtain an optimized second transformation matrix.

[0124] Specifically, it is optional but not limited to using an iterative closest point (ICP) algorithm to accurately align the second transformation matrix according to the calibration workpiece point cloud and the current point cloud to obtain an optimized second transformation matrix.

[0125] P7: Correct the current posture of the workpiece digital model according to the first transformation matrix and the second transformation matrix to obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

[0126] Specifically, it is optional but not limited to converting the current pose of the workpiece digital model to the robot base coordinate system according to the first transformation matrix between the acquisition device and the robotic arm, and then converting the current pose of the workpiece digital model according to the second transformation matrix, so that the workpiece digital model point cloud is aligned with the calibrated workpiece point cloud, and the corrected pose of the workpiece digital model in the robot base coordinate system can be obtained, so that the relative pose between each device in the virtual scene is the same as the relative pose between each device in the real scene, so that when performing simulation operations in the virtual scene, the same results as in the real scene can be obtained, which greatly saves operating costs, improves operating accuracy, and improves operating efficiency.

[0127] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned virtual-reality mapping methods based on visual feature matching or virtual-reality mapping methods based on 3D point cloud feature matching.

[0128] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned virtual-to-real mapping methods based on visual feature matching or virtual-to-real mapping methods based on 3D point cloud feature matching.

[0129] On the other hand, Figure 3 As shown, the present invention also provides a device for a virtual-reality mapping method based on visual feature matching, which is used to execute any of the above virtual-reality mapping methods based on visual feature matching, including:

[0130] The calibration module 101 is used for the internal parameter calibration of the acquisition device and the hand-eye calibration to obtain the internal parameter matrix of the acquisition device and the first conversion matrix between the acquisition device and the robotic arm;

[0131] The image acquisition module 102 is used to set a number of acquisition points, control the robot arm to move to each acquisition point to acquire a 2D image of the calibration workpiece, and record the current posture of the end of the robot arm when it is located at each acquisition point;

[0132] The virtual scene module 103 is used to create an initial virtual scene, and import the virtual robot arm, the virtual acquisition device and the workpiece digital model according to the current posture of the end of the robot arm at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix; the workpiece digital model includes the coordinates of several digital model feature points and the corresponding numbers;

[0133] Optionally, the virtual scene module 103 includes:

[0134] The import unit is used to import the industrial robot arm model, acquisition equipment model and the design digital model of the calibrated workpiece of the same model as the actual scene into the virtual scene, obtain the virtual robot arm, virtual acquisition equipment and workpiece digital model, and obtain the coordinates and corresponding numbers of several digital model feature points on the workpiece digital model;

[0135] A correction unit, used to correct the corresponding posture of the virtual robotic arm in the virtual scene according to the current posture of the end of the robotic arm when it is located at each acquisition point;

[0136] A collection unit, configured to configure the parameters of the virtual collection device according to the internal parameter matrix, and obtain the relative position and posture of the virtual collection device in the virtual scene according to the first conversion matrix;

[0137] The placement unit is used to place the corresponding workpiece digital model in the virtual scene according to the position of the workpiece calibrated in the real scene.

[0138] The computing module 104 includes:

[0139] A feature point acquisition unit is used to obtain the coordinates and corresponding numbers of each image feature point in the 2D image of the calibration workpiece according to the prior region of interest;

[0140] A posture calculation unit is used to construct an objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of each image feature point and the digital model feature point, and obtain a second conversion matrix between each image feature point and the digital model feature point, as well as the current posture of the workpiece digital model under the acquisition device;

[0141] Optionally, the posture calculation unit includes:

[0142] A feature point conversion element is used to convert each image feature point and digital-analog feature point into the same scene;

[0143] The objective function construction component is used to construct the objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of the image feature points and the digital-analog feature points;

[0144] The first function solving element is used to solve the target function to obtain the second conversion matrix between each image feature point and the digital model feature point, as well as the current position and posture of the workpiece digital model under the acquisition device.

[0145] A judging element, used to judge whether the number of acquisition points is greater than 1, if not, construct a first objective function, and if so, construct a second objective function;

[0146] The second function solving element is used to solve the target function to obtain the rotation and translation matrix between each image feature point and the digital model feature point, which is the second transformation matrix, and the current position and posture of the workpiece digital model under the acquisition device.

[0147] Optionally, the first objective function is expressed as:

[0148]

[0149] where u i , v i are the coordinates of the i-th image feature point, X i , Y i , Z i are the coordinates of the i-th digital model feature point, 0 < i ≤ N, N is the number of image feature points and digital model feature points, R is the rotation matrix between each image feature point and digital model feature point, and t is the translation matrix between each image feature point and digital model feature point; the combination of R and t is the second transformation matrix;

[0150] The second objective function is expressed as Equation 5-2:

[0151]

[0152] where K is the internal parameter matrix of the acquisition device, i is the i-th acquisition point position, N is the total number of acquisition point positions, 0 < i ≤ N, j is the j-th feature point of the image corresponding to the i-th acquisition point position, M is the total number of feature points of the image corresponding to the i-th acquisition point position, 0 < j ≤ M, is the pixel coordinate of the j-th image feature point in the i-th acquisition point position, is the three-dimensional coordinate of the digital model feature point corresponding to the j-th image feature point in the i-th acquisition point position, R is the rotation matrix between each image feature point and digital model feature point, and t is the translation matrix between each image feature point and digital model feature point; the combination of R and t is the second transformation matrix.

[0153] The pose correction unit is used to correct the current pose of the workpiece digital model according to the first transformation matrix and the second transformation matrix, and obtain the corrected workpiece digital model pose in the robot base coordinate system, so as to obtain the final virtual scene.

[0154] Exemplarily, the program code can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the program code in the terminal device.

[0155] The terminal device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the terminal device may further include input / output devices, network access devices, buses, etc.

[0156] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0157] The memory may be an internal storage unit of the terminal device, such as a hard disk or a memory. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store the program code and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0158] The above-mentioned computer storage medium, terminal equipment and device are created based on the above-mentioned virtual-reality mapping method based on visual feature matching or the virtual-reality mapping method based on 3D point cloud feature matching. Their technical functions and beneficial effects are no longer repeated here. The various technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the various technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A virtual-real mapping method based on visual feature matching, characterized in that: include: The acquisition device is installed at the end of the robotic arm, and the internal parameter calibration and hand-eye calibration of the acquisition device are performed to obtain the internal parameter matrix of the acquisition device and the first conversion matrix between the acquisition device and the robotic arm; Set several acquisition points, control the robot arm to move to each acquisition point to acquire the 2D image of the calibration workpiece, and record the current position of the end of the robot arm when it is at each acquisition point; Create an initial virtual scene, import the virtual robot, virtual acquisition device and workpiece digital model according to the current posture of the end of the robot at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix; Digital model, including the coordinates of several digital model feature points and their corresponding numbers; According to the a priori region of interest, the coordinates and corresponding numbers of each image feature point in the 2D image of the calibration workpiece are obtained; According to the internal parameter matrix of the acquisition device, as well as the coordinates and corresponding numbers of each image feature point and digital model feature point, the objective function is constructed to obtain the second conversion matrix between each image feature point and digital model feature point, as well as the current position and posture of the workpiece digital model under the acquisition device; The current posture of the workpiece digital model is corrected according to the first conversion matrix and the second conversion matrix to obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

2. The method according to claim 1, characterized in that The steps to create a virtual scene include: Import the industrial robot arm model, acquisition equipment model and the design digital model of the calibrated workpiece of the same model as the actual scene into the virtual scene, obtain the virtual robot arm, virtual acquisition equipment and workpiece digital model, and obtain the coordinates and corresponding numbers of several digital model feature points on the workpiece digital model; According to the current posture of the end of the robot arm when it is located at each acquisition point, the corresponding posture of the virtual robot arm in the virtual scene is corrected; The virtual acquisition device parameters are configured according to the internal parameter matrix, and the relative position and posture of the virtual acquisition device in the virtual scene is obtained according to the first conversion matrix; According to the position of the calibrated workpiece in the real scene, the corresponding workpiece digital model is placed in the virtual scene.

3. The method according to claim 1, characterized in that The step of obtaining a second conversion matrix between each image feature point and the digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device includes: Convert each image feature point and digital model feature point into the same scene; Construct the objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of the image feature points and the digital model feature points; The objective function is solved to obtain the second transformation matrix between each image feature point and the digital model feature point, as well as the current position and posture of the workpiece digital model under the acquisition device.

4. The method according to claim 3, characterized in that The step of obtaining a second conversion matrix between each image feature point and the digital model feature point, and the current position and posture of the workpiece digital model under the acquisition device also includes: Determine whether the number of acquisition points is greater than 1, if not, construct the first objective function, if yes, construct the second objective function; The objective function is solved to obtain the rotation and translation matrix between each image feature point and the digital model feature point, which is the second transformation matrix, and the current position and posture of the workpiece digital model under the acquisition device.

5. The method according to claim 4, characterized in that The first objective function is expressed as: where u i , v i are the coordinates of the i-th image feature point, X i , Y i , Z i are the coordinates of the i-th digital model feature point, 0 < i ≤ N, N is the number of image feature points and digital model feature points, R is the rotation matrix between each image feature point and digital model feature point, t is the translation matrix between each image feature point and digital model feature point; the combination of R and t is the second transformation matrix; The second objective function is expressed as formula 5-2: Among them, K is the internal parameter matrix of the acquisition device, i is the i-th acquisition point, N is the total number of acquisition points, 0 < i ≤ N, j is the j-th feature point of the image corresponding to the i-th acquisition point, M is the total number of feature points of the image corresponding to the i-th acquisition point, 0 < j ≤ M, is the pixel coordinate of the j-th image feature point in the i-th acquisition point, is the three-dimensional coordinate of the digital-analog feature point corresponding to the j-th image feature point in the i-th acquisition point, R is the rotation matrix between each image feature point and the digital-analog feature point, t is the translation matrix between each image feature point and the digital-analog feature point; the combination of R and t is the second transformation matrix.

6. A virtual-real mapping method based on 3D point cloud feature matching, characterized in that: include: The acquisition device is installed at the end of the robotic arm, and the internal parameter calibration and hand-eye calibration of the acquisition device are performed to obtain the internal parameter matrix of the acquisition device and the first conversion matrix between the acquisition device and the robotic arm; Set several acquisition points, control the robot arm to move to each acquisition point to collect and calibrate the workpiece surface point cloud, obtain several local calibration workpiece point clouds, and record the current posture of the robot arm when it is located at each acquisition point; According to the first transformation matrix, each local calibration workpiece point cloud is transformed into the robot base coordinate system to obtain a global calibration workpiece point cloud; Create an initial virtual scene, import the virtual robot arm, virtual acquisition device and workpiece digital model according to the current posture of the end of the robot arm when it is located at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix; Control the virtual robot arm to move to each acquisition point to scan the surface of the workpiece digital model, obtain several partial point clouds of the workpiece digital model, and transform each partial point cloud of the workpiece digital model into the robot base coordinate system according to the first transformation matrix to obtain the global workpiece digital model point cloud; Performing point cloud registration on the global calibration workpiece point cloud and the global workpiece digital model point cloud to obtain a second transformation matrix between the global calibration workpiece point cloud and the global workpiece digital model point cloud, as well as the current position and posture of the workpiece digital model; The current posture of the workpiece digital model is corrected according to the first conversion matrix and the second conversion matrix to obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

7. The method according to claim 6, characterized in that The step of obtaining a second transformation matrix comprises: Calculate the feature descriptors of each point in the calibration workpiece point cloud and the workpiece digital model point cloud to construct a number of feature point pairs; Construct the least squares equation based on the feature point pairs to obtain the second transformation matrix; The second transformation matrix is ​​precisely aligned to obtain an optimized second transformation matrix.

8. A computer storage medium, characterized in that An executable program code is stored; the executable program code is used to execute the virtual-to-real mapping method based on visual feature matching as described in any one of claims 1-5 or the virtual-to-real mapping method based on 3D point cloud feature matching as described in any one of claims 6-7.

9. A terminal device, characterized in that: It comprises a memory and a processor; the memory stores program code executable by the processor; the program code is used to execute the virtual-to-real mapping method based on visual feature matching as described in any one of claims 1-5 or the virtual-to-real mapping method based on 3D point cloud feature matching as described in any one of claims 6-7.

10. A device for a virtual-real mapping method based on visual feature matching, characterized in that: include: A calibration module, used for acquisition device internal parameter calibration and hand-eye calibration, to obtain the acquisition device internal parameter matrix and the first conversion matrix between the acquisition device and the robotic arm; The image acquisition module is used to set a number of acquisition points, control the robot arm to move to each acquisition point to acquire a 2D image of the calibration workpiece, and record the current position of the end of the robot arm when it is located at each acquisition point; The virtual scene module is used to create an initial virtual scene, import the virtual robot arm, the virtual acquisition device and the workpiece digital model according to the current posture of the end of the robot arm when it is located at each acquisition point, the acquisition device internal parameter matrix and the first conversion matrix; Workpiece Digital model, including the coordinates of several digital model feature points and their corresponding numbers; Computing module, including: A feature point acquisition unit is used to obtain the coordinates and corresponding numbers of each image feature point in the 2D image of the calibration workpiece according to the prior region of interest; A posture calculation unit is used to construct an objective function according to the internal parameter matrix of the acquisition device, the coordinates and corresponding numbers of each image feature point and the digital model feature point, and obtain a second conversion matrix between each image feature point and the digital model feature point, as well as the current posture of the workpiece digital model under the acquisition device; The posture correction unit is used to correct the current posture of the workpiece digital model according to the first conversion matrix and the second conversion matrix, and obtain the corrected posture of the workpiece digital model in the robot base coordinate system to obtain the final virtual scene.

Images