Positioning method and device and electronic equipment
By adopting a positioning method combining 2D and 3D in a complex industrial production environment, the two-dimensional image is first segmented in instances, and then segmenting the target instance objects from the three-dimensional point cloud, the problem of low positioning efficiency caused by high target and background similarity is solved, and more efficient positioning and segmentation is achieved.
Patent Information
- Application Number
- CN202411973112.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
In a complex industrial production environment, the target instance objects have a high similarity to the background and are difficult to distinguish. The prior art directly locates and segments on 3D data, with high algorithm complexity and low positioning efficiency.
Using a combination of 2D and 3D, the two-dimensional image is first segmented in an instance to obtain the target mask map, and then, based on the mapping relationship between the two-dimensional and three-dimensional coordinate points, the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud.
It reduces the complexity of the algorithm, improves positioning efficiency, can handle complex scenes more effectively, and avoids calibration operations when the camera's internal parameter matrix is unknown.
Smart Images
Figure CN119850731A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology and industrial vision technology, and in particular relates to a positioning method, device and electronic equipment. Background Art
[0002] In modern intelligent production lines, robotic grasping is a common task. Before grasping, it is necessary to first take a picture with a camera to estimate the pose information (6D pose, including displacement and rotation pose) of the target instance object (such as an industrial component) to be grasped in the scene. The quality of the pose estimation result directly affects the accuracy of the overall process of robotic grasping.
[0003] In industrial production environments, target scenes are relatively complex (stacking, occlusion, and similar colors), and the target instance objects and the background are highly similar and difficult to distinguish. The positioning methods of related technologies directly locate and segment the target instance objects on the 3D data of the target scene. The algorithm complexity is high and the positioning efficiency is low. Summary of the invention
[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a positioning method, device and electronic device, which adopts a method combining 2D and 3D to first segment a target mask map of a target instance object from a 2D image, and then segment a target 3D point cloud instance of the target instance object from a 3D point cloud, thereby effectively reducing the algorithm complexity and improving the positioning efficiency.
[0005] In a first aspect, the present application provides a positioning method, the method comprising:
[0006] Acquire a two-dimensional image and a three-dimensional point cloud of a target scene; the target scene includes a target instance object;
[0007] Calling the trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image to obtain an instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes a target mask image of the target instance object;
[0008] According to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask image, a target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud;
[0009] A target instance object is located in a target scene according to the target 3D point cloud instance.
[0010] According to the positioning method of the present application, the instance segmentation processing of the two-dimensional image is performed by calling the trained two-dimensional instance segmentation model to obtain the instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes the target mask map of the target instance object; according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask map, the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud. Compared with directly positioning and segmenting the target three-dimensional point cloud instance of the target instance object in the three-dimensional point cloud, the target mask map of the target instance object is first segmented from the two-dimensional image, and then the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud based on the mapping relationship. The segmentation method is more efficient and has lower computational complexity. In addition, the two-dimensional image is processed and the deep learning instance segmentation method is introduced, which can improve the efficiency of the overall solution and better cope with complex scenes. In addition, the method of combining 2D and 3D is adopted. Compared with the common method of using RGBD as input to directly estimate the pose, the process does not rely on the intrinsic parameter matrix of the camera, avoiding the need to calibrate the camera when the intrinsic parameter matrix is unknown.
[0011] According to one embodiment of the present application, the two-dimensional instance segmentation model is trained based on the following method:
[0012] Constructing a three-dimensional virtual simulation scene of a target scene; the three-dimensional virtual simulation scene includes a three-dimensional virtual instance object of a target instance object;
[0013] Collecting and rendering a large number of sample images containing three-dimensional virtual instance objects from a three-dimensional virtual simulation scene; the three-dimensional virtual instance objects in the sample images are marked with external frames;
[0014] Generate a sample mask image of the three-dimensional virtual instance object in each sample image;
[0015] The sample images are used as training samples, and the categories of the target instance objects, the bounding boxes in each sample image, and the sample mask images are used as training labels to train the initial two-dimensional instance segmentation model to obtain a trained two-dimensional instance segmentation model.
[0016] According to an embodiment of the present application, a large number of sample images containing three-dimensional virtual instance objects are collected and rendered from a three-dimensional virtual simulation scene, including:
[0017] Updating target parameters in the three-dimensional virtual simulation scene; the target parameters include at least one of background, lighting, texture, position and posture of the three-dimensional virtual instance object;
[0018] According to the updated target parameters, a large number of sample images containing three-dimensional virtual instance objects are collected and rendered from the three-dimensional virtual simulation scene.
[0019] According to one embodiment of the present application, locating a target instance object in a target scene according to a target three-dimensional point cloud instance includes:
[0020] Performing feature matching on the target 3D point cloud instance and the reference 3D point cloud instance of the target instance object to obtain relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance;
[0021] The target instance object is located in the target scene according to the relative pose information.
[0022] According to one embodiment of the present application, feature matching is performed on a target 3D point cloud instance and a reference 3D point cloud instance of the target instance object to obtain relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance, including:
[0023] Obtain a first feature descriptor of a reference three-dimensional point cloud instance;
[0024] Generate a second feature descriptor of the target three-dimensional point cloud instance;
[0025] Performing feature matching on the first feature descriptor and the second feature descriptor to obtain a point correspondence relationship between the reference three-dimensional point cloud instance and the target three-dimensional point cloud instance;
[0026] According to the point correspondence, the relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance is obtained.
[0027] According to one embodiment of the present application, locating a target instance object in a target scene according to relative pose information includes:
[0028] Obtaining target pose information of the target instance object in the target scene according to the reference pose information and relative pose information of the reference 3D point cloud instance;
[0029] According to the target pose information, the target instance object is located in the target scene.
[0030] According to an embodiment of the present application, according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask map, segmenting the target three-dimensional point cloud instance of the target instance object from the three-dimensional point cloud includes:
[0031] Determine a target three-dimensional coordinate point that has a mapping relationship with a two-dimensional coordinate point in a target mask image from the three-dimensional point cloud;
[0032] A point cloud instance composed of each target three-dimensional coordinate point is determined as a target three-dimensional point cloud instance of a target instance object.
[0033] In a second aspect, the present application provides a positioning device, the device comprising:
[0034] An acquisition module, used for acquiring a two-dimensional image and a three-dimensional point cloud of a target scene; the target scene includes a target instance object;
[0035] The first processing module is used to call the trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image to obtain the instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes a target mask image of the target instance object;
[0036] A second processing module is used to segment a target three-dimensional point cloud instance of a target instance object from the three-dimensional point cloud according to a mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud and a target mask image;
[0037] The third processing module is used to locate the target instance object in the target scene according to the target three-dimensional point cloud instance.
[0038] According to the positioning device of the present application, the instance segmentation processing is performed on the two-dimensional image by calling the trained two-dimensional instance segmentation model to obtain the instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes the target mask map of the target instance object; according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask map, the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud. Compared with directly positioning and segmenting the target three-dimensional point cloud instance of the target instance object in the three-dimensional point cloud, the target mask map of the target instance object is first segmented from the two-dimensional image, and then the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud based on the mapping relationship. The segmentation method is more efficient and has lower computational complexity. In addition, the two-dimensional data is processed and the deep learning instance segmentation method is introduced, which can improve the efficiency of the overall solution and better cope with complex scenes. In addition, the method of combining 2D and 3D is used. Compared with the common method of using RGBD as input to directly estimate the pose, the process does not rely on the intrinsic parameter matrix of the camera, avoiding the need to calibrate the camera when the intrinsic parameter matrix is unknown.
[0039] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the positioning method provided in the first aspect is implemented.
[0040] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the positioning method provided in the first aspect described above is implemented.
[0041] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the positioning method provided in the first aspect.
[0042] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the positioning method provided in the first aspect above.
[0043] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0044] By calling a trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image, an instance segmentation result output by the two-dimensional instance segmentation model is obtained; the instance segmentation result includes a target mask map of the target instance object; according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask map, a target three-dimensional point cloud instance of the target instance object is obtained from the three-dimensional point cloud. Compared with directly locating and segmenting the target three-dimensional point cloud instance of the target instance object in the three-dimensional point cloud, the target mask map of the target instance object is first segmented from the two-dimensional image, and then the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud based on the mapping relationship. The segmentation method is more efficient and has lower computational complexity. In addition, the two-dimensional data is processed and the deep learning instance segmentation method is introduced, which can improve the efficiency of the overall solution and better cope with and process complex scenes.
[0045] In addition, compared with the common method of using RGBD as input to directly estimate the pose, the method that combines 2D and 3D does not rely on the camera's intrinsic parameter matrix, avoiding the need to calibrate the camera when the intrinsic parameter matrix is unknown.
[0046] By constructing a three-dimensional virtual simulation scene and updating the target parameters in the three-dimensional virtual simulation scene, rich training samples can be obtained without manual labeling. In addition, the training samples can be made closer to reality and richer, simulating data changes in reality and improving background complexity.
[0047] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0049] Figure 1 This is one of the flowcharts of the positioning method provided in the embodiment of the present application;
[0050] Figure 2 is a schematic diagram of a two-dimensional image provided in an embodiment of the present application;
[0051] Figure 3 is a schematic diagram of a target scenario provided by an embodiment of the present application;
[0052] Figure 4 is a schematic diagram of a target two-dimensional mask image of a two-dimensional image provided in an embodiment of the present application;
[0053] Figure 5a and Figure 5b It is a schematic diagram of the scene after the disorderly falling process of the three-dimensional virtual instance object and falling on the ground in the three-dimensional virtual simulation scene provided by the embodiment of the present application;
[0054] Figure 6 This is one of the flowcharts of positioning provided in the embodiments of the present application;
[0055] Figure 7 is a schematic diagram of the structure of a positioning device provided in an embodiment of the present application;
[0056] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0058] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0059] In industrial production environments, target scenes are relatively complex (stacking, occlusion, and similar colors), and the target instance objects and the background are highly similar and difficult to distinguish. The positioning methods of related technologies directly locate and segment the target instance objects on the 3D data of the target scene. The algorithm complexity is high and the positioning efficiency is low.
[0060] like Figure 3 As shown, an embodiment of the present application provides a schematic diagram of a target scene. It can be seen that the colors of these components are relatively close, and there is stacking and occlusion between the components. In this scene, how to quickly and accurately locate the target industrial components is an urgent problem to be solved.
[0061] To solve the above problems, embodiments of the present application provide a positioning method, a positioning device, an electronic device and a readable storage medium.
[0062] The positioning method, positioning device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0063] The positioning method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.
[0064] The positioning method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the positioning method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The positioning method provided in the embodiments of the present application is described below using an electronic device as an example of an execution subject.
[0065] like Figure 1 As shown, the positioning method includes: step 110, step 120, step 130 and step 140.
[0066] Step 110: Acquire a two-dimensional image and a three-dimensional point cloud of a target scene; the target scene includes a target instance object.
[0067] The target scene of the embodiment of the present application can be any scene that contains a target instance object. In the field of industrial vision, the target scene can be a production scene, usage scene, etc. of the target instance object, and there is no limitation on this.
[0068] The target instance object of the embodiment of the present application can be an instance object (person, object) in any field. For example, in the field of industrial vision, the target instance object can be an industrial component, and the number can be one or more. In actual applications, the target instance object can be an industrial component grasped in a robotic arm grasping task.
[0069] In the embodiments of the present application, a two-dimensional image and a three-dimensional point cloud of a target scene can be captured by a visual camera. The two-dimensional image and the three-dimensional point cloud of the target scene correspond to each moment. Each two-dimensional image has a corresponding three-dimensional point cloud captured based on the same camera perspective. Therefore, based on the corresponding relationship of the same camera perspective, a mapping relationship between the two-dimensional coordinate points of each two-dimensional image and the corresponding three-dimensional coordinate points in the three-dimensional point cloud can be established.
[0070] like Figure 2 As shown, an embodiment of the present application provides a schematic diagram of a two-dimensional image, which includes 7 industrial components, and these industrial components can all be used as target instance objects.
[0071] Step 120: call the trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image to obtain an instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes a target mask image of the target instance object.
[0072] For each two-dimensional image, the embodiment of the present application can call a trained two-dimensional instance segmentation model (2DInstance Segmentation Model) to perform instance segmentation processing on the two-dimensional image to obtain an instance segmentation result output by the two-dimensional instance segmentation model.
[0073] The 2D instance segmentation model is a computer vision task that aims to simultaneously identify each object instance in an image and accurately segment it. Unlike traditional object detection tasks, instance segmentation is not just about identifying the category of an object (such as "dog", "car" or "industrial part"), but also requires providing accurate pixel-level segmentation for each object instance in the image, that is, distinguishing different instances of the same class of objects.
[0074] The training samples of the two-dimensional instance segmentation model in the embodiment of the present application can be sample images containing target instance objects, and the training labels can be the category of the target instance object, the target instance object external frame in each sample image, and the sample mask map. The method of obtaining training samples can be found in the subsequent part.
[0075] The instance segmentation result of the embodiment of the present application may include the category, bounding box and target mask image of the target instance object.
[0076] The bounding box refers to an outer rectangular box, which is used to mark the position and size of the target instance object in the image and to mark the boundary range of the target instance object.
[0077] The mask image refers to a binary image, in which the "mask" part (value 1 or 255) represents the area that needs to be processed, which refers to the area where the target instance object is located in the embodiment of the present application, and the "non-mask" part (value 0 or other) represents the area that does not need to be processed, which refers to other areas except the area where the target instance object is located in the embodiment of the present application. The role of the mask image is to limit certain operations to only a part of the image area and ignore other parts.
[0078] The mask image of the target instance object in the embodiment of the present application is called a target mask image. The size of the target mask image is the same as the size of the two-dimensional image. The mask part in the target mask image is the area corresponding to the target instance object.
[0079] like Figure 4 As shown, the embodiment of the present application provides a schematic diagram of a target two-dimensional mask image of a two-dimensional image, and the two-dimensional image is Figure 2 The image in .
[0080] In practical applications, the two-dimensional instance segmentation model can be trained with reference to the Mask R-CNN model (an extension based on Faster R-CNN, which not only generates bounding boxes but also generates pixel-level masks for each object instance), the YOLACT model (a lightweight instance segmentation model that achieves efficient instance segmentation at a fast speed by generating prototype masks and coefficients), etc. It can also be trained with reference to other models, without limitation.
[0081] Step 130 : segment the target 3D point cloud instance of the target instance object from the 3D point cloud according to the mapping relationship between the 2D coordinate points of the 2D image and the 3D coordinate points in the 3D point cloud, and the target mask image.
[0082] After obtaining the target mask map of the target instance object, the embodiment of the present application can determine the target three-dimensional coordinate points that have a mapping relationship with the two-dimensional coordinate points in the target mask map from the three-dimensional point cloud. The point cloud instance composed of these target three-dimensional coordinate points is the target three-dimensional point cloud instance of the target instance object, thereby realizing instance segmentation of the three-dimensional point cloud.
[0083] Step 140: locate the target instance object in the target scene according to the target three-dimensional point cloud instance.
[0084] After determining the target three-dimensional point cloud instance of the target instance object from the three-dimensional point cloud of the target scene, the embodiment of the present application can locate the target instance object in the target scene based on the target three-dimensional point cloud instance. Specifically, feature matching can be performed on the target three-dimensional point cloud instance and the benchmark three-dimensional point cloud instance of the target instance object to obtain relative pose information between the target three-dimensional point cloud instance and the benchmark three-dimensional point cloud instance; the relative pose information can be 6D pose information, including position information and rotation pose information. Then, the target instance object can be located in the target scene based on the relative pose information.
[0085] The embodiment of the present application performs instance segmentation processing on a two-dimensional image by calling a trained two-dimensional instance segmentation model to obtain an instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes a target mask map of the target instance object; according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask map, a target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud. Compared with directly locating and segmenting the target three-dimensional point cloud instance of the target instance object in the three-dimensional point cloud, the target mask map of the target instance object is first segmented from the two-dimensional image, and then the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud based on the mapping relationship. The segmentation method is more efficient and has lower computational complexity; in addition, processing the two-dimensional data and introducing a deep learning instance segmentation method can improve the efficiency of the overall solution and better cope with complex scenes. In addition, the method of combining 2D and 3D is used. Compared with the common method of using RGBD as input to directly estimate the pose, the process does not rely heavily on the intrinsic parameter matrix of the camera, thereby avoiding the need to calibrate the camera when the intrinsic parameter matrix is unknown.
[0086] In some embodiments, locating a target instance object in a target scene according to the target three-dimensional point cloud instance includes:
[0087] Performing feature matching on the target 3D point cloud instance and the reference 3D point cloud instance of the target instance object to obtain relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance;
[0088] The target instance object is located in the target scene according to the relative pose information.
[0089] After determining the target three-dimensional point cloud instance of the target instance object, the embodiment of the present application can perform feature matching on the target three-dimensional point cloud instance and the benchmark three-dimensional point cloud instance of the target instance object. The benchmark three-dimensional point cloud instance is also called the template three-dimensional point cloud instance, which is a reference object. By performing feature matching on the two, the relative pose information between the target three-dimensional point cloud instance and the benchmark three-dimensional point cloud instance can be obtained.
[0090] The relative posture information package includes relative position (△x, △y, △z) and relative rotation posture (△E, △R, △Q), where △x represents the relative position in the X-axis direction, △y represents the relative position in the Y-axis direction, △z represents the relative position in the Z-axis direction, △E represents the change angle of Euler Angles, △R represents the rotation matrix, and △Q represents the quaternion.
[0091] After the relative position information is obtained, the target instance object can be located in the target scene according to the relative posture information.
[0092] In some embodiments, feature matching is performed on a target 3D point cloud instance and a reference 3D point cloud instance of the target instance object to obtain relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance, including:
[0093] Obtain a first feature descriptor of a reference three-dimensional point cloud instance;
[0094] Generate a second feature descriptor of the target three-dimensional point cloud instance;
[0095] Performing feature matching on the first feature descriptor and the second feature descriptor to obtain a point correspondence relationship between the reference three-dimensional point cloud instance and the target three-dimensional point cloud instance;
[0096] According to the point correspondence, the relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance is obtained.
[0097] In the process of feature matching between a target three-dimensional point cloud instance and a reference three-dimensional point cloud instance of a target instance object, an embodiment of the present application can obtain a first feature descriptor of the reference three-dimensional point cloud instance, which can be pre-generated. At the same time, a second feature descriptor of the target three-dimensional point cloud instance can also be generated. Each feature descriptor can be generated using a DeepLearning-Based Descriptors algorithm or a Point Feature Histograms (PFH) algorithm.
[0098] The feature descriptor describes and represents the 3D coordinate points in a 3D point cloud or 3D point cloud instance to facilitate subsequent matching, recognition, classification and other tasks. The feature descriptor is usually a vector or digital representation that can effectively capture the local information of a specific area or point, so that the point can be identified and compared under different images, angles and lighting conditions.
[0099] The first feature descriptor and the second feature descriptor can be feature matched to obtain a point correspondence between the reference 3D point cloud instance and the target 3D point cloud instance. The point correspondence can be a correspondence between the 3D coordinate points of the two. Then, using coarse and fine registration methods such as RANSAC and ICP, the relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance can be obtained.
[0100] In some embodiments, locating the target instance object in the target scene according to the relative pose information includes:
[0101] Obtaining target pose information of the target instance object in the target scene according to the reference pose information and relative pose information of the reference 3D point cloud instance;
[0102] According to the target pose information, the target instance object is located in the target scene.
[0103] After obtaining the relative pose information between the target three-dimensional point cloud instance and the benchmark three-dimensional point cloud instance, the embodiment of the present application can obtain the target pose information of the target instance object in the target scene based on the benchmark pose information and relative pose information of the benchmark three-dimensional point cloud instance, and then locate the target instance object in the target scene based on the target pose information.
[0104] In some embodiments, segmenting a target 3D point cloud instance of a target instance object from the 3D point cloud according to a mapping relationship between 2D coordinate points of the 2D image and 3D coordinate points in the 3D point cloud and a target mask image includes:
[0105] Determine a target three-dimensional coordinate point that has a mapping relationship with a two-dimensional coordinate point in a target mask image from the three-dimensional point cloud;
[0106] A point cloud instance composed of each target three-dimensional coordinate point is determined as a target three-dimensional point cloud instance of a target instance object.
[0107] Since there is a mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, for the two-dimensional coordinate points in the target mask image, the target three-dimensional coordinate points that have a mapping relationship with the two-dimensional coordinate points in the target mask image can be searched from the three-dimensional coordinate points of the three-dimensional point cloud. The point cloud instance composed of each target three-dimensional coordinate point is the target three-dimensional point cloud instance of the target instance object, thereby realizing instance segmentation of the three-dimensional point cloud.
[0108] In some embodiments, the 2D instance segmentation model is trained based on:
[0109] Constructing a three-dimensional virtual simulation scene of a target scene; the three-dimensional virtual simulation scene includes a three-dimensional virtual instance object of a target instance object;
[0110] Collecting and rendering a large number of sample images containing three-dimensional virtual instance objects from a three-dimensional virtual simulation scene; the three-dimensional virtual instance objects in the sample images are marked with external frames;
[0111] Generate a sample mask image of the three-dimensional virtual instance object in each sample image;
[0112] The sample images are used as training samples, and the categories of the target instance objects, the bounding boxes in each sample image, and the sample mask images are used as training labels to train the initial two-dimensional instance segmentation model to obtain a trained two-dimensional instance segmentation model.
[0113] The data source of the two-dimensional instance segmentation model in the embodiment of the present application in the model training phase and the data source of the model online application phase are different.
[0114] It is understandable that in deep learning, the quality of the data set directly affects the final implementation effect of the model. At present, most deep learning algorithms are trained with public data sets or manually annotated data sets. There is a certain gap between the former and the actual application scenarios, and the latter requires a lot of manpower and time, both of which have certain shortcomings. Therefore, the virtual simulation method is introduced to model the target scenario of the task and simulate the relevant sample data sets, so that the deep learning model can fully play its role.
[0115] Specifically, the target scene can be modeled and simulated with the help of 3D modeling software according to the actual characteristics and parameters of the target scene (such as scene geometry information, materials, camera parameters, etc.), thereby constructing a 3D virtual simulation scene of the target scene; the 3D virtual simulation scene includes a 3D virtual instance object of the target instance object.
[0116] After constructing the 3D virtual simulation scene, you can use the virtual camera in the software to take pictures to collect and render a large number of sample images containing 3D virtual instance objects from the 3D virtual simulation scene. During the rendering process, you can annotate the 3D virtual instance objects with an external frame to achieve automatic annotation.
[0117] In some embodiments, collecting and rendering a large number of sample images containing three-dimensional virtual instance objects from a three-dimensional virtual simulation scene includes:
[0118] Updating target parameters in the three-dimensional virtual simulation scene; the target parameters include at least one of background, lighting, texture, position and posture of the three-dimensional virtual instance object;
[0119] According to the updated target parameters, a large number of sample images containing three-dimensional virtual instance objects are collected and rendered from the three-dimensional virtual simulation scene.
[0120] The embodiment of the present application updates the target parameters in the three-dimensional virtual simulation scene, so that the training sample data set can be closer to reality and richer, simulate the data changes of the simulation reality, and improve the background complexity.
[0121] In practical applications, similar task scenarios can be simulated in modeling software, and the placement of target objects in actual scenarios can be simulated, and finally virtual simulation data can be obtained by shooting with a virtual camera. With the help of the powerful physics engine inside the software, the physical factors of gravity, collision, and friction in real scenes can be simulated, making the simulated training samples more realistic and minimizing the gap with the real domain. At the same time, with the help of its software API, the number, texture, lighting, background, position, and posture of target instance objects can be quickly randomized, so that the final simulated data can truly randomize the non-fixed features of the target instance objects.
[0122] like Figure 5a and Figure 5b This is a schematic diagram of the scene captured during the disorderly falling process of a three-dimensional virtual instance object in a three-dimensional virtual simulation scene and after it falls to the ground.
[0123] In addition, it is important to use virtual simulation methods to quickly obtain the true value labels of targets in simulation scenarios, greatly saving the time and labor costs of traditional manual labeling.
[0124] like Figure 6 As shown, the present application embodiment provides a second flow chart of a positioning method, comprising the following steps:
[0125] Step 601, constructing a three-dimensional virtual simulation scene of a target scene;
[0126] Step 602, collecting and rendering a large number of sample images containing three-dimensional virtual instance objects from the three-dimensional virtual simulation scene; the three-dimensional virtual instance objects in the sample images are marked with external frames;
[0127] Step 603, generating a sample mask image of the three-dimensional virtual instance object in each sample image;
[0128] Step 604, using the sample images as training samples, and using the category of the target instance object, the bounding box in each sample image, and the sample mask image as training labels, to train the initial two-dimensional instance segmentation model to obtain a trained two-dimensional instance segmentation model;
[0129] Step 605, obtaining a two-dimensional image and a three-dimensional point cloud of a target scene; the target scene includes a target instance object;
[0130] Step 606, calling the trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image, and obtaining an instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes a target mask image of the target instance object;
[0131] Step 607, segmenting the target 3D point cloud instance of the target instance object from the 3D point cloud according to the mapping relationship between the 2D coordinate points of the 2D image and the 3D coordinate points in the 3D point cloud, and the target mask image;
[0132] Step 608, obtaining a first feature descriptor of the reference three-dimensional point cloud;
[0133] Step 609, generating a second feature descriptor of the target three-dimensional point cloud instance;
[0134] Step 610, performing feature matching on the first feature descriptor and the second feature descriptor to obtain a point correspondence relationship between the reference 3D point cloud instance and the target 3D point cloud instance;
[0135] Step 611, obtaining relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance according to the point correspondence relationship;
[0136] Step 612: locate the target instance object in the target scene according to the relative pose information.
[0137] The detailed implementation process of the above steps 601 to 612 is shown in the above embodiments and will not be described again here.
[0138] The positioning method provided in the embodiment of the present application can be executed by a positioning device. In the embodiment of the present application, the positioning method executed by the positioning device is taken as an example to illustrate the positioning device provided in the embodiment of the present application.
[0139] The embodiment of the present application also provides a positioning device.
[0140] like Figure 7 As shown, the positioning device includes: an acquisition module 710 , a first processing module 720 , a second processing module 730 and a third processing module 740 .
[0141] The acquisition module 710 is used to acquire a two-dimensional image and a three-dimensional point cloud of a target scene; the target scene includes a target instance object;
[0142] The first processing module 720 is used to call the trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image to obtain the instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes a target mask image of the target instance object;
[0143] The second processing module 730 is used to segment the target 3D point cloud instance of the target instance object from the 3D point cloud according to the mapping relationship between the 2D coordinate points of the 2D image and the 3D coordinate points in the 3D point cloud and the target mask image;
[0144] The third processing module 740 is used to locate the target instance object in the target scene according to the target three-dimensional point cloud instance.
[0145] According to the positioning device provided in the embodiment of the present application, the instance segmentation processing is performed on the two-dimensional image by calling the trained two-dimensional instance segmentation model to obtain the instance segmentation result output by the two-dimensional instance segmentation model; the instance segmentation result includes the target mask map of the target instance object; according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask map, the target three-dimensional point cloud instance of the target instance object is segmented from the three-dimensional point cloud. Compared with directly locating and segmenting the target three-dimensional point cloud instance of the target instance object in the three-dimensional point cloud, it is more efficient to segment the two-dimensional image first, and the computational complexity is lower. In addition, processing the two-dimensional data and introducing the deep learning instance segmentation method can improve the efficiency of the overall solution and better cope with complex scenes. In addition, the method of combining 2D and 3D is compared with the common method of using RGBD as input to directly estimate the pose. The process does not rely heavily on the intrinsic parameter matrix of the camera, avoiding the need to calibrate the camera when the intrinsic parameter matrix is unknown.
[0146] In some embodiments, the positioning device further includes a training module, the training module being used to: train the two-dimensional instance segmentation model based on the following method:
[0147] Constructing a three-dimensional virtual simulation scene of a target scene; the three-dimensional virtual simulation scene includes a three-dimensional virtual instance object of a target instance object;
[0148] Collecting and rendering a large number of sample images containing three-dimensional virtual instance objects from a three-dimensional virtual simulation scene; the three-dimensional virtual instance objects in the sample images are marked with external frames;
[0149] Generate a sample mask image of the three-dimensional virtual instance object in each sample image;
[0150] The sample images are used as training samples, and the categories of the target instance objects, the bounding boxes in each sample image, and the sample mask images are used as training labels to train the initial two-dimensional instance segmentation model to obtain a trained two-dimensional instance segmentation model.
[0151] In some embodiments, the training module is further used to update target parameters in the three-dimensional virtual simulation scene; the target parameters include at least one of background, lighting, texture, position and posture of the three-dimensional virtual instance object;
[0152] According to the updated target parameters, a large number of sample images containing three-dimensional virtual instance objects are collected and rendered from the three-dimensional virtual simulation scene.
[0153] In some embodiments, the third processing module includes:
[0154] The first processing submodule is used to perform feature matching on the target 3D point cloud instance and the reference 3D point cloud instance of the target instance object to obtain relative position information between the target 3D point cloud instance and the reference 3D point cloud instance;
[0155] The second processing submodule is used to locate the target instance object in the target scene according to the relative posture information.
[0156] In some embodiments, the first processing submodule is used to:
[0157] Obtain a first feature descriptor of a reference three-dimensional point cloud instance;
[0158] Generate a second feature descriptor of the target three-dimensional point cloud instance;
[0159] Performing feature matching on the first feature descriptor and the second feature descriptor to obtain a point correspondence relationship between the reference three-dimensional point cloud instance and the target three-dimensional point cloud instance;
[0160] According to the point correspondence, the relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance is obtained.
[0161] In some embodiments, the second processing submodule is used to:
[0162] Obtaining target pose information of the target instance object in the target scene according to the reference pose information and relative pose information of the reference 3D point cloud instance;
[0163] According to the target pose information, the target instance object is located in the target scene.
[0164] In some embodiments, the first processing module is specifically configured to:
[0165] Determine a target three-dimensional coordinate point that has a mapping relationship with a two-dimensional coordinate point in a target mask image from the three-dimensional point cloud;
[0166] A point cloud instance composed of each target three-dimensional coordinate point is determined as a target three-dimensional point cloud instance of a target instance object.
[0167] The positioning device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0168] The positioning device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0169] The positioning device provided in the embodiment of the present application can achieve Figures 1 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0170] In some embodiments, Figure 8 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, each process of the above-mentioned positioning method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0171] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0172] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned positioning method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0173] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0174] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above positioning method when executed by a processor.
[0175] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0176] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned positioning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0177] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0178] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0180] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
[0181] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0182] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A positioning method, characterized in that: include: Obtain 2D images and 3D point clouds of the target scene; The target scene includes a target instance object; Calling a trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image to obtain an instance segmentation result output by the two-dimensional instance segmentation model; The instance segmentation result includes a target mask image of the target instance object; Segmenting a target three-dimensional point cloud instance of the target instance object from the three-dimensional point cloud according to a mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud, and the target mask image; The target instance object is located in the target scene according to the target three-dimensional point cloud instance.
2. The positioning method according to claim 1, characterized in that: The 2D instance segmentation model is trained based on: Constructing a three-dimensional virtual simulation scene of a target scene; the three-dimensional virtual simulation scene includes a three-dimensional virtual instance object of a target instance object; Collecting and rendering a large number of sample images containing the three-dimensional virtual instance objects from the three-dimensional virtual simulation scene; the three-dimensional virtual instance objects in the sample images are marked with external frames; Generate a sample mask image of the three-dimensional virtual instance object in each sample image; The sample images are used as training samples, and the category of the target instance object, the bounding box in each sample image, and the sample mask image are used as training labels to train the initial two-dimensional instance segmentation model to obtain a trained two-dimensional instance segmentation model.
3. The positioning method according to claim 2, characterized in that: The step of collecting and rendering a large number of sample images containing the three-dimensional virtual instance objects from the three-dimensional virtual simulation scene includes: Updating target parameters in the three-dimensional virtual simulation scene; the target parameters include at least one of background, lighting, texture, position and posture of the three-dimensional virtual instance object; According to the updated target parameters, a large number of sample images containing the three-dimensional virtual instance objects are collected and rendered from the three-dimensional virtual simulation scene.
4. The positioning method according to claim 1, characterized in that: The step of locating the target instance object in the target scene according to the target three-dimensional point cloud instance includes: Performing feature matching on the target 3D point cloud instance and a reference 3D point cloud instance of the target instance object to obtain relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance; The target instance object is located in the target scene according to the relative pose information.
5. The positioning method according to claim 4, characterized in that: The performing feature matching on the target 3D point cloud instance and the reference 3D point cloud instance of the target instance object to obtain relative pose information between the target 3D point cloud instance and the reference 3D point cloud instance includes: Acquire a first feature descriptor of the reference three-dimensional point cloud instance; Generating a second feature descriptor of the target three-dimensional point cloud instance; Performing feature matching on the first feature descriptor and the second feature descriptor to obtain a point correspondence relationship between the reference three-dimensional point cloud instance and the target three-dimensional point cloud instance; According to the point correspondence, relative pose information between the target three-dimensional point cloud instance and the reference three-dimensional point cloud instance is obtained.
6. The positioning method according to claim 4, characterized in that: The locating the target instance object in the target scene according to the relative posture information includes: Obtaining target pose information of the target instance object in the target scene according to the reference pose information of the reference three-dimensional point cloud instance and the relative pose information; The target instance object is located in the target scene according to the target pose information.
7. The positioning method according to any one of claims 1 to 6, characterized in that: The method of segmenting the target 3D point cloud instance of the target instance object from the 3D point cloud according to the mapping relationship between the 2D coordinate points of the 2D image and the 3D coordinate points in the 3D point cloud and the target mask image comprises: Determine, from the three-dimensional point cloud, a target three-dimensional coordinate point having the mapping relationship with the two-dimensional coordinate point in the target mask image; A point cloud instance composed of each target three-dimensional coordinate point is determined as a target three-dimensional point cloud instance of the target instance object.
8. A positioning device, characterized in that: include: An acquisition module, used to acquire a two-dimensional image and a three-dimensional point cloud of a target scene; The target scene includes a target instance object; A first processing module, configured to call a trained two-dimensional instance segmentation model to perform instance segmentation processing on the two-dimensional image, and obtain an instance segmentation result output by the two-dimensional instance segmentation model; The instance segmentation result includes a target mask image of the target instance object; A second processing module is used to segment the target three-dimensional point cloud instance of the target instance object from the three-dimensional point cloud according to the mapping relationship between the two-dimensional coordinate points of the two-dimensional image and the three-dimensional coordinate points in the three-dimensional point cloud and the target mask image; The third processing module is used to locate the target instance object in the target scene according to the target three-dimensional point cloud instance.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the positioning method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the positioning method according to any one of claims 1 to 7 is implemented.