Point cloud vision fusion method and device, electronic equipment and storage medium

Abstract

The invention relates to a point cloud visual fusion method and device, electronic equipment and a storage medium. The method comprises the following steps: constructing a semantic point cloud map, and obtaining a current picture; the current picture corresponds to the current pose, semantic segmentation is carried out on the current picture to obtain the category of pixels in the current picture, likelihood fields and likelihood field gradients of different categories are calculated, a residual block is constructed based on the semantic point cloud map, the likelihood fields and the likelihood field gradients, and the current pose is updated through the residual block. The updated current pose is used for updating the semantic point cloud map. According to the scheme provided by the invention, data fusion of the point cloud and the image can be realized without cross-sensor feature extraction and feature matching, and the accuracy of information in the semantic point cloud map is improved, so that the stability and precision of SLAM are improved.

Description

Technical Field

[0001] The present application relates to the field of navigation technology, and in particular to a point cloud vision fusion method, device, electronic device and storage medium. Background Art

[0002] Currently, robots, autonomous driving, AR (Augmented Reality) glasses and other devices are equipped with multiple sensors including lidar, cameras, etc., and require mapping and positioning (Simultaneous Localization and Mapping, SLAM) during use. In this process, it is usually necessary to consider fusing the data detected by point clouds and visual sensors. In related technologies, the data of these two types of sensors are subjected to feature extraction, feature matching, and tight coupling optimization during data fusion to obtain unified mapping and positioning results. Moreover, since more information is used, the results are relatively accurate.

[0003] However, since the information expressed by the point cloud is spatial position and reflection intensity, and the information expressed by the image is pixel coordinates and color, the two are very different in expression. When fusing point cloud data and images, matching is difficult, accuracy is not easy to guarantee, and errors are easy to occur. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related art, the present application provides a point cloud vision fusion method, device, electronic device and storage medium, which can realize data fusion of point cloud and image without cross-sensor feature extraction and feature matching, thereby improving the stability and accuracy of SLAM.

[0005] The first aspect of the present application provides a point cloud vision fusion method, comprising: Constructing a semantic point cloud map and obtaining a current image; the current image corresponds to the current posture; Performing semantic segmentation on the current image to obtain categories of pixels in the current image; Calculate the likelihood field and likelihood field gradient for different categories; Constructing a residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient; The current pose is updated through the residual block; the updated current pose is used to update the semantic point cloud map.

[0006] Preferably, the calculating of likelihood fields and likelihood field gradients of different categories includes: For each category of the current image, establish a category matrix; Performing Gaussian filtering on the category matrix of each category respectively to obtain a likelihood field of each category; Performing Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category.

[0007] Preferably, the likelihood field gradient includes a first sub-gradient in the x-direction and a second sub-gradient in the y-direction, and performing Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category includes: For the likelihood field of each category, Sobel filtering is performed in the x direction and the y direction respectively to obtain the first sub-gradient and the second sub-gradient.

[0008] Preferably, the constructing a residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient comprises: Get the initial pose corresponding to the initial image; Determine in the semantic point cloud map a set of data points belonging to the pixel category of the current image within a preset range in front of the initial pose; Respectively obtain the likelihood field and likelihood field gradient of each category in the data point set; A residual block is constructed according to the likelihood field and the likelihood field gradient.

[0009] Preferably, the step of respectively obtaining the likelihood field and likelihood field gradient of each category of point cloud in the data point set comprises: For each category of point cloud in the data point set, project the point cloud of each category to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point; According to the projection result, the likelihood field and the likelihood field gradient of each of the effective points are obtained by bilinear interpolation, and the likelihood field is determined as the error corresponding to the effective point.

[0010] Preferably, constructing a residual block according to the likelihood field and the likelihood field gradient comprises: For the likelihood field and likelihood field gradient of each of the valid points, calculate the optimization matrix of each of the valid points relative to the current posture by the chain rule; The residual block is constructed according to the errors of each of the valid points and the optimization matrix.

[0011] Preferably, the step after updating the current pose by the residual block comprises: The step of returning the point cloud for each category in the data point set, projecting the point cloud of each category to the current image through the current posture, and determining the point cloud in the corresponding range of the current image as a valid point, until a preset condition is met.

[0012] A second aspect of the present application provides a point cloud vision fusion device, comprising: A construction module is used to construct a semantic point cloud map and obtain a current image; the current image corresponds to the current posture; A segmentation module, used to perform semantic segmentation on the current image to obtain categories of pixels in the current image; A calculation module, used to calculate likelihood fields and likelihood field gradients of different categories; A residual block module, used for constructing a residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient; An updating module is used to update the current pose through the residual block; the updated current pose is used to update the semantic point cloud map.

[0013] Preferably, the computing module comprises: A category matrix submodule, used to establish a category matrix for each category of the current image; A first filtering submodule, used for performing Gaussian filtering on the category matrix of each category to obtain a likelihood field of each category; The second filtering submodule is used to perform Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category.

[0014] Preferably, the likelihood field gradient includes a first sub-gradient in the x-direction and a second sub-gradient in the y-direction, and the second filtering submodule is further used to perform Sobel filtering in the x-direction and the y-direction for the likelihood field of each category, respectively, to obtain the first sub-gradient and the second sub-gradient.

[0015] Preferably, the residual block module includes: The initial pose submodule is used to obtain the initial pose corresponding to the initial image; A point set submodule, used to determine in the semantic point cloud map a data point set belonging to the pixel category of the current image within a preset range in front of the initial pose; An acquisition submodule, used to respectively acquire the likelihood field and likelihood field gradient of each category in the data point set; A construction submodule is used to construct a residual block according to the likelihood field and the likelihood field gradient.

[0016] Preferably, the calculation submodule includes: A projection unit, configured to project the point cloud of each category in the data point set to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point; An acquisition unit is used to acquire the likelihood field and likelihood field gradient of each of the effective points through bilinear interpolation according to the projection result, and determine the likelihood field as the error corresponding to the effective point.

[0017] Preferably, the residual block module includes: An optimization matrix submodule, for calculating the optimization matrix of each of the effective points relative to the current posture by the chain rule for the likelihood field and likelihood field gradient of each of the effective points; The residual block submodule is used to construct the residual block according to the errors of each of the valid points and the optimization matrix.

[0018] Preferably, the device further comprises: A return module is used to return the point cloud for each category in the data point set, project the point cloud of each category to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point until the preset conditions are met.

[0019] A third aspect of the present application provides an electronic device, including: Processor; and The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0020] A fourth aspect of the present application provides a computer-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0021] The technical solution provided by the present application may include the following beneficial effects: an embodiment of the present application discloses a point cloud vision fusion method, including constructing a semantic point cloud map and obtaining a current picture; the current picture corresponds to the current posture, semantic segmentation is performed on the current picture, the category of pixels in the current picture is obtained, the likelihood field and likelihood field gradient of different categories are calculated, a residual block is constructed based on the semantic point cloud map, the likelihood field and the likelihood field gradient, the current posture is updated through the residual block, and the updated current posture is used to update the semantic point cloud map, which can realize point cloud and image data fusion without cross-sensor feature extraction and feature matching, thereby improving the accuracy of information in the semantic point cloud map, thereby improving the stability and precision of SLAM.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0024] Figure 1 It is a flow chart of the point cloud vision fusion method shown in the embodiment of the present application; Figure 2 is another flow chart of the point cloud vision fusion method shown in the embodiment of the present application; Figure 3 is another flow chart of the point cloud vision fusion method shown in the embodiment of the present application; Figure 4 is a schematic diagram of the structure of point cloud vision fusion shown in an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0026] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0028] Since the information expressed by the point cloud is spatial position and reflection intensity, and the information expressed by the image is pixel coordinates and color, the two are very different in expression. When fusing point cloud data and images, matching is difficult, accuracy is not easy to guarantee, and errors are easy to occur.

[0029] In response to the above problems, an embodiment of the present application provides a point cloud vision fusion method, which can realize data fusion of point cloud and image without cross-sensor feature extraction and feature matching, thereby improving the accuracy of information in the semantic point cloud map, thereby improving the stability and accuracy of SLAM.

[0030] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0031] Figure 1 It is a flow chart of the point cloud vision fusion method shown in the embodiment of the present application.

[0032] See also Figure 1 , the method comprising: Step 101, construct a semantic point cloud map and obtain the current image; the current image corresponds to the current posture.

[0033] Different methods can be used to construct semantic point cloud maps. For example, for point cloud maps, the point cloud in the original point cloud map and the current posture can be used to project the point cloud onto the current image, and then the current image can be semantically segmented, and the corresponding semantic categories of the point cloud in the original point cloud map can be assigned based on the results of the semantic segmentation; 3D (3 Dimensions) semantic segmentation can be performed directly for spatial point clouds (which can be point clouds in point cloud maps or raw LiDAR (Light Laser Detection and Ranging) data), but the drawback of this method is that the current 3D semantic segmentation accuracy is low, especially at the edges of objects, and needs to be used in conjunction with the above-mentioned solution; if an RGBD camera is used to collect data, semantic segmentation can be performed on the collected images, and the 3D point categories can be assigned directly using the image semantic segmentation results.

[0034] In addition, pictures can also be collected through a visual sensor, and each picture corresponds to the posture when the picture is collected. The visual sensor can be a monocular camera or a multi-camera, and this application does not impose any restrictions on this.

[0035] Step 102: perform semantic segmentation on the current image to obtain categories of pixels in the current image.

[0036] Semantic segmentation is an important task in computer vision, which involves assigning each pixel in an image to a specific category, thereby achieving a detailed understanding of the image content. In the SLAM process, the categories of pixels can usually be divided into vehicles, pedestrians, roads, buildings, sky, traffic signs, vegetation, etc. The number of categories can be recorded as N (indexed 0-(N-1)). If the size of the current image is H×W, the size of the image semantic segmentation result Semanti is also H×W.

[0037] Step 103, calculating likelihood fields and likelihood field gradients of different categories.

[0038] For each category, the likelihood field and likelihood field gradient of each category can be calculated. The Likelihood Field Model is a mathematical model for LiDAR, mainly used for robot navigation and obstacle avoidance. Compared with the traditional beam model, the Likelihood Field Model is more flexible and effective in dealing with sensor noise and pose changes. It uses a Gaussian distribution to describe the uncertainty of the sensor and pose, thereby obtaining a Gaussian distribution, and uses a Gaussian kernel to perform Gaussian smoothing on each obstacle. This method does not require the calculation of the expected value because the image has been blurred, avoiding complex raytracing calculations, and directly obtaining the score by looking up the table, which greatly reduces the amount of calculation. It has high computational efficiency and is applicable to various environments.

[0039] Step 104: construct a residual block based on the semantic point cloud map, the likelihood field, and the likelihood field gradient.

[0040] According to the semantic point cloud map, likelihood field and likelihood field gradient, a residual block can be constructed. The optimization of the residual block is usually easily compatible with the original optimization system.

[0041] Step 105, updating the current pose through the residual block; the updated current pose is used to update the semantic point cloud map.

[0042] The current pose can be updated and optimized through the residual block. Further, the semantic point cloud map can be updated and iterated according to the updated current pose, at least including adding and correcting the semantic attributes of the point cloud, correcting the spatial coordinates of the point cloud data, and expanding the point cloud map using visual feature points.

[0043] In related fusion SLAM solutions, the solution provided in the embodiments of the present application can be integrated into most tightly coupled solutions as a module or plug-in to improve the accuracy of the original solution.

[0044] The embodiment of the present application discloses a point cloud vision fusion method, including constructing a semantic point cloud map and obtaining a current image; the current image corresponds to the current posture, semantic segmentation is performed on the current image, the category of pixels in the current image is obtained, the likelihood field and likelihood field gradient of different categories are calculated, a residual block is constructed based on the semantic point cloud map, the likelihood field and the likelihood field gradient, the current posture is updated through the residual block, and the updated current posture is used to update the semantic point cloud map. The data fusion of point cloud and image can be realized without cross-sensor feature extraction and feature matching, thereby improving the accuracy of information in the semantic point cloud map, thereby improving the stability and precision of SLAM.

[0045] Figure 2 It is another flow chart of the point cloud vision fusion method shown in an embodiment of the present application.

[0046] See also Figure 2 , the method comprising: Step 201, construct a semantic point cloud map and obtain the current image; the current image corresponds to the current posture.

[0047] Different methods can be used to construct a semantic point cloud map. In addition, pictures can be collected by a visual sensor, each picture corresponds to the pose when the picture was collected, wherein the visual sensor can be a monocular camera or a multi-camera, and this application does not limit this.

[0048] Step 202: perform semantic segmentation on the current image to obtain categories of pixels in the current image.

[0049] By performing semantic segmentation on the current image, we can get the number of categories N of the pixels in the current image. The category index is 0-(N-1). If the size of the current image is H×W, the size of the image semantic segmentation result Semanti is also H×W. Semanti can be expressed by the following formula (1):

[0050] in( ) represents the pixel, n is the category index, the meaning of this formula is pixel ( ) has a category index of n.

[0051] Step 203: Create a category matrix for each category of the current image.

[0052] For each category of the current image, a three-dimensional category matrix AntiSemantic can be established, and the size of the category matrix is ​​N×H×W.

[0053] Step 204 , Gaussian filtering is performed on the category matrix of each category to obtain the likelihood field of each category.

[0054] The category matrix of each category can be Gaussian filtered or mean filtered to obtain the likelihood field of each category. The likelihood field of each category can be expressed by the following formula (2):

[0055] The likelihood field of all categories can be expressed by the following formula (3):

[0056] Formula (2) indicates that the likelihood field of the kth category is obtained by filtering the kth graph of AntiSemantic. Gaussian filtering and mean filtering can be used here. The result can be understood as follows: the closer to the center of the category, the smaller the likelihood field value, the closer to 0, and the minimum is 0; the farther away from the center of the category, the larger the likelihood field value, the closer to 1, and the maximum is 1.

[0057] The Likelihood size in formula (3) is N×H×W, representing the likelihood field of all categories.

[0058] Step 205: Perform Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category.

[0059] By performing Sobel filtering on the likelihood field of each category, the likelihood field gradient of each pixel can be obtained.

[0060] In an optional embodiment of the present application, the likelihood field gradient includes a first sub-gradient in the x-direction and a second sub-gradient in the y-direction, and step 205 includes: Sub-step 2051, for the likelihood field of each category, Sobel filtering is performed in the x direction and the y direction respectively to obtain a first sub-gradient and a second sub-gradient.

[0061] The likelihood field gradient includes a first sub-gradient in the x direction and the second sub-gradient in the y direction For the likelihood field of the kth category, Sobel filtering is performed in the x and y directions respectively, and we can get and , and It can be expressed by the following formula (4) and formula (5):

[0062]

[0063] Step 206: construct a residual block based on the semantic point cloud map, the likelihood field, and the likelihood field gradient.

[0064] According to the semantic point cloud map, the likelihood field and the likelihood field gradient can construct a residual block, which can be added to the original optimization problem and is usually easily compatible with the original optimization system.

[0065] Step 207: Update the current pose through the residual block.

[0066] The current pose can be updated and optimized through the residual block.

[0067] Step 208: Update the semantic point cloud map according to the updated current pose.

[0068] The point cloud in the iterative semantic point cloud map can be updated according to the updated current posture, including adding and correcting the semantic attributes of the point cloud, correcting the spatial coordinates of the point cloud data, and expanding the point cloud map using visual feature points.

[0069] The embodiment of the present application discloses a point cloud vision fusion method, including constructing a semantic point cloud map and obtaining a current image; the current image corresponds to the current posture, semantically segmenting the current image to obtain the category of the pixels in the current image, establishing a category matrix for each category of the current image, Gaussian filtering the category matrix of each category to obtain the likelihood field of each category, Sobel filtering the likelihood field of each category to obtain the likelihood field gradient of each category, constructing a residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient, updating the current posture through the residual block, and updating the semantic point cloud map according to the updated current posture, which can realize data fusion of point cloud and image without cross-sensor feature extraction and feature matching, thereby improving the accuracy of information in the semantic point cloud map, thereby improving the stability and precision of SLAM.

[0070] Figure 3 It is another flow chart of the point cloud vision fusion method shown in an embodiment of the present application.

[0071] See also Figure 3 , the method comprising: Step 301, construct a semantic point cloud map and obtain the current image; the current image corresponds to the current posture.

[0072] Different methods can be used to construct a semantic point cloud map. In addition, pictures can be collected by a visual sensor, each picture corresponds to the pose when the picture was collected, wherein the visual sensor can be a monocular camera or a multi-camera, and this application does not limit this.

[0073] Step 302: perform semantic segmentation on the current image to obtain categories of pixels in the current image.

[0074] By performing semantic segmentation on the current image, we can get the number of categories N of the pixels in the current image, and the category index is 0-(N-1). If the size of the current image is H×W, the size of the image semantic segmentation result Semianti is also H×W.

[0075] Step 303, calculating likelihood fields and likelihood field gradients of different categories.

[0076] For each category, the likelihood field and likelihood field gradient of each category can be calculated.

[0077] Step 304, obtaining an initial pose corresponding to the initial image.

[0078] You can first get the corresponding initial pose Pose0 based on the initial image.

[0079] Step 305 , determining in the semantic point cloud map a set of data points that belong to the pixel category of the current image within a preset range in front of the initial pose.

[0080] After obtaining the initial pose Pose0, the image features can be used to adopt the inter-frame matching method to determine the rectangular box of the preset range in front of the initial pose Pose0 in the semantic point cloud map according to the initial pose Pose0, and the data point set PointCloud belonging to the category of the current image pixel can be selected in the rectangular box.

[0081] Step 306 , for each category of point cloud in the data point set, project the point cloud of each category to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point.

[0082] Starting from this step, iterate the point clouds of each category k in the data point set PointCloud, that is, k=0, 1, 2, ..., N-1. Select all point clouds with index category k in the data point set PointCloud, use the current pose Pose to project the point cloud with index category k onto the current image, and retain the points in the corresponding range of the current image as valid points for subsequent calculations. This process can be roughly divided into three steps: image dedistortion, coordinate system conversion, and image drawing. The coordinate system conversion includes the conversion between the point cloud coordinate system and the camera coordinate system and the conversion between the camera coordinate system and the pixel coordinate system.

[0083] Step 307, according to the projection result, the likelihood field value and likelihood field gradient of each valid point are obtained by bilinear interpolation, and the likelihood field is determined as the error corresponding to the valid point.

[0084] Bilinear interpolation is often used in image processing. It is an interpolation method that performs linear interpolation in two directions. First, linear interpolation is performed in the x direction to obtain two temporary points, and then linear interpolation is performed in the y direction to finally obtain the value of the target point. The result of this method is independent of the order of interpolation, that is, interpolation in the y direction can be performed first, and then interpolation in the x direction, and the result is the same.

[0085] During the projection process, the pixel points and pixel coordinates corresponding to the effective points can be obtained. According to the pixel coordinates, the likelihood field and likelihood field gradient of each effective point can be obtained by bilinear interpolation, and the likelihood field is determined as the error of the corresponding effective point.

[0086] Step 308, for the likelihood field and likelihood field gradient of each valid point, calculate the optimization matrix of each valid point relative to the current posture by using the chain rule.

[0087] The chain rule is a derivation rule in calculus. It is used to find the derivative of a composite function. It is a common method in the derivation operation of calculus. The derivative of a composite function will be the product of the derivatives of the finite functions that constitute the composite at the corresponding points, just like a chain, one link after another, so it is called the chain rule.

[0088] For the likelihood field and likelihood field gradient of each valid point, the chain rule can be used to calculate the optimization matrix Jacobi matrix required for each valid point relative to the current pose Pose.

[0089] Step 309: construct a residual block according to the error of each valid point and the optimization matrix.

[0090] According to the error and optimization matrix of each effective point, an optimizable residual block can be constructed. Different optimization methods usually have their own construction methods. For example: if the optimization uses the Gauss-Newton method, the Jacobi matrix and error can be used to construct the H matrix and the G matrix, and added to the H matrix and the G matrix of the original system, and the solution can be unified; if the ceres optimization library or the g2o optimization library is used, the Jacobi matrix and error of each effective point can construct a residual block, and all the residual blocks can be added to the original optimization problem to solve it. No matter what method is used, it is usually easy to be compatible with the original optimization system.

[0091] Step 3010, update the current pose through the residual block.

[0092] The current pose Pose can be updated through the optimization results of the residual block.

[0093] In an optional embodiment of the present application, the method further includes: Step S1, return to step 306 until the preset condition is met.

[0094] After updating the current pose Pose, the updated current pose Pose may be used to return to step 306 until a preset condition is met. The preset condition may be that the overall error is small enough or the overall error remains unchanged.

[0095] Step 3011, update the semantic point cloud map according to the updated current pose.

[0096] According to the updated current pose Pose, the point cloud in the iterative semantic point cloud map can be updated, including adding and correcting the semantic attributes of the point cloud, correcting the spatial coordinates of the point cloud data, and expanding the point cloud map using visual feature points.

[0097] The embodiment of the present application discloses a point cloud visual fusion method, including constructing a semantic point cloud map and obtaining a current image; the current image corresponds to the current posture, semantic segmentation is performed on the current image, the category of pixels in the current image is obtained, the likelihood field and likelihood field gradient of different categories are calculated, the initial posture corresponding to the initial image is obtained, and the data point set belonging to the current image category within a preset range in front of the initial posture is determined in the semantic point cloud map, for each category of point cloud in the data point set, the point cloud of each category is projected to the current image through the current posture, and the point cloud in the corresponding range of the current image is determined as a valid point, and according to the projection result, the point cloud is determined to be a valid point. The likelihood field value and likelihood field gradient of each valid point are obtained through bilinear interpolation, and the likelihood field is determined as the error of the corresponding valid point. For the likelihood field and likelihood field gradient of each valid point, the optimization matrix of each valid point relative to the current pose is calculated by the chain rule. A residual block is constructed according to the error and optimization matrix of each valid point. The current pose is updated through the residual block. The semantic point cloud map is updated according to the updated current pose. This can realize data fusion of point cloud and image without cross-sensor feature extraction and feature matching, thereby improving the accuracy of information in the semantic point cloud map, thereby improving the stability and precision of SLAM.

[0098] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a point cloud vision fusion device, an electronic device and corresponding embodiments.

[0099] Figure 4 It is a structural schematic diagram of the point cloud vision fusion device shown in an embodiment of the present application.

[0100] See also Figure 4 , the point cloud vision fusion device 400 includes: Construction module 401, used to construct a semantic point cloud map and obtain a current image; the current image corresponds to the current posture; The segmentation module 402 is used to perform semantic segmentation on the current image to obtain the categories of pixels in the current image; A calculation module 403 is used to calculate likelihood fields and likelihood field gradients of different categories; A residual block module 404, configured to construct a residual block based on the semantic point cloud map, the likelihood field, and the likelihood field gradient; The updating module 405 is used to update the current pose through the residual block; the updated current pose is used to update the semantic point cloud map.

[0101] In an optional embodiment of the present application, the calculation module 403 includes: The category matrix submodule is used to establish a category matrix for each category of the current image; The first filtering submodule is used to perform Gaussian filtering on the category matrix of each category to obtain the likelihood field of each category; The second filtering submodule is used to perform Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category.

[0102] In an optional embodiment of the present application, the likelihood field gradient includes a first sub-gradient in the x-direction and a second sub-gradient in the y-direction, and the second filtering submodule is further used to perform Sobel filtering in the x-direction and the y-direction for the likelihood field of each category, respectively, to obtain the first sub-gradient and the second sub-gradient.

[0103] In an optional embodiment of the present application, the residual block module 404 includes: The initial pose submodule is used to obtain the initial pose corresponding to the initial image; The point set submodule is used to determine the data point set belonging to the current image pixel category within a preset range in front of the initial pose in the semantic point cloud map; The acquisition submodule is used to obtain the likelihood field and likelihood field gradient of each category in the data point set respectively; A submodule is constructed to construct a residual block based on the likelihood field and the likelihood field gradient.

[0104] In an optional embodiment of the present application, the acquisition submodule includes: A projection unit is used to project the point cloud of each category in the data point set to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point; The acquisition unit is used to acquire the likelihood field and likelihood field gradient of each valid point through bilinear interpolation according to the projection result, and determine the likelihood field as the error corresponding to the valid point.

[0105] In an optional embodiment of the present application, the residual block module 404 includes: The optimization matrix submodule is used to calculate the optimization matrix of each valid point relative to the current pose by the chain rule for the likelihood field and likelihood field gradient of each valid point; The residual block submodule is used to construct a residual block according to the error of each valid point and the optimization matrix.

[0106] In an optional embodiment of the present application, the device further includes: The return module is used to return the point cloud for each category in the data point set, project the point cloud of each category to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point until the preset conditions are met.

[0107] The embodiment of the present application discloses a point cloud vision fusion device, including constructing a semantic point cloud map and obtaining a current image; the current image corresponds to the current posture, semantic segmentation is performed on the current image, the category of pixels in the current image is obtained, the likelihood field and likelihood field gradient of different categories are calculated, a residual block is constructed based on the semantic point cloud map, the likelihood field and the likelihood field gradient, the current posture is updated through the residual block, and the updated current posture is used to update the semantic point cloud map. The data fusion of point cloud and image can be realized without cross-sensor feature extraction and feature matching, thereby improving the accuracy of information in the semantic point cloud map, thereby improving the stability and precision of SLAM.

[0108] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0109] Figure 5 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0110] See also Figure 5 , the electronic device 500 includes a memory 510 and a processor 520 .

[0111] The processor 520 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0112] The memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 520 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at runtime. In addition, the memory 510 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 510 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0113] The memory 510 stores executable codes, and when the executable codes are processed by the processor 520 , the processor 520 can execute part or all of the methods described above.

[0114] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0115] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0116] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A point cloud vision fusion method, characterized in that: The method comprises: Constructing a semantic point cloud map and obtaining a current image; the current image corresponds to the current posture; Performing semantic segmentation on the current image to obtain categories of pixels in the current image; Calculate the likelihood field and likelihood field gradient for different categories; Constructing a residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient; The current pose is updated by the residual block; the updated current pose is used to update the semantic point cloud map.

2. The method according to claim 1, characterized in that: The calculating of likelihood fields and likelihood field gradients of different categories includes: For each category of the current image, establish a category matrix; Performing Gaussian filtering on the category matrix of each category respectively to obtain a likelihood field of each category; Performing Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category.

3. The method according to claim 2, characterized in that The likelihood field gradient includes a first sub-gradient in the x-direction and a second sub-gradient in the y-direction, and performing Sobel filtering on the likelihood field of each category to obtain the likelihood field gradient of each category includes: For the likelihood field of each category, Sobel filtering is performed in the x direction and the y direction respectively to obtain the first sub-gradient and the second sub-gradient.

4. The method according to claim 1, characterized in that: The constructing of the residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient comprises: Get the initial pose corresponding to the initial image; Determine in the semantic point cloud map a set of data points belonging to the pixel category of the current image within a preset range in front of the initial pose; Respectively obtain the likelihood field and likelihood field gradient of each category in the data point set; A residual block is constructed according to the likelihood field and the likelihood field gradient.

5. The method according to claim 4, characterized in that The step of respectively obtaining the likelihood field and likelihood field gradient of each category of point cloud in the data point set comprises: For each category of point cloud in the data point set, project the point cloud of each category to the current image through the current posture, and determine the point cloud in the corresponding range of the current image as a valid point; According to the projection result, the likelihood field and the likelihood field gradient of each of the effective points are obtained by bilinear interpolation, and the likelihood field is determined as the error corresponding to the effective point.

6. The method according to claim 5, characterized in that The constructing a residual block according to the likelihood field and the likelihood field gradient comprises: For the likelihood field and likelihood field gradient of each of the valid points, calculate the optimization matrix of each of the valid points relative to the current posture by the chain rule; The residual block is constructed according to the errors of each of the valid points and the optimization matrix.

7. The method according to claim 5, characterized in that The steps after updating the current pose by the residual block include: The step of returning the point cloud for each category in the data point set, projecting the point cloud of each category to the current image through the current posture, and determining the point cloud in the corresponding range of the current image as a valid point, until a preset condition is met.

8. A point cloud vision fusion device, characterized in that: The device comprises: A construction module is used to construct a semantic point cloud map and obtain a current image; the current image corresponds to the current posture; A segmentation module, used to perform semantic segmentation on the current image to obtain categories of pixels in the current image; A calculation module, used to calculate likelihood fields and likelihood field gradients of different categories; A residual block module, used for constructing a residual block based on the semantic point cloud map, the likelihood field and the likelihood field gradient; An updating module is used to update the current pose through the residual block; the updated current pose is used to update the semantic point cloud map.

9. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 7.

Images