Three-dimensional semantic point cloud determination method and device and mobile platform

Abstract

The invention is suitable for the technical field of software, and provides a three-dimensional semantic point cloud determination method and device and a mobile platform, and the method comprises the steps: obtaining a two-dimensional semantic image; obtaining point cloud data, the point cloud data being obtained by scanning of a low-precision or low-harness radar, and the point cloud data comprising point cloud data of the ground; according to the point cloud data and external parameters from a radar coordinate system to a photographing equipment coordinate system, determining a ground plane equation under the photographing equipment coordinate system; determining depth information of ground plane points under the photographing equipment coordinate system according to the ground plane equation under the photographing equipment coordinate system and a preset photographing equipment model; and determining a three-dimensional semantic point cloud corresponding to the two-dimensional semantic image according to the depth information of the ground plane point under the photographing equipment coordinate system and the photographing equipment model. By means of the method, the three-dimensional semantic point cloud with high precision can be generated with low cost.

Description

Technical Field

[0001] This application belongs to the field of software technology, and in particular relates to methods, apparatus, mobile platforms, computer-readable storage media and computer program products for determining three-dimensional semantic point clouds. Background Technology

[0002] As mobile platforms (such as vehicles and robots) become increasingly intelligent, their integrated functions are also becoming richer. Taking vehicles as an example, they can recognize lane lines and provide real-time lane information, as well as detect vehicles ahead and issue obstacle avoidance warnings. However, before realizing these functions, it is necessary to first construct a corresponding "3D semantic point cloud." A 3D semantic point cloud is based on the geometric coordinates of a traditional 3D point cloud, assigning a semantic label to each point indicating "what it is," enabling the data to simultaneously carry spatial location and object category information, thereby providing a reliable basis for subsequent tasks (such as lane recognition and vehicle detection).

[0003] Existing methods can generate 3D semantic point clouds from radar data acquired by high-beam and high-precision radar. However, due to the high cost of high-beam and high-precision radar, generating 3D semantic point clouds using the above method will result in high costs. Summary of the Invention

[0004] This application provides a method, apparatus, and mobile platform for determining three-dimensional semantic point clouds, which can solve the problem of high cost in generating three-dimensional semantic point clouds using existing methods.

[0005] In a first aspect, embodiments of this application provide a method for determining three-dimensional semantic point clouds, including:

[0006] A two-dimensional semantic image is acquired, which includes a semantic image of the ground. The two-dimensional semantic image is obtained by processing a two-dimensional image captured by a camera.

[0007] Acquire point cloud data, which is obtained by low-precision or low-beam radar scanning, and includes point cloud data of the ground.

[0008] Based on the point cloud data and the external parameters from the radar coordinate system to the imaging device coordinate system, determine the ground plane equation in the imaging device coordinate system;

[0009] Based on the ground plane equation in the shooting device coordinate system and the preset shooting device model, determine the depth information of the ground plane points in the shooting device coordinate system;

[0010] Based on the depth information of ground plane points in the coordinate system of the shooting device and the model of the shooting device, the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image is determined.

[0011] The beneficial effects of the embodiments in this application compared with the prior art are:

[0012] In this embodiment, ground point cloud data is obtained from low-precision or low-beam radar scanning. Based on this point cloud data, the ground plane equation in the coordinate system of the capturing device can be determined. Then, based on this ground plane equation and a preset capturing device model, the depth information of the ground plane points in the capturing device coordinate system is determined. Finally, by combining this depth information and the capturing device model, the 3D semantic point cloud corresponding to the acquired 2D semantic image can be calculated. Since the point cloud data is obtained through low-precision or low-beam radar scanning, and the cost of low-precision or low-beam radar is lower than that of high-precision and high-beam radar, the 3D semantic point cloud determined in the above manner can effectively reduce the cost of the mobile platform. Furthermore, since low-precision or low-beam radar can operate in all weather conditions and has a long range and high accuracy (even low-precision radar can achieve an accuracy of 0.3 meters to 1 meter), the combination of the capturing device and low-precision or low-beam radar facilitates the generation of high-accuracy 3D semantic point clouds at a lower cost.

[0013] Secondly, embodiments of this application provide a three-dimensional semantic point cloud determination device, comprising:

[0014] A two-dimensional semantic image acquisition module is used to acquire two-dimensional semantic images, including semantic images of the ground, which are obtained by processing two-dimensional images captured by the imaging device.

[0015] A point cloud data acquisition module is used to acquire point cloud data, which is obtained by low-precision or low-beam radar scanning, and includes point cloud data of the ground.

[0016] The ground plane equation determination module is used to determine the ground plane equation in the camera coordinate system based on the point cloud data and the external parameters from the radar coordinate system to the camera coordinate system.

[0017] The depth information determination module is used to determine the depth information of ground plane points in the shooting device coordinate system based on the ground plane equation in the shooting device coordinate system and the preset shooting device model.

[0018] The three-dimensional semantic point cloud determination module is used to determine the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image based on the depth information of ground plane points in the coordinate system of the shooting device and the model of the shooting device.

[0019] Thirdly, embodiments of this application provide a mobile platform, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0021] Fifthly, embodiments of this application provide a computer program product that, when run on a mobile platform, causes the mobile platform to execute the method described in the first aspect.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 This is a flowchart illustrating a method for determining three-dimensional semantic point clouds provided by existing technology;

[0025] Figure 2 This is a schematic diagram of a two-dimensional semantic image of the ground provided in an embodiment of this application;

[0026] Figure 3 This is one embodiment provided in this application. Figure 2 A schematic diagram of the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image of the ground;

[0027] Figure 4 This is a schematic diagram of the structure of a three-dimensional semantic point cloud determination device provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a mobile platform provided in one embodiment of this application. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0030] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0031] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0032] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0034] A mobile platform refers to a self-powered transportation or operational base that can actively change its position in the environment. Vehicles, robotic vacuum cleaners, unmanned vehicles, and quadruped robots are all examples of mobile platforms.

[0035] For mobile platforms with ground information recognition capabilities, 3D semantic point clouds are typically generated from radar data acquired by high-beam and high-precision radar. However, due to the high cost of high-beam and high-precision radar, generating 3D semantic point clouds in this way will result in a high cost for the mobile platform. If a lower-cost, low-beam or low-precision radar is used instead, the lower resolution of the radar will result in a sparse, striped point cloud, making it difficult to capture the subtle geometric features of the object being measured, thus leading to lower accuracy of the generated 3D semantic point cloud.

[0036] To generate accurate 3D semantic point clouds of the ground at a lower cost, embodiments of this application provide a method for determining 3D semantic point clouds. In this method, a 2D semantic image of the ground is acquired using an imaging device, and then depth information of the 2D semantic image is acquired using a low-precision or low-beam radar. Finally, a 3D semantic point cloud is generated based on the 2D semantic image and the corresponding depth information.

[0037] The method for determining three-dimensional semantic point clouds provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0038] Figure 1 The diagram illustrates a flowchart of a three-dimensional semantic point cloud determination method provided in an embodiment of this application. This method can be applied to a mobile platform equipped with a camera and a low-precision radar or a low-beam radar, as detailed below:

[0039] S11, acquire a two-dimensional semantic image, which includes a semantic image of the ground, and is obtained by processing a two-dimensional image captured by the shooting device.

[0040] In this embodiment of the application, a shooting device is installed on a mobile platform. The shooting device can be a monocular camera or a depth camera.

[0041] The lens of the aforementioned imaging device is pointed in the direction the mobile platform is moving, and the installation angle of the imaging device (i.e., the angle between the lens of the imaging device and the horizontal plane) is slightly inclined towards the ground to capture a two-dimensional image of the ground that the mobile platform will pass over. After the imaging device captures the two-dimensional image of the ground, a semantic category label is assigned to each pixel of the two-dimensional image to obtain a two-dimensional semantic image with the same size as the original two-dimensional image. Optionally, in order to obtain ground information that is far away in the direction the mobile platform is moving, the installation angle of the aforementioned imaging device can be set to be greater than 0° and less than 45°.

[0042] In this embodiment of the application, noise in the two-dimensional image can be filtered out before processing the image. The noise is determined according to the specific task. For example, if the specific task is to detect lane lines and guide arrows, and the acquired two-dimensional image includes vehicles in addition to lane lines and guide arrows, then the vehicles are considered noise in the two-dimensional image, and the relevant information of the vehicles needs to be filtered out. After that, a semantic category label is assigned to each pixel of the two-dimensional image after the noise is filtered out.

[0043] Optionally, the two-dimensional image can be labeled pixel by pixel using a pre-trained semantic segmentation model (i.e., semantic annotation) to obtain the aforementioned two-dimensional semantic image. Of course, the two-dimensional semantic image can also be obtained by manually assigning labels to the two-dimensional image; this is not a limitation here.

[0044] In this embodiment of the application, the two-dimensional semantic image of the ground can be as follows: Figure 2 As shown. In Figure 2In this example, different colors are used to mark lane lines and guide arrows: yellow for lane lines and green for guide arrows. Since lane lines and guide arrows belong to different categories of information, using different colors to mark them helps to quickly determine the category of pixels in the 2D image based on color. Of course, in practice, other information can also be used to mark pixels of different categories, which will not be elaborated upon here.

[0045] S12, acquire point cloud data, which is obtained by low-precision or low-beam radar scanning, including the point cloud data of the ground.

[0046] In this embodiment of the application, low-precision or low-beam radar refers to low-precision radar or low-beam radar.

[0047] Low-precision radar refers to a type of radar that is significantly inferior to high-end radar systems in terms of spatial resolution (angle or distance), velocity accuracy, target discrimination capability, or anti-jamming capability. When low-precision radar scans the ground, it cannot extract complete geometric road features due to the low intensity of ground features, but it can obtain relatively accurate distance information between the radar and the ground.

[0048] The main characteristics of low-beam radar are fewer vertical lines and lower cost. The direct consequence of fewer vertical lines is sparse point clouds and low angular resolution. Due to the low vertical angular resolution of low-beam radar, when scanning the ground, the resulting point cloud data will present as sparse linear distributions rather than dense, continuous surfaces. This sparse point cloud data cannot fully capture small, detailed features on the ground, such as stop lines or speed bumps, because these features may only appear in the gaps between two scan lines and are not scanned at all. Therefore, it is difficult to obtain accurate three-dimensional semantic information of the ground using low-beam radar, but it can obtain relatively accurate distance information between the radar and the ground.

[0049] In this embodiment, a low-precision or low-beam radar is installed on a mobile platform. The scanning plane of the low-precision or low-beam radar is slightly tilted downwards (e.g., the main beam axis is tilted downwards by about 5–10° relative to the horizontal plane) towards the direction of platform movement, for acquiring point cloud data of the ground ahead. This point cloud data includes distance information between the radar and the ground.

[0050] Considering that the acquired point cloud data may be distorted, distortion correction processing is performed on the acquired point cloud data to improve the accuracy of the subsequent ground plane equation obtained from the point cloud data. That is, in some embodiments, after acquiring the point cloud data in S12, the process further includes:

[0051] The obtained point cloud data is then subjected to distortion correction processing.

[0052] In this embodiment of the application, distortion correction refers to correcting the points in the point cloud data that are 'distorted' due to radar motion or their own errors back to their proper positions.

[0053] Optionally, considering that the point cloud data acquired by the radar is usually distorted when it is in motion, but not when it is stationary, the distortion correction processing of the point cloud data can be performed after determining that the radar is in motion, in order to save system resources. That is, when the three-dimensional semantic point cloud determination method provided in this application embodiment is applied to a mobile platform, the above-mentioned distortion correction processing of the acquired point cloud data includes:

[0054] After determining that the mobile platform is in motion, the acquired point cloud data is subjected to distortion correction processing.

[0055] Since the radar is mounted on the mobile platform, determining that the platform is moving is equivalent to determining that the radar is moving as well. At this point, distortion correction processing is then performed on the acquired point cloud data. Specifically, sensors capable of outputting signals such as speed, displacement, or wheel speed can be installed on the mobile platform. The signals output by these sensors are then compared with a preset movement threshold to determine whether the platform is moving. Alternatively, if the aforementioned sensors are not installed on the mobile platform, the movement of the platform can be determined by observing changes in the position of its external environment.

[0056] In this embodiment of the application, when performing distortion correction processing on point cloud data, it can be combined with data from the inertial measurement unit (IMU) of the mobile platform. In this case, the aforementioned distortion correction processing of the acquired point cloud data includes:

[0057] Obtain the IMU data from the aforementioned mobile platform, and perform distortion correction processing on the obtained point cloud data based on the IMU data.

[0058] In this embodiment, the timestamp of the IMU data is aligned with the timestamp of the point cloud data to be distorted. The IMU data is integrated to obtain the pose increment of the moving platform relative to a unified reference time at each point cloud point. The pose increment is converted into a pose increment in the radar coordinate system through the extrinsic parameters of the IMU and the radar. The coordinate transformation of each point cloud point is performed using this pose increment to unify all point cloud points to the same reference time, thereby completing the distortion removal process of the point cloud data to be distorted in this frame, and obtaining the distorted point cloud data.

[0059] Of course, in addition to distortion correction of point cloud data based on IMU data, other methods can also be used in practice. For example, the pose increment of the mobile platform can be calculated using wheel speed and steering angle, which will not be elaborated here.

[0060] S13. Based on the above point cloud data and the external parameters from the radar coordinate system to the shooting device coordinate system, determine the ground plane equation in the above shooting device coordinate system.

[0061] In this embodiment, the radar coordinate system refers to the coordinate system constructed using radar, and the imaging device coordinate system refers to the coordinate system constructed using the imaging device. The extrinsic parameter from the radar coordinate system to the imaging device coordinate system is a rigid mapping relationship between the radar coordinate system and the imaging device coordinate system, which is used to transform any point in the radar coordinate system to the imaging device coordinate system.

[0062] In this embodiment, point cloud data that may represent the ground can be filtered from the point cloud data. The point cloud data that may represent the ground is then fitted to obtain a ground plane equation. Based on this ground plane equation, a normal vector perpendicular to the ground plane and the radar's height above the ground plane are obtained. A data point in a radar coordinate system is determined on the ground plane equation based on the radar's height above the ground plane. This data point and normal vector in the radar coordinate system are then used as extrinsic parameters to connect the radar coordinate system to the imaging device coordinate system to determine the data point and normal vector in the imaging device coordinate system. Based on this data point and normal vector in the imaging device coordinate system, the ground plane equation in that imaging device coordinate system is determined.

[0063] In the above method, the normal vector in the radar coordinate system is calculated first, and then the normal vector is transformed to the imaging device coordinate system. In practice, the point cloud data can also be transformed from the radar coordinate system to the imaging device coordinate system first, and then the corresponding normal vector and ground plane equation are calculated. This will not be elaborated here.

[0064] Optionally, when the point cloud data is point cloud data after distortion correction, then in step S13 above, based on the point cloud data and the extrinsic parameters from the radar coordinate system to the shooting device coordinate system, the ground plane equation in the shooting device coordinate system is determined, including:

[0065] Based on the distortion-corrected point cloud data and the external parameters from the radar coordinate system to the imaging device coordinate system, the ground plane equation in the imaging device coordinate system is determined.

[0066] Since the distortion-corrected point cloud data can more accurately reflect the actual situation, the determined ground plane equation is more accurate when the distortion-corrected point cloud data is used to determine the ground plane equation in the coordinate system of the shooting device.

[0067] S14. Based on the ground plane equation in the above-mentioned shooting device coordinate system and the preset shooting device model, determine the depth information of the ground plane points in the above-mentioned shooting device coordinate system.

[0068] The aforementioned imaging equipment model is used to project three-dimensional world points onto a two-dimensional image plane.

[0069] In this embodiment of the application, since the shooting device model contains both three-dimensional depth information and two-dimensional image information, after calculating the ground plane equation (equivalent to two-dimensional image information) in the shooting device coordinate system, the correspondence between the two-dimensional image information and the depth information can be calculated based on the ground plane equation and the preset shooting device model.

[0070] S15. Based on the depth information of the ground plane points in the above-mentioned shooting device coordinate system and the above-mentioned shooting device model, determine the three-dimensional semantic point cloud corresponding to the above-mentioned two-dimensional semantic image.

[0071] In this embodiment of the application, the depth information corresponding to each pixel in the two-dimensional semantic image can be calculated based on the correspondence between two-dimensional image information and depth information, thereby obtaining the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image. For example... Figure 3 As shown, the Figure 3 It shows Figure 2 A schematic diagram of the corresponding 3D semantic point cloud.

[0072] In this embodiment, ground point cloud data is obtained from low-precision or low-beam radar scanning. Based on this point cloud data, the ground plane equation in the coordinate system of the capturing device can be determined. Then, based on this ground plane equation and a preset capturing device model, the depth information of the ground plane points in the capturing device coordinate system is determined. Finally, by combining this depth information and the capturing device model, the 3D semantic point cloud corresponding to the acquired 2D semantic image can be calculated. Since the point cloud data is obtained through low-precision or low-beam radar scanning, and the cost of low-precision or low-beam radar is lower than that of high-precision and high-beam radar, the 3D semantic point cloud determined in the above manner can effectively reduce the cost of the mobile platform. Furthermore, since low-precision or low-beam radar can operate in all weather conditions and has a long range and high accuracy (even low-precision radar can achieve an accuracy of 0.3 meters to 1 meter), the combination of the capturing device and low-precision or low-beam radar facilitates the generation of high-accuracy 3D semantic point clouds at a lower cost.

[0073] In some embodiments, when the normal vector and height of the ground plane in the radar coordinate system are first calculated based on point cloud data, and then the equation of the ground plane in the imaging device coordinate system is determined based on the normal vector and height of the ground plane in the radar coordinate system, step S13 above, determining the equation of the ground plane in the imaging device coordinate system based on the point cloud data and the external parameters from the radar coordinate system to the imaging device coordinate system, includes:

[0074] A1. Based on the point cloud data above, determine the first normal vector perpendicular to the ground plane and the height of the radar reaching the ground plane.

[0075] A2. Select a data point from the above point cloud data, and determine the first ground point of the data point on the above ground plane based on the height of the radar reaching the above ground plane.

[0076] A3. Based on the extrinsic parameters from the radar coordinate system to the imaging device coordinate system, the first normal vector and the first ground point are transformed from the radar coordinate system to the imaging device coordinate system to obtain the second normal vector and the second ground point.

[0077] A4. Determine the equation of the ground plane in the coordinate system of the above-mentioned shooting equipment based on the above-mentioned second normal vector and the above-mentioned second ground point.

[0078] Specifically, point cloud data that is likely to represent the ground is filtered out from the point cloud data. Then, the filtered point cloud data is fitted using a Random Sample Consensus (RANSAC) algorithm or a region growing algorithm to obtain the ground plane equation. Based on this ground plane equation, the normal vector perpendicular to the ground plane (i.e., the first normal vector) and the radar's altitude above the ground plane are obtained. Assuming the ground plane equation is ax + by + cz + h = 0, normalizing the coefficients (a, b, c) of this equation yields the unit normal vector perpendicular to the ground plane (i.e., the first normal vector) n. l And, the height h of the radar above the ground plane is obtained.

[0079] After obtaining the radar's height h above the ground plane, select a data point from the point cloud data, such as the origin of the radar coordinate system (0,0,0). T Set the radar origin point (0,0,0) to... T After translating a distance h along the first normal vector towards the ground plane, a ground point (i.e., the first ground point) in the radar coordinate system can be obtained: p l =hn l .

[0080] Based on the external parameters from the radar coordinate system to the imaging equipment coordinate system The first normal vector n can be lTransform to the coordinate system of the shooting device to obtain the second normal vector n c : And, the first ground point p can be... l Transform to the coordinate system of the shooting device to obtain the second ground point p. c :

[0081] Based on the second normal vector and the second ground point, the equation of the ground plane in the coordinate system of the shooting device is obtained as follows:

[0082]

[0083] make Based on the above ground plane equation, we can obtain The shooting equipment model p = zK -1 (u,v,1) T Substituting into the equation for the ground plane, we get:

[0084]

[0085] Where z represents the depth information of the ground point in the coordinate system of the shooting device, K represents the intrinsic parameter of the shooting device, and (u,v) represents the position coordinates of the image point.

[0086] After obtaining the depth information z of the ground points, we substitute it into the model of the imaging device to solve for the three-dimensional coordinates corresponding to the semantic pixel information in the two-dimensional semantic image, thus obtaining the three-dimensional semantic point cloud:

[0087]

[0088] In this embodiment of the application, when the shooting device is a depth camera, since the depth camera can output depth information and two-dimensional images, the depth information output by the depth camera or the depth information determined by the point cloud data obtained by low-beam or low-precision radar can be selected according to the actual situation, and then the three-dimensional semantic point cloud of the two-dimensional semantic image corresponding to the two-dimensional image can be determined according to the selected depth information.

[0089] In some embodiments, considering that in short-range scenarios, the ranging accuracy of depth cameras is generally better than that of low-beam or low-precision radars, while in longer-range scenarios, the ranging accuracy of low-beam or low-precision radars is better than that of depth cameras, therefore, when the mobile platform is moving at a high speed, it is possible to select to measure targets at longer distances. In this case, the depth information determined by the low-beam or low-precision radar is selected. That is, after obtaining the two-dimensional semantic image in step S11 above, the method further includes:

[0090] The speed of the aforementioned mobile platforms.

[0091] Correspondingly, step S12 above, which involves acquiring point cloud data, includes:

[0092] The point cloud data is acquired when the speed of the mobile platform exceeds a preset speed threshold.

[0093] In this embodiment, when the mobile platform is equipped with a Global Positioning System (GPS), its speed can be determined based on the GPS. When the mobile platform determines its speed through an Electronic Stability Program (ESP), its speed can be obtained via the CAN bus. Of course, other methods can also be used to obtain the mobile platform's speed, which will not be elaborated here.

[0094] In this embodiment of the application, after the speed of the mobile platform is obtained, the speed of the mobile platform is compared with a preset speed threshold. If it is greater than the preset speed threshold, point cloud data is obtained to perform the step of determining the ground plane equation in the shooting device coordinate system based on the point cloud data, the external parameters from the radar coordinate system to the shooting device coordinate system, etc.

[0095] Optionally, if the shooting device is a depth camera, which is used to output the aforementioned two-dimensional image and the depth information corresponding to the aforementioned two-dimensional image, then after obtaining the speed of the aforementioned mobile platform in step S11, the method further includes:

[0096] If the speed of the mobile platform is less than a preset threshold, the depth information corresponding to the two-dimensional semantic image is obtained, and the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image is determined based on the two-dimensional semantic image and the depth information.

[0097] In this embodiment of the application, since both the depth information and the two-dimensional image are acquired by a depth camera, and the two-dimensional semantic image is obtained after processing the two-dimensional image, the depth information corresponds to the two-dimensional semantic image. Therefore, the position coordinates of the two-dimensional semantic image and the corresponding depth information can be substituted into the shooting device model corresponding to the depth camera to determine the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0099] Corresponding to the three-dimensional semantic point cloud determination method described in the above embodiments, Figure 4 The diagram shows a structural block diagram of a three-dimensional semantic point cloud determination device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0100] Reference Figure 4 The three-dimensional semantic point cloud determination device 4 is applied to a mobile platform and includes: a two-dimensional semantic image acquisition module 41, a point cloud data acquisition module 42, a ground plane equation determination module 43, a depth information determination module 44, and a three-dimensional semantic point cloud determination module 45. Wherein:

[0101] The two-dimensional semantic image acquisition module 41 is used to acquire two-dimensional semantic images, including semantic images of the ground, which are obtained by processing two-dimensional images captured by the shooting device.

[0102] The point cloud data acquisition module 42 is used to acquire point cloud data, which is obtained by low-precision or low-beam radar scanning and includes the point cloud data of the ground.

[0103] The ground plane equation determination module 43 is used to determine the ground plane equation in the above-mentioned shooting device coordinate system based on the above-mentioned point cloud data and the external parameters from the radar coordinate system to the shooting device coordinate system.

[0104] The depth information determination module 44 is used to determine the depth information of the ground plane points in the above-mentioned shooting device coordinate system based on the ground plane equation in the above-mentioned shooting device coordinate system and the preset shooting device model.

[0105] The 3D semantic point cloud determination module 45 is used to determine the 3D semantic point cloud corresponding to the 2D semantic image based on the depth information of the ground plane points in the coordinate system of the shooting device and the shooting device model.

[0106] In this embodiment, ground point cloud data is obtained from low-precision or low-beam radar scanning. Based on this point cloud data, the ground plane equation in the coordinate system of the capturing device can be determined. Then, based on this ground plane equation and a preset capturing device model, the depth information of the ground plane points in the capturing device coordinate system is determined. Finally, by combining this depth information and the capturing device model, the 3D semantic point cloud corresponding to the acquired 2D semantic image can be calculated. Since the point cloud data is obtained through low-precision or low-beam radar scanning, and the cost of low-precision or low-beam radar is lower than that of high-precision and high-beam radar, the 3D semantic point cloud determined in the above manner can effectively reduce the cost of the mobile platform. Furthermore, since low-precision or low-beam radar can operate in all weather conditions and has a long range and high accuracy (even low-precision radar can achieve an accuracy of 0.3 meters to 1 meter), the combination of the capturing device and low-precision or low-beam radar facilitates the generation of high-accuracy 3D semantic point clouds at a lower cost.

[0107] Optionally, the above-mentioned ground plane equation determination module 43 includes:

[0108] The first normal vector determination unit is used to determine, based on the point cloud data, a first normal vector perpendicular to the ground plane and the height of the radar reaching the ground plane.

[0109] The first ground point determination module is used to select a data point in the point cloud data and determine the first ground point of the data point on the ground plane based on the height of the radar reaching the ground plane.

[0110] The second normal vector determination unit is used to transform the first normal vector and the first ground point from the radar coordinate system to the imaging device coordinate system based on the extrinsic parameters from the radar coordinate system to the imaging device coordinate system, so as to obtain the second normal vector and the second ground point.

[0111] The ground plane equation determination unit is used to determine the ground plane equation in the coordinate system of the shooting device based on the second normal vector and the second ground point.

[0112] Optionally, the three-dimensional semantic point cloud determination device 4 provided in this application embodiment further includes:

[0113] The distortion correction module is used to perform distortion correction on the point cloud data after it has been acquired.

[0114] Correspondingly, the aforementioned ground plane equation determination module 43 is specifically used for:

[0115] Based on the distortion-corrected point cloud data and the external parameters from the radar coordinate system to the imaging device coordinate system, the ground plane equation in the imaging device coordinate system is determined.

[0116] Optionally, the above-described 3D semantic point cloud determination method is applied to a mobile platform, and the distortion correction module includes:

[0117] The IMU data acquisition unit is used to acquire IMU data from the aforementioned mobile platform.

[0118] The distortion correction unit is used to perform distortion correction processing on the acquired point cloud data based on the IMU data.

[0119] Optionally, the distortion correction module described above is specifically used for:

[0120] After determining that the mobile platform is in motion, the acquired point cloud data is subjected to distortion correction processing.

[0121] Optionally, the three-dimensional semantic point cloud determination device 4 provided in this application embodiment further includes:

[0122] The mobile platform speed acquisition module is used to acquire the speed of the mobile platform after acquiring the two-dimensional semantic image.

[0123] Correspondingly, the point cloud data acquisition module 42 mentioned above is specifically used for:

[0124] The point cloud data is acquired when the speed of the mobile platform exceeds a preset speed threshold.

[0125] Optionally, the aforementioned imaging device is a depth camera, which is used to output the aforementioned two-dimensional image and the depth information corresponding to the aforementioned two-dimensional image. The three-dimensional semantic point cloud determination device 4 provided in this application embodiment further includes:

[0126] The depth information acquisition module corresponding to the two-dimensional semantic image is used to acquire the depth information corresponding to the two-dimensional semantic image if the speed of the mobile platform is less than a preset threshold after the speed of the mobile platform is acquired.

[0127] The semantic point cloud determination module of the depth camera is used to determine the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image based on the two-dimensional semantic image and the depth information.

[0128] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0129] Figure 5 This is a schematic diagram of the structure of a mobile platform provided in one embodiment of this application. Figure 5 As shown, the mobile platform 5 of this embodiment includes: at least one processor 50 ( Figure 5 The diagram shows only one processor, a memory 51, and a computer program 52 stored in the memory 51 and executable on at least one processor 50. When the processor 50 executes the computer program 52, it implements the steps in any of the above method embodiments. Of course, the mobile platform also includes an actuator for moving the mobile platform. Figure 5 It is not shown in the document. The actuator may include tracks, wheels, etc., which will not be elaborated here.

[0130] The aforementioned mobile platform 5 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This mobile platform may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of mobile platform 5 and does not constitute a limitation on mobile platform 5. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0131] The processor 50 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0132] In some embodiments, the aforementioned memory 51 may be an internal storage unit of the mobile platform 5, such as a hard disk or memory of the mobile platform 5. In other embodiments, the aforementioned memory 51 may be an external storage device of the mobile platform 5, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the mobile platform 5. Furthermore, the aforementioned memory 51 may include both internal storage units and external storage devices of the mobile platform 5. The aforementioned memory 51 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the aforementioned computer programs. The aforementioned memory 51 may also be used to temporarily store data that has been output or will be output.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0134] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0135] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0136] This application provides a computer program product that, when run on a mobile platform, enables the mobile platform to implement the steps described in the various method embodiments.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / mobile platform, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] It should be noted that the information collection process (such as the facial image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is carried out with the user's knowledge and permission. That is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

Claims

1. A method for determining three-dimensional semantic point clouds, characterized in that, include: A two-dimensional semantic image is acquired, which includes a semantic image of the ground. The two-dimensional semantic image is obtained by processing a two-dimensional image captured by a camera. Acquire point cloud data, which is obtained by low-precision or low-beam radar scanning, and includes point cloud data of the ground. Based on the point cloud data and the external parameters from the radar coordinate system to the imaging device coordinate system, determine the ground plane equation in the imaging device coordinate system; Based on the ground plane equation in the shooting device coordinate system and the preset shooting device model, determine the depth information of the ground plane points in the shooting device coordinate system; Based on the depth information of ground plane points in the coordinate system of the shooting device and the model of the shooting device, the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image is determined.

2. The three-dimensional semantic point cloud determination method as described in claim 1, characterized in that, The step of determining the ground plane equation in the camera's coordinate system based on the point cloud data and the extrinsic parameters from the radar coordinate system to the camera's coordinate system includes: Based on the point cloud data, a first normal vector perpendicular to the ground plane and the height of the radar reaching the ground plane are determined. Select a data point from the point cloud data, and determine the first ground point of the data point on the ground plane based on the height of the radar reaching the ground plane. Based on the extrinsic parameters from the radar coordinate system to the imaging device coordinate system, the first normal vector and the first ground point are transformed from the radar coordinate system to the imaging device coordinate system to obtain the second normal vector and the second ground point. The equation of the ground plane in the coordinate system of the shooting device is determined based on the second normal vector and the second ground point.

3. The three-dimensional semantic point cloud determination method as described in claim 1, characterized in that, After acquiring the point cloud data, the process also includes: The acquired point cloud data is subjected to distortion correction processing; The step of determining the ground plane equation in the camera's coordinate system based on the point cloud data and the extrinsic parameters from the radar coordinate system to the camera's coordinate system includes: Based on the distortion-corrected point cloud data and the extrinsic parameters from the radar coordinate system to the imaging device coordinate system, the ground plane equation in the imaging device coordinate system is determined.

4. The three-dimensional semantic point cloud determination method as described in claim 3, characterized in that, The three-dimensional semantic point cloud determination method is applied to a mobile platform, and the distortion correction processing of the acquired point cloud data includes: Acquire IMU data from the mobile platform; The point cloud data is subjected to distortion correction based on the IMU data.

5. The three-dimensional semantic point cloud determination method as described in claim 3, characterized in that, The three-dimensional semantic point cloud determination method is applied to a mobile platform, and the distortion correction processing of the acquired point cloud data includes: After determining that the mobile platform is in a moving state, the acquired point cloud data is subjected to distortion correction processing.

6. The method for determining three-dimensional semantic point clouds as described in any one of claims 1 to 5, characterized in that, The three-dimensional semantic point cloud determination method is applied to a mobile platform, and after acquiring the two-dimensional semantic image, it further includes: Obtain the speed of the mobile platform; The acquisition of point cloud data includes: The point cloud data is acquired when the speed of the mobile platform exceeds a preset speed threshold.

7. The three-dimensional semantic point cloud determination method as described in claim 6, characterized in that, The imaging device is a depth camera, which is used to output the two-dimensional image and the depth information corresponding to the two-dimensional image. After acquiring the speed of the mobile platform, the method further includes: When the speed of the mobile platform is less than a preset threshold, depth information corresponding to the two-dimensional semantic image is acquired; Based on the two-dimensional semantic image and the depth information, the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image is determined.

8. A three-dimensional semantic point cloud determination device, characterized in that, include: A two-dimensional semantic image acquisition module is used to acquire two-dimensional semantic images, including semantic images of the ground, which are obtained by processing two-dimensional images captured by the imaging device. A point cloud data acquisition module is used to acquire point cloud data, which is obtained by low-precision or low-beam radar scanning, and includes point cloud data of the ground. The ground plane equation determination module is used to determine the ground plane equation in the camera coordinate system based on the point cloud data and the external parameters from the radar coordinate system to the camera coordinate system. The depth information determination module is used to determine the depth information of ground plane points in the shooting device coordinate system based on the ground plane equation in the shooting device coordinate system and the preset shooting device model. The three-dimensional semantic point cloud determination module is used to determine the three-dimensional semantic point cloud corresponding to the two-dimensional semantic image based on the depth information of ground plane points in the coordinate system of the shooting device and the model of the shooting device.

9. A mobile platform comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1 to 7 to be performed.

Images