Three-dimensional reconstruction method, four-dimensional labeling method, device and equipment

By identifying and filtering dynamic obstacle areas, combining RTK positioning data and camera color information, high-quality target point cloud data is generated, which solves the problem of sparse point cloud data and dynamic obstacles, and improves the performance of autonomous driving technology.

CN119048664APending Publication Date: 2024-11-29HAOMO TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310619809.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-29
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In the prior art, point cloud data is too sparse to accurately express regional features, and the movement of dynamic obstacles affects three-dimensional reconstruction, resulting in low quality of four-dimensional labeling and affecting the performance of autonomous driving technology.

Method used

By obtaining initial point cloud data, camera-shooting images and RTK positioning data, the pre-trained dynamic obstacle identification network is used to identify dynamic obstacle areas, filter the obstacle point cloud data, and color the camera color information and splicing the RTK positioning data to generate target points Cloud data.

Benefits of technology

It improves the accuracy of three-dimensional reconstruction and the quality of four-dimensional annotations, improves the performance of autonomous driving technology, and can efficiently rebuild the road surface and surrounding environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048664A_ABST
    Figure CN119048664A_ABST
Patent Text Reader

Abstract

The invention is suitable for the technical field of automatic driving, and provides a three-dimensional reconstruction method, a four-dimensional labeling method, a device and equipment, and the three-dimensional reconstruction method comprises the steps: obtaining a plurality of frames of initial point cloud data, a camera shooting image corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; inputting the image shot by the camera into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle area in the image shot by the camera; filtering out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data; coloring the first point cloud data according to color information of an image shot by the camera to obtain second point cloud data; and splicing the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data. According to the scheme, the influence of dynamic obstacles on point cloud data splicing is effectively avoided, the accuracy of three-dimensional reconstruction is improved, and the road surface and environment conditions can be efficiently reconstructed in an automatic driving scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and in particular relates to a three-dimensional reconstruction method, a four-dimensional labeling method, a device and equipment. Background Art

[0002] Four-dimensional labeling is an important part of realizing autonomous driving technology. It determines the performance upper limit of autonomous driving technology to a certain extent. Whether the real world can be accurately reconstructed in three dimensions directly affects the quality of four-dimensional labeling.

[0003] At present, the point cloud data directly obtained is too sparse to express the characteristics of the corresponding area based on the sparse point cloud data. Furthermore, the real world is dynamically changing during autonomous driving. Whether it is the change of vehicle position and posture or the continuous movement of dynamic obstacles, it will affect the 3D reconstruction, making it impossible to accurately perform 4D annotation. Summary of the invention

[0004] The embodiments of the present application provide a three-dimensional reconstruction method, a four-dimensional labeling method, a device and an apparatus, which can solve the above-mentioned problems.

[0005] In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method, including: obtaining several frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; inputting the camera-captured images into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle area in the camera-captured images; filtering out the obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data; coloring the first point cloud data according to the color information of the camera-captured images to obtain second point cloud data; and splicing the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data.

[0006] Furthermore, the obstacle point cloud data corresponding to the dynamic obstacle area is filtered out from the corresponding initial point cloud data to obtain the first point cloud data, including: projecting the initial point cloud data onto the corresponding camera-captured image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image; obtaining the obstacle point cloud data corresponding to the dynamic obstacle area according to the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image; filtering the obstacle point cloud data from the corresponding initial point cloud data to obtain the first point cloud data.

[0007] Furthermore, the initial point cloud data is projected onto the corresponding camera-captured image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image, including the steps of: obtaining the coordinate information of the points in the initial point cloud data, the coordinate transformation matrix, and the intrinsic parameter matrix corresponding to the camera; obtaining the target coordinate information according to the coordinate information of the points in the initial point cloud data, the coordinate transformation matrix, and the intrinsic parameter matrix corresponding to the camera; determining the pixel points corresponding to the target coordinate information from the camera-captured image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image.

[0008] Furthermore, the first point cloud data is colored according to the color information of the image taken by the camera to obtain the second point cloud data, including: obtaining the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera; according to the color information of each pixel point in the image taken by the camera and the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera, the points in the first point cloud data are colored to obtain the second point cloud data.

[0009] Furthermore, according to the RTK positioning data corresponding to the initial point cloud data, the second point cloud data is spliced ​​to obtain the target point cloud data, including: according to the RTK positioning data corresponding to the initial point cloud data and a preset iterative closest point algorithm, the second point cloud data is spliced ​​to obtain the target point cloud data.

[0010] Furthermore, before inputting the camera-captured image into the pre-trained dynamic obstacle recognition network to obtain the dynamic obstacle area in the camera-captured image, it includes: acquiring a plurality of training images; wherein the training images are marked with dynamic obstacle areas; and iteratively training the dynamic obstacle recognition network according to the plurality of training images, a preset loss function and a preset network optimization algorithm until a preset iteration termination condition is met, thereby obtaining the pre-trained dynamic obstacle recognition network.

[0011] In a second aspect, an embodiment of the present application provides a four-dimensional labeling method, comprising: acquiring target point cloud data according to the three-dimensional reconstruction method of the first aspect; identifying target objects in the autonomous driving scene according to the target point cloud data, labeling the target objects, and obtaining four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs, and traffic lights.

[0012] In a third aspect, an embodiment of the present application provides a three-dimensional reconstruction device, including: an acquisition unit, used to acquire several frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; an obstacle recognition unit, used to input the camera-captured images into a pre-trained dynamic obstacle recognition network to obtain dynamic obstacle areas in the camera-captured images; a filtering unit, used to filter out obstacle point cloud data corresponding to the dynamic obstacle areas from the corresponding initial point cloud data to obtain first point cloud data; a coloring unit, used to color the first point cloud data according to the color information of the camera-captured images to obtain second point cloud data; and a splicing and reconstruction unit, used to splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data.

[0013] Furthermore, the filtering unit includes: a first acquisition unit, used to project the initial point cloud data onto the corresponding camera-captured image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image; a second acquisition unit, used to obtain obstacle point cloud data corresponding to the dynamic obstacle area based on the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image; and a first filtering unit, used to filter out the obstacle point cloud data from the corresponding initial point cloud data to obtain first point cloud data.

[0014] Furthermore, the first acquisition unit is specifically used to: obtain the coordinate information of the midpoint in the initial point cloud data, the coordinate transformation matrix and the intrinsic parameter matrix corresponding to the camera; obtain the target coordinate information according to the coordinate information of the midpoint in the initial point cloud data, the coordinate transformation matrix and the intrinsic parameter matrix corresponding to the camera; determine the pixel points corresponding to the target coordinate information from the image taken by the camera, and obtain the correspondence between the points in the initial point cloud data and the pixel points in the image taken by the camera.

[0015] Furthermore, the coloring unit is specifically used to: obtain the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera; color the points in the first point cloud data according to the color information of each pixel point in the image taken by the camera and the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera to obtain the second point cloud data.

[0016] Furthermore, the splicing and reconstruction unit is specifically used to: splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data and a preset iterative closest point algorithm to obtain the target point cloud data.

[0017] Furthermore, the three-dimensional reconstruction device also includes: a third acquisition unit, used to acquire a plurality of training images; wherein the training images are marked with dynamic obstacle areas; a training unit, used to iteratively train the dynamic obstacle recognition network according to the plurality of training images, a preset loss function and a preset network optimization algorithm, until a preset iteration termination condition is met, thereby obtaining a pre-trained dynamic obstacle recognition network.

[0018] In a fourth aspect, an embodiment of the present application provides a four-dimensional labeling device, comprising: an acquisition unit, used to acquire target point cloud data according to the three-dimensional reconstruction method of the first aspect; a labeling unit, used to identify target objects in the autonomous driving scene according to the target point cloud data, label the target objects, and obtain four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs, and traffic lights.

[0019] In the fifth aspect, an embodiment of the present application provides a device that establishes a data connection with a laser radar, a camera, and an RTK positioning device. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first or second aspect above.

[0020] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect above is implemented.

[0021] In the embodiment of the present application, on the one hand, the device obtains several frames of initial point cloud data, camera images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; the camera images are input into the pre-trained dynamic obstacle recognition network to obtain the dynamic obstacle area in the camera images; the obstacle point cloud data corresponding to the dynamic obstacle area is filtered out from the corresponding initial point cloud data to obtain the first point cloud data; the first point cloud data is colored according to the color information of the camera images to obtain the second point cloud data; the second point cloud data is spliced ​​according to the RTK positioning data corresponding to the initial point cloud data to obtain the target point cloud data. The above method uses the pre-trained dynamic obstacle recognition network to identify the dynamic obstacle area in the camera image, and then filters out the obstacle point cloud data corresponding to the dynamic obstacle area from the initial point cloud data, thereby avoiding the influence of dynamic obstacles on the splicing of point cloud data, and also realizes the coloring of point cloud data based on the color information of the camera image, and finally uses the RTK positioning data to splice the colored sparse point cloud data into dense point cloud data, which effectively improves the accuracy of three-dimensional reconstruction, and can efficiently reconstruct the road conditions and surrounding environment conditions in the autonomous driving scenario.

[0022] On the other hand, the device obtains target point cloud data according to the three-dimensional reconstruction method of the first aspect; identifies target objects in the autonomous driving scene according to the target point cloud data, and labels the target objects to obtain four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs, and traffic lights. The above method can more accurately perform four-dimensional labeling based on the acquired target point cloud data, thereby effectively improving the quality of the four-dimensional labeling data, and further improving the performance of the autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 is a schematic flow chart of a three-dimensional reconstruction method provided in the first embodiment of the present application;

[0025] Figure 2 is a schematic flow chart of S103 in a three-dimensional reconstruction method provided in the first embodiment of the present application;

[0026] Figure 3 is a schematic flow chart of S103 in a three-dimensional reconstruction method provided in the first embodiment of the present application;

[0027] Figure 4 is a schematic flow chart of a four-dimensional labeling method provided in the second embodiment of the present application;

[0028] Figure 5 is a schematic diagram of a three-dimensional reconstruction device provided in a third embodiment of the present application;

[0029] Figure 6 is a schematic diagram of a four-dimensional labeling device provided in a fourth embodiment of the present application;

[0030] Figure 7 is a schematic diagram of a three-dimensional reconstruction device provided in a fifth embodiment of the present application;

[0031] Figure 8 It is a schematic diagram of a four-dimensional labeling device provided in the sixth embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0033] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0034] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0035] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0036] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0038] See also Figure 1 , Figure 11 is a schematic flow chart of a 3D reconstruction method provided in the first embodiment of the present application. In this embodiment, the execution subject of a 3D reconstruction method is a device with a 3D reconstruction function, which may be a personal computer, a server, etc. Figure 1 The three-dimensional reconstruction method shown may include:

[0039] S101: Acquire several frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data.

[0040] The device obtains several frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data.

[0041] The initial point cloud data may refer to visual point cloud data or laser point cloud data.

[0042] In the embodiment of the present application, the initial point cloud data obtained is laser point cloud data. Specifically, the device establishes a data connection with a laser radar (Lidar, Light Detection and Ranging) to obtain the initial point cloud data from the laser radar.

[0043] Each point in the initial point cloud data collected by the laser radar corresponds to at least coordinate information and laser reflection intensity.

[0044] In the autonomous driving scenario, the lidar can be deployed on the vehicle, and the specific deployment details are not limited here.

[0045] In addition to arranging the laser radar, it is also necessary to arrange a camera to obtain camera-captured images. Among them, the camera-captured images in the embodiment of the present application are 2D images. Since 2D images can be used more efficiently for dynamic obstacle capture and color extraction, the efficiency of three-dimensional reconstruction can be improved to a certain extent by taking 2D images by the camera. In other optional implementations, the camera-captured images can also be three-dimensional images, etc.

[0046] The following is an explanation of the camera images corresponding to the initial point cloud data. In short, the camera images acquired at the same time as the initial point cloud data can be used as the camera images corresponding to the initial point cloud data. However, since the acquisition frequency of the laser radar and the shooting frequency of the camera may be inconsistent, when the acquisition frequency of the laser radar is higher than the shooting frequency of the camera, the initial point cloud data acquired at a preset time interval can be corresponded to the same camera image. When the acquisition frequency of the laser radar is lower than the shooting frequency of the camera, the camera image closest to the acquisition time of the initial point cloud data can be used as the camera image corresponding to the initial point cloud data.

[0047] While acquiring the initial point cloud data, the device also acquires the RTK positioning data corresponding to the initial point cloud data. Among them, RTK (Real Time Kinematic), that is, carrier phase differential technology, can provide real-time three-dimensional positioning results of the target in the specified coordinate system with centimeter-level accuracy. Specifically, the device can obtain the vehicle's posture information in the autonomous driving scenario based on the RTK positioning data corresponding to the initial point cloud data.

[0048] S102: Inputting the camera-captured image into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle region in the camera-captured image.

[0049] The device inputs the camera-captured image into a pre-trained dynamic obstacle recognition network to obtain the dynamic obstacle area in the camera-captured image.

[0050] Among them, dynamic obstacles refer to obstacles that can move in the real world, such as vehicles, pedestrians, etc.

[0051] The dynamic obstacle area may be expressed in the form of a regular rectangular frame or an irregular connected area.

[0052] In an optional implementation, the dynamic obstacle recognition network may refer to an image segmentation network Unet, the network structure of which mainly includes downsampling, upsampling, and skip connections, and is often used to segment a predetermined target from an image.

[0053] The following describes the training process of the dynamic obstacle recognition network. The device obtains several training images, wherein the training images are annotated with dynamic obstacle areas. Afterwards, the device iteratively trains the dynamic obstacle recognition network according to the several training images, the preset loss function and the preset network optimization algorithm until the preset iteration termination condition is met, thereby obtaining a pre-trained dynamic obstacle recognition network.

[0054] Specifically, the device outputs several training images to the initialized dynamic obstacle recognition network to obtain the currently recognized dynamic obstacle area, obtains loss data according to the currently recognized dynamic obstacle area, the dynamic obstacle area annotated by the training image, and a preset loss function, and determines whether the loss data meets the preset iteration termination condition. When the preset iteration termination condition is not met, the trainable parameters in the dynamic obstacle recognition network are optimized according to the loss data and the preset network optimization algorithm, and the above network training process is repeated until the preset iteration termination condition is met to obtain a pre-trained dynamic obstacle recognition network.

[0055] S103: Filter out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data.

[0056] The device filters out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data.

[0057] Specifically, the device can determine whether the pixel point corresponding to each point in the initial point cloud data is within the dynamic obstacle area of ​​the image captured by the camera. If so, the point is an obstacle point and is filtered out from the initial point cloud data.

[0058] In an alternative embodiment, see Figure 2 , Figure 2 is a schematic flow chart of S103 in a three-dimensional reconstruction method provided in the first embodiment of the present application, where S103 includes:

[0059] S1031: Projecting the initial point cloud data onto the corresponding camera-captured image to obtain a correspondence between points in the initial point cloud data and pixel points in the camera-captured image.

[0060] The device projects the initial point cloud data onto the corresponding camera-captured image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image.

[0061] In order to obtain the correspondence between the points in the initial point cloud data and the pixel points in the image captured by the camera, the initial point cloud data needs to be projected onto the corresponding image captured by the camera.

[0062] Specifically, the device obtains the coordinate information of the midpoint in the initial point cloud data, the coordinate transformation matrix, and the intrinsic parameter matrix corresponding to the camera, and then obtains the target coordinate information based on the coordinate information of the midpoint in the initial point cloud data, the coordinate transformation matrix, and the intrinsic parameter matrix corresponding to the camera. Finally, the pixel point corresponding to the target coordinate information is determined from the image taken by the camera to obtain the correspondence between the points in the initial point cloud data and the pixel points in the image taken by the camera.

[0063] Among them, the coordinate transformation matrix refers to the coordinate transformation rotation and translation matrix. According to the coordinate transformation matrix and the intrinsic parameter matrix corresponding to the camera, the coordinate information of the points in the initial point cloud data can be converted into the coordinate information of the pixel points in the image taken by the camera.

[0064] In an optional embodiment, if there are multiple cameras collaborating in a shooting operation, then the device needs to obtain the correction matrix corresponding to the camera. The correction matrix refers to the correction matrix from each camera to the target camera. The target camera can be the camera of any camera, and the remaining cameras use it as a reference to obtain the corresponding correction matrix. Furthermore, in this case, the device needs to obtain the target coordinate information based on the coordinate information of the midpoint of the initial point cloud data, the coordinate transformation matrix, the correction matrix corresponding to the camera, and the intrinsic parameter matrix corresponding to the camera.

[0065] S1032: Obtain obstacle point cloud data corresponding to the dynamic obstacle area according to the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image.

[0066] If the correspondence between the points in the initial point cloud data and the pixel points in the camera image has been obtained, then the obstacle point cloud data corresponding to the dynamic obstacle area can be obtained, that is, it is possible to determine which points in the initial point cloud data belong to the obstacle points that need to be filtered.

[0067] S1033: Filter out obstacle point cloud data from the corresponding initial point cloud data to obtain first point cloud data.

[0068] The device filters out the obstacle point cloud data from the corresponding initial point cloud data to obtain the first point cloud data. Since the obstacle point cloud data no longer exists in the first point cloud data, the influence of dynamic obstacles on the subsequent splicing of the point cloud data can be effectively avoided.

[0069] S104: Coloring the first point cloud data according to the color information of the image captured by the camera to obtain second point cloud data.

[0070] The device colors the first point cloud data according to the color information of the image taken by the camera to obtain the second point cloud data.

[0071] The color space used for the color information is not limited here.

[0072] Specifically, see Figure 3 , Figure 3 is a schematic flow chart of S104 in a three-dimensional reconstruction method provided in the first embodiment of the present application, where S104 includes:

[0073] S1041: Obtain the correspondence between points in the first point cloud data and pixel points in the image captured by the camera.

[0074] How to obtain the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera is not elaborated here. It can be understood that the correspondence between the points in the initial point cloud data and the pixel points in the image taken by the camera is obtained, that is, the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera is obtained.

[0075] S1042: Coloring the points in the first point cloud data according to the color information of each pixel in the image taken by the camera and the correspondence between the points in the first point cloud data and the pixels in the image taken by the camera to obtain second point cloud data.

[0076] The device colors the points in the first point cloud data according to the color information of each pixel in the image captured by the camera and the correspondence between the points in the first point cloud data and the pixels in the image captured by the camera, thereby obtaining the second point cloud data. Specifically, the device configures the color information of the points in the first point cloud data as the color information of the corresponding pixels in the image captured by the camera, thereby coloring the points in the first point cloud data, thereby obtaining the second point cloud data.

[0077] S105: splicing the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data.

[0078] The device obtains the vehicle's posture information based on the RTK positioning data corresponding to the initial point cloud data, and then aligns the second point cloud data of the current frame with the second point cloud data of the next frame based on the posture information, and so on, completing the alignment of two adjacent frames of second point cloud data, and splicing several frames of second point cloud data after alignment to obtain the target point cloud data.

[0079] The registration refers to determining corresponding points in two adjacent frames of second point cloud data.

[0080] In an optional embodiment, since there may be errors in the posture information obtained based on the RTK positioning data, the device can splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data and a preset iterative closest point algorithm to obtain the target point cloud data.

[0081] The iterative closest point algorithm (ICP) iteratively calculates the optimal coordinate transformation based on the least squares method, namely the rotation matrix R and the translation matrix T, so as to minimize the error function, thereby making the registration more precise and the acquired target point cloud data more accurate.

[0082] In this embodiment, the device obtains several frames of initial point cloud data, camera images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; the camera images are input into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle area in the camera images; the obstacle point cloud data corresponding to the dynamic obstacle area is filtered out from the corresponding initial point cloud data to obtain the first point cloud data; the first point cloud data is colored according to the color information of the camera images to obtain the second point cloud data; the second point cloud data is spliced ​​according to the RTK positioning data corresponding to the initial point cloud data to obtain the target point cloud data. The above method uses a pre-trained dynamic obstacle recognition network to identify the dynamic obstacle area in the camera images, and then filters out the obstacle point cloud data corresponding to the dynamic obstacle area from the initial point cloud data, thereby avoiding the influence of dynamic obstacles on the splicing of point cloud data, and also realizes the coloring of point cloud data based on the color information of the camera images, and finally uses the RTK positioning data to splice the colored sparse point cloud data into dense point cloud data, which effectively improves the accuracy of three-dimensional reconstruction, and can efficiently reconstruct the road conditions and surrounding environment conditions in the autonomous driving scenario.

[0083] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0084] See also Figure 4 , Figure 4 is a schematic flow chart of a four-dimensional labeling method provided in the second embodiment of the present application. In this embodiment, the execution subject of a four-dimensional labeling method is a device with a four-dimensional labeling function, which can be a personal computer, a server, etc. Figure 4 The four-dimensional annotation method shown may include:

[0085] S201: According to the three-dimensional reconstruction method of the first embodiment, target point cloud data is acquired.

[0086] The device acquires target point cloud data according to the 3D reconstruction method of the first embodiment. For how to acquire target point cloud data, refer to the first embodiment.

[0087] S202: Identify target objects in the autonomous driving scene according to the target point cloud data, and label the target objects to obtain four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs, and traffic lights.

[0088] The device identifies the target objects in the autonomous driving scene based on the target point cloud data, labels the target objects, and obtains four-dimensional labeled data.

[0089] Among them, the specific form of four-dimensional annotation (also known as 4D annotation) is not limited here. It can be understood that four-dimensional annotation is a annotation method that combines the time dimension and the space dimension.

[0090] Target objects include at least lane lines, ground signs, and traffic lights, and other objects that affect the performance of autonomous driving technology may also be marked.

[0091] In this embodiment, the device obtains target point cloud data according to the three-dimensional reconstruction method of the first aspect; identifies target objects in the autonomous driving scene according to the target point cloud data, and labels the target objects to obtain four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs, and traffic lights. The above method can more accurately perform four-dimensional labeling based on the acquired target point cloud data, thereby effectively improving the quality of the four-dimensional labeling data, and further improving the performance of the autonomous driving technology.

[0092] See also Figure 5 , Figure 5 3D reconstruction apparatus provided in the third embodiment of the present application. Figure 1 For details, please refer to the steps in the corresponding embodiment. Figure 1 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 5 , the three-dimensional reconstruction device 5 comprises:

[0093] An acquisition unit 51 is used to acquire a plurality of frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data;

[0094] The obstacle recognition unit 52 is used to input the camera-captured image into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle area in the camera-captured image;

[0095] A filtering unit 53, configured to filter out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data;

[0096] A coloring unit 54, used to color the first point cloud data according to the color information of the image taken by the camera to obtain second point cloud data;

[0097] The splicing and reconstruction unit 55 is used to splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain the target point cloud data.

[0098] Furthermore, the filtering unit 53 includes: a first acquisition unit, used to project the initial point cloud data onto the corresponding camera-captured image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image; a second acquisition unit, used to obtain obstacle point cloud data corresponding to the dynamic obstacle area based on the correspondence between the points in the initial point cloud data and the pixel points in the camera-captured image; and a first filtering unit, used to filter out the obstacle point cloud data from the corresponding initial point cloud data to obtain first point cloud data.

[0099] Furthermore, the first acquisition unit is specifically used to: obtain the coordinate information of the midpoint in the initial point cloud data, the coordinate transformation matrix and the intrinsic parameter matrix corresponding to the camera; obtain the target coordinate information according to the coordinate information of the midpoint in the initial point cloud data, the coordinate transformation matrix and the intrinsic parameter matrix corresponding to the camera; determine the pixel points corresponding to the target coordinate information from the image taken by the camera, and obtain the correspondence between the points in the initial point cloud data and the pixel points in the image taken by the camera.

[0100] Furthermore, the coloring unit 54 is specifically used to: obtain the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera; color the points in the first point cloud data according to the color information of each pixel point in the image taken by the camera and the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera to obtain the second point cloud data.

[0101] Furthermore, the splicing and reconstruction unit 55 is specifically used to splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data and a preset iterative closest point algorithm to obtain the target point cloud data.

[0102] Furthermore, the three-dimensional reconstruction device 5 also includes: a third acquisition unit, used to acquire a plurality of training images; wherein the training images are marked with dynamic obstacle areas; a training unit, used to iteratively train the dynamic obstacle recognition network according to the plurality of training images, a preset loss function and a preset network optimization algorithm, until a preset iteration termination condition is met, thereby obtaining a pre-trained dynamic obstacle recognition network.

[0103] See also Figure 6 , Figure 6 Schematic diagram of a four-dimensional labeling device provided in the fourth embodiment of the present application. The units included are used to execute Figure 4 For details, please refer to the steps in the corresponding embodiment. Figure 4 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 6 , the four-dimensional labeling device 6 includes:

[0104] An acquisition unit 61 is used to acquire target point cloud data according to the three-dimensional reconstruction method of the first embodiment;

[0105] The labeling unit 62 is used to identify the target objects in the autonomous driving scene according to the target point cloud data, label the target objects, and obtain four-dimensional labeling data; wherein the target objects at least include lane lines, ground signs and traffic lights.

[0106] The devices mentioned above in the embodiments of the present application are divided into three-dimensional reconstruction devices and four-dimensional labeling devices, and the three-dimensional reconstruction devices and the four-dimensional labeling devices are explained separately.

[0107] See also Figure 7 , Figure 7 Schematic diagram of a 3D reconstruction device provided in the fifth embodiment of the present application. Figure 7 As shown, the 3D reconstruction device 7 of this embodiment establishes data connection with the laser radar 100, the camera 200 and the RTK positioning device 300, and the 3D reconstruction device 7 includes: a processor 70, a memory 71 and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a 3D reconstruction program. When the processor 70 executes the computer program 72, the steps in the above-mentioned 3D reconstruction method embodiments are implemented, such as Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of the acquisition unit 51 to the stitching and reconstruction unit 55 are shown.

[0108] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the three-dimensional reconstruction device 7. For example, the computer program 72 may be divided into an acquisition unit, an obstacle recognition unit, a filtering unit, a coloring unit, and a splicing and reconstruction unit, and the specific functions of each unit are as follows:

[0109] An acquisition unit, used to acquire a plurality of frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data;

[0110] An obstacle recognition unit is used to input the camera-captured image into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle area in the camera-captured image;

[0111] A filtering unit, used to filter out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data;

[0112] A coloring unit, used to color the first point cloud data according to the color information of the image taken by the camera, so as to obtain second point cloud data;

[0113] The splicing and reconstruction unit is used to splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain the target point cloud data.

[0114] The three-dimensional reconstruction device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the three-dimensional reconstruction device 7 and does not constitute a limitation of the three-dimensional reconstruction device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the three-dimensional reconstruction device may also include input and output devices, network access devices, buses, etc.

[0115] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0116] The memory 71 may be an internal storage unit of the 3D reconstruction device 7, such as a hard disk or memory of the 3D reconstruction device 7. The memory 71 may also be an external storage device of the 3D reconstruction device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the 3D reconstruction device 7. Further, the 3D reconstruction device 7 may also include both an internal storage unit and an external storage device of the 3D reconstruction device 7. The memory 71 is used to store the computer program and other programs and data required by the 3D reconstruction device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0117] See also Figure 8 , Figure 8 Schematic diagram of a four-dimensional labeling device provided in the sixth embodiment of the present application. Figure 8As shown, the four-dimensional labeling device 8 of this embodiment establishes a data connection with the laser radar 100, the camera 200 and the RTK positioning device 300, and the four-dimensional labeling device 8 includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a three-dimensional reconstruction program. When the processor 80 executes the computer program 82, the steps in the above-mentioned three-dimensional reconstruction method embodiments are implemented, such as Figure 4 Alternatively, when the processor 80 executes the computer program 82, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 6 The functions of obtaining units and labeling units are shown.

[0118] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 82 in the four-dimensional labeling device 8. For example, the computer program 82 may be divided into an acquisition unit and a labeling unit, and the specific functions of each unit are as follows:

[0119] An acquisition unit, used for acquiring target point cloud data according to the three-dimensional reconstruction method of the first embodiment;

[0120] The labeling unit is used to identify the target objects in the autonomous driving scene according to the target point cloud data, label the target objects, and obtain four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs and traffic lights.

[0121] The four-dimensional labeling device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the four-dimensional labeling device 8 and does not constitute a limitation of the four-dimensional labeling device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the four-dimensional labeling device may also include input and output devices, network access devices, buses, etc.

[0122] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0123] The memory 81 may be an internal storage unit of the four-dimensional annotation device 8, such as a hard disk or memory of the four-dimensional annotation device 8. The memory 81 may also be an external storage device of the four-dimensional annotation device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the four-dimensional annotation device 8. Furthermore, the four-dimensional annotation device 8 may also include both an internal storage unit and an external storage device of the four-dimensional annotation device 8. The memory 81 is used to store the computer program and other programs and data required by the four-dimensional annotation device. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0124] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0125] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above-mentioned method embodiments when executing the computer program.

[0126] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0127] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0131] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire several frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; Inputting the camera-photographed image into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle region in the camera-photographed image; Filtering out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data; Coloring the first point cloud data according to color information of the image captured by the camera to obtain second point cloud data; The second point cloud data is spliced ​​according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The step of filtering out obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data includes: Projecting the initial point cloud data onto a corresponding camera-photographed image to obtain a correspondence between points in the initial point cloud data and pixel points in the camera-photographed image; Obtaining obstacle point cloud data corresponding to the dynamic obstacle area according to the correspondence between the points in the initial point cloud data and the pixel points in the image captured by the camera; The obstacle point cloud data is filtered out from the corresponding initial point cloud data to obtain the first point cloud data.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The projecting of the initial point cloud data onto the corresponding camera-photographed image to obtain the correspondence between the points in the initial point cloud data and the pixel points in the camera-photographed image comprises the steps of: Obtaining coordinate information of the midpoint of the initial point cloud data, a coordinate transformation matrix, and an intrinsic parameter matrix corresponding to the camera; Obtaining target coordinate information according to the coordinate information of the midpoint of the initial point cloud data, the coordinate transformation matrix, and the intrinsic parameter matrix corresponding to the camera; The pixel points corresponding to the target coordinate information are determined from the image taken by the camera, and the corresponding relationship between the points in the initial point cloud data and the pixel points in the image taken by the camera is obtained.

4. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that: The step of coloring the first point cloud data according to the color information of the image captured by the camera to obtain the second point cloud data includes: Acquire the correspondence between the points in the first point cloud data and the pixel points in the image taken by the camera; According to the color information of each pixel in the image taken by the camera and the correspondence between the points in the first point cloud data and the pixels in the image taken by the camera, the points in the first point cloud data are colored to obtain the second point cloud data.

5. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that: The step of splicing the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data includes: The second point cloud data is spliced ​​according to the RTK positioning data corresponding to the initial point cloud data and a preset iterative closest point algorithm to obtain the target point cloud data.

6. The three-dimensional reconstruction method according to any one of claims 1 to 3, characterized in that: Before inputting the camera-photographed image into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle region in the camera-photographed image, the method includes: Acquire a plurality of training images; wherein the training images are marked with dynamic obstacle areas; According to the plurality of training images, a preset loss function and a preset network optimization algorithm, the dynamic obstacle recognition network is iteratively trained until a preset iteration termination condition is met, thereby obtaining the pre-trained dynamic obstacle recognition network.

7. A four-dimensional labeling method, characterized in that: include: According to the three-dimensional reconstruction method according to any one of claims 1 to 6, obtaining target point cloud data; Target objects in the autonomous driving scene are identified according to the target point cloud data, and the target objects are labeled to obtain four-dimensional labeled data; wherein the target objects include at least lane lines, ground signs, and traffic lights.

8. A three-dimensional reconstruction device, characterized in that: include: An acquisition unit, used to acquire a plurality of frames of initial point cloud data, camera-captured images corresponding to the initial point cloud data, and RTK positioning data corresponding to the initial point cloud data; An obstacle recognition unit, used for inputting the camera-photographed image into a pre-trained dynamic obstacle recognition network to obtain a dynamic obstacle region in the camera-photographed image; A filtering unit, configured to filter out the obstacle point cloud data corresponding to the dynamic obstacle area from the corresponding initial point cloud data to obtain first point cloud data; a coloring unit, configured to color the first point cloud data according to color information of the image captured by the camera, so as to obtain second point cloud data; The splicing and reconstruction unit is used to splice the second point cloud data according to the RTK positioning data corresponding to the initial point cloud data to obtain target point cloud data.

9. A four-dimensional labeling device, characterized in that: include: An acquisition unit, configured to acquire target point cloud data according to the three-dimensional reconstruction method according to any one of claims 1 to 6; A labeling unit is used to identify target objects in the autonomous driving scene according to the target point cloud data, label the target objects, and obtain four-dimensional labeling data; wherein the target objects include at least lane lines, ground signs, and traffic lights.

10. A device that establishes a data connection with a laser radar, a camera, and an RTK positioning device, the device comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 or claim 7 when executing the computer program.