A point cloud data processing method and related apparatus

Abstract

The application discloses a point cloud data processing method in the field of artificial intelligence, which can be applied to an automatic driving vehicle. The method comprises the following steps: acquiring a first image collected by a first camera; acquiring first point cloud data collected by a laser radar; inputting the first image and the first point cloud data into a target network to obtain first target point cloud data; wherein the number of beams corresponding to the first target point cloud data is greater than the number of beams corresponding to the first point cloud data, the target network is used for extracting a depth feature of the first image, performing feature extraction and up-sampling on the first point cloud data to obtain a first feature, and performing fusion processing on the depth feature and the first feature to obtain the first target point cloud data. According to the scheme, high-beam point cloud data can be obtained without deploying a high-beam laser radar, and the cost of the automatic driving vehicle is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), and in particular to a point cloud data processing method and related apparatus. BACKGROUND

[0002] Artificial intelligence is to use digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science, which aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0003] Autonomous driving is a mainstream application in the field of artificial intelligence. Autonomous driving technology relies on computer vision, radar, monitoring devices and global positioning systems to cooperate, so that motor vehicles can drive automatically without human operation. Autonomous vehicles use various computing systems to help transport passengers from one location to another. Some autonomous vehicles may require some initial input or continuous input from the operator, such as a navigator, a driver, or a passenger. Autonomous vehicles allow operators to switch from manual mode to autonomous mode or a mode between the two. Since autonomous driving technology does not require humans to drive motor vehicles, it can theoretically effectively avoid human driving errors, reduce traffic accidents, and improve highway transportation efficiency. Therefore, autonomous driving technology is increasingly valued.

[0004] In the autonomous driving scenario, a laser radar is usually used to obtain three-dimensional (3D) information around the vehicle to realize 3D target detection. Generally, the detection accuracy of the 3D target is closely related to the original point cloud data provided by the laser radar. For example, a low-beam laser radar (such as a 16-line laser radar) has fewer scanning beams, and the obtained point cloud data is sparse, resulting in low 3D target detection accuracy; a high-beam laser radar (such as a 128-line laser radar) obtains more dense point cloud data, which can effectively improve the 3D target detection accuracy.

[0005] At present, autonomous vehicles generally use high-beam laser radars to obtain 3D information to ensure the 3D target detection accuracy. However, high-beam laser radars are expensive, resulting in high cost of autonomous driving systems. Summary of the Invention

[0006] This application provides a method and related apparatus for processing point cloud data. It acquires images captured by a camera and point cloud data collected by a low-beam LiDAR, and inputs the images and point cloud data into a fusion network to fuse the depth information of the images and the low-beam point cloud data to obtain high-beam point cloud data. This allows for the acquisition of high-beam point cloud data without deploying a high-beam LiDAR, reducing the cost of autonomous vehicles.

[0007] This application provides a method for processing point cloud data, applicable to the onboard computing platform of an autonomous vehicle. The method includes: acquiring a first image captured by a first camera; acquiring first point cloud data captured by a LiDAR, wherein both the first camera and the LiDAR are installed on the autonomous vehicle, and the LiDAR can be, for example, a low-cost low-beam LiDAR, such as a 4-beam, 8-beam, or 16-beam LiDAR. The first image and the first point cloud data are input into a target network to obtain first target point cloud data; wherein the number of beams corresponding to the first target point cloud data is greater than the number of beams corresponding to the first point cloud data, the target network is used to extract depth features from the first image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and fuse the depth features and the first feature to obtain the first target point cloud data. In simple terms, this target network is used to predict point cloud data collected by a high-beam LiDAR in the same scene based on the image and the point cloud data collected by the low-beam LiDAR. That is, the first target point cloud data is the point cloud data collected by the high-beam lidar in the same scene, which is predicted. Therefore, the number of beams corresponding to the first target point cloud data is greater than the number of beams corresponding to the first point cloud data.

[0008] In this solution, images captured by a camera and point cloud data collected by a low-beam LiDAR are acquired. These images and point cloud data are then input into a fusion network to fuse the depth information of the images and the low-beam point cloud data, resulting in high-beam point cloud data. This allows for the acquisition of high-beam point cloud data without deploying a high-beam LiDAR, reducing the cost of autonomous vehicles.

[0009] Optionally, in one possible implementation, when the resolution of the first image is not high, the target network may further be used to extract depth features from the first image, upsample the depth features to obtain a second feature, extract and upsample the first point cloud data to obtain a first feature, and fuse the second feature and the first feature to obtain the first target point cloud data. That is, the target network further upsamples the extracted depth features of the first image to obtain higher-resolution features, and then fuses them with the upsampled features corresponding to the first point cloud data to obtain the target point cloud data. For example, if the depth image corresponding to the first image corresponds to the point cloud data collected by a 32-beam LiDAR, and the first point cloud data is 16-beam point cloud data, the target network can extract the depth features of the first image and upsample the depth features to obtain a second feature, which corresponds to the features of the point cloud data collected by a 64-beam LiDAR. The target network further extracts and upsamples the first point cloud data to obtain a first feature, which corresponds to the features of the point cloud data collected by the 64-beam LiDAR. Thus, by fusing the second feature and the first feature, the first target point cloud data can be obtained.

[0010] Optionally, in one possible implementation, the target network includes a first network and a second network; the step of inputting the first image and the first point cloud data into the target network to obtain first target point cloud data includes: inputting the first image into the first network to obtain a depth image, wherein the first network is used to predict the depth image based on a monocular image; inputting the depth image and the first point cloud data into the second network to obtain the first target point cloud data, wherein the second network is used to extract depth features from the depth image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data. That is, the onboard computing platform in an autonomous vehicle can first predict the depth image corresponding to the first image through the first network, and then, based on the obtained depth image and low-beam point cloud data, fuse it through the second network to obtain high-beam point cloud data.

[0011] In this scheme, by dividing the target network into two networks with different functions, each network can be trained independently to ensure the performance of each network and thus improve the final prediction result. In other words, by using an independent first network to extract the depth feature information of the image, the accuracy of the depth feature extraction can be guaranteed, thereby ensuring the accuracy of subsequent predictions.

[0012] Optionally, in one possible implementation, the second network includes a third network and a fourth network; the step of inputting the depth image and the point cloud data into the second network to obtain the first target point cloud data includes: inputting the first point cloud data into the third network to obtain second point cloud data, wherein the number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data; the third network is used to perform feature extraction and upsampling on the first point cloud data to obtain the second point cloud data; inputting the depth image and the second point cloud data into the fourth network to obtain the first target point cloud data; the fourth network is used to extract depth features from the depth image, perform feature extraction on the second point cloud data to obtain the first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data. In simpler terms, the second network is further divided into two independent networks (i.e., the third network and the fourth network). The third network is used to predict high-wire bundle point cloud data based only on low-wire bundle point cloud data, while the fourth network is used to predict the target point cloud data based on the depth image and the point cloud data. The number of wire bundles corresponding to the target point cloud data predicted by the fourth network and the point cloud data input to the fourth network can be the same.

[0013] In this scheme, since the point cloud data with low line bundles is predicted to obtain the point cloud data with high line bundles before the fusion of image depth information and point cloud data, it can ensure that the point cloud data prediction effect is still good even when effective image depth information cannot be obtained, thus improving the point cloud data prediction effect in various scenarios.

[0014] Optionally, in one possible implementation, the second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a fourth convolutional layer; the first set of network layers is used to extract and upsample the depth image to obtain a second feature; the second set of network layers is used to extract and upsample the first point cloud data to obtain the first feature; the third set of network layers is used to concatenate the first feature and the second feature to obtain a third feature; and the fourth convolutional layer is used to process the third feature to obtain the first target point cloud data.

[0015] Optionally, in one possible implementation, the target network includes a first network and a second network; the step of inputting the first image and the point cloud data into the target network to obtain first target point cloud data includes: inputting the first image into the first network and obtaining a feature map generated by the first network, the feature map including depth information in the first image, and the first network being used to predict a depth image based on a monocular image. For example, when the first network is used to predict a depth image, after inputting the first image into the first network, a feature map generated by the penultimate layer of the first network can be obtained, the feature map having the same size as the first image. The feature map and the first point cloud data are then input into the second network to obtain the first target point cloud data, and the second network is used to predict the first target point cloud data based on the feature map and the first point cloud data.

[0016] Since the feature map actually includes the depth feature information in the first image, the second network can fuse the depth feature information in the feature map with the first point cloud data to predict the first target point cloud data, thereby eliminating the need to extract the depth feature information from the depth image and improving the processing efficiency of the point cloud data.

[0017] Optionally, in one possible implementation, the method further includes: projecting the first point cloud data onto the first image to obtain third point cloud data corresponding to the first image, wherein the first point cloud data includes the third point cloud data. The first image and the third point cloud data are then input into a target network to obtain first target point cloud data. Specifically, the process of projecting the first point cloud data onto the first image may involve performing a coordinate system transformation on the first point cloud data, that is, transforming the coordinate system of the first point cloud data to the same coordinate system as the first image, thereby transforming the coordinates of the first point cloud data into coordinates under the coordinate system of the first image; then, based on the coordinates corresponding to each pixel in the first image, determining the points in the first point cloud data with the same coordinates, thereby obtaining the third point cloud data corresponding to the first image.

[0018] In simple terms, before fusing point cloud data with the depth information of the image, the part of the point cloud data that corresponds to the current image can be identified. Then, prediction of the point cloud data is performed based on this part of the point cloud data and the current image. Specifically, the part of the point cloud data that corresponds to the current image covers the same field of view as the current image; that is, the scene scanned by this part of the point cloud data is the same as the scene captured by the current image.

[0019] Optionally, in one possible implementation, the method further includes: acquiring a second image captured by a second camera; projecting the first point cloud data onto the second image to obtain fourth point cloud data corresponding to the second image, wherein the first point cloud data includes the fourth point cloud data; inputting the second image and the fourth point cloud data into a target network to obtain second target point cloud data; and stitching the first target point cloud data and the second target point cloud data together to obtain third target point cloud data.

[0020] In simple terms, autonomous vehicles can acquire images from multiple cameras and project point cloud data collected by LiDAR onto different images to obtain point cloud data for each image. Based on these images and their corresponding point cloud data, autonomous vehicles can fuse multiple target point cloud data sets with different viewing angles. Finally, these multiple target point cloud data sets with different viewing angles are stitched together to obtain target point cloud data with a larger viewing angle.

[0021] Optionally, in one possible implementation, the first image is an image captured by a monocular camera.

[0022] Optionally, in one possible implementation, the first camera and the lidar have been calibrated.

[0023] A second aspect of this application provides a method for training a network, comprising: acquiring a training sample set, the training sample set including a first image captured by a camera, first point cloud data captured by a first lidar, and second point cloud data captured by a second lidar, wherein the number of beams of the first lidar is less than the number of beams of the second lidar; inputting the first image and the first point cloud data into a network to be trained to obtain predicted point cloud data, wherein the network to be trained is used to extract depth features from the first image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth features and the first feature to obtain first target point cloud data; acquiring a first loss value based on the second point cloud data and the predicted point cloud data, the first loss value being used to describe the difference between the second point cloud data and the predicted point cloud data; updating the parameters of the network to be trained based on the first loss value until the network training conditions are met to obtain a target network.

[0024] Optionally, in one possible implementation, the network to be trained includes a first network and a second network; the step of inputting the first image and the first point cloud data into the network to be trained includes: inputting the first image into the first network to obtain a predicted depth image, wherein the first network is used to predict the depth image based on a monocular image; inputting the predicted depth image and the first point cloud data into the second network to obtain the predicted point cloud data, wherein the second network is used to extract depth features from the predicted depth image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data.

[0025] Optionally, in one possible implementation, the second network includes a third network and a fourth network; the step of inputting the predicted depth image and the first point cloud data into the second network to obtain the predicted point cloud data includes: inputting the first point cloud data into the third network to obtain second point cloud data, wherein the number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data; the third network is used to perform feature extraction and upsampling on the first point cloud data to obtain the second point cloud data; inputting the predicted depth image and the second point cloud data into the fourth network to obtain the predicted point cloud data, wherein the fourth network is used to extract depth features from the predicted depth image, perform feature extraction on the second point cloud data to obtain the first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data.

[0026] Optionally, in one possible implementation, the network to be trained includes a first network and a second network; the step of inputting the first image and the first point cloud data into the network to be trained includes: inputting the first image into the first network to obtain a feature map generated by the first network, the feature map including depth information in the first image, the first network being used to predict a depth image based on a monocular image; inputting the feature map and the first point cloud data into the second network to obtain the predicted point cloud data, the second network being used to predict the predicted point cloud data based on the feature map and the first point cloud data.

[0027] Optionally, in one possible implementation, the first image is an image captured by a monocular camera.

[0028] Optionally, in one possible implementation, the first camera and the lidar have been calibrated.

[0029] A third aspect of this application provides a point cloud data processing apparatus, including an acquisition unit and a processing unit; the acquisition unit is used to acquire a first image captured by a first camera; the acquisition unit is also used to acquire first point cloud data captured by a lidar; the processing unit is used to input the first image and the first point cloud data into a target network to obtain first target point cloud data; wherein the number of line bundles corresponding to the first target point cloud data is greater than the number of line bundles corresponding to the first point cloud data, the target network is used to extract depth features from the first image, upsample the depth features to obtain a first feature, extract and upsample the first point cloud data to obtain a second feature, and fuse the first feature and the second feature to obtain the first target point cloud data.

[0030] Optionally, in one possible implementation, the target network includes a first network and a second network; the processing unit is further configured to input the first image into the first network to obtain a depth image, the first network being configured to predict the depth image based on the monocular image; the processing unit is further configured to input the depth image and the first point cloud data into the second network to obtain the first target point cloud data, the second network being configured to extract depth features from the depth image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data.

[0031] Optionally, in one possible implementation, the second network includes a third network and a fourth network; the processing unit is further configured to input the first point cloud data into the third network to obtain second point cloud data, wherein the number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data, and the third network is configured to perform feature extraction and upsampling on the first point cloud data to obtain the second point cloud data; the processing unit is further configured to input the depth image and the second point cloud data into the fourth network to obtain the first target point cloud data, wherein the fourth network is configured to extract the depth features of the depth image, perform feature extraction on the second point cloud data to obtain the first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data.

[0032] Optionally, in one possible implementation, the second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a stitching layer; the first set of network layers is used to extract and upsample the depth image to obtain a first feature; the second set of network layers is used to extract and upsample the first point cloud data to obtain a second feature; the stitching layer is used to stitch the first feature and the second feature to obtain a third feature; the third set of network layers is used to process the third feature to obtain the first target point cloud data.

[0033] Optionally, in one possible implementation, the target network includes a first network and a second network; the processing unit is further configured to input the first image into the first network and obtain a feature map generated by the first network, the feature map including depth information in the first image, the first network being used to predict a depth image based on the monocular image; the processing unit is further configured to input the feature map and the first point cloud data into the second network to obtain the first target point cloud data, the second network being used to predict the first target point cloud data based on the feature map and the first point cloud data.

[0034] Optionally, in one possible implementation, the processing unit is further configured to project the first point cloud data onto the first image to obtain third point cloud data corresponding to the first image, wherein the first point cloud data includes the third point cloud data; the processing unit is further configured to input the first image and the third point cloud data into a target network to obtain first target point cloud data.

[0035] Optionally, in one possible implementation, the acquisition unit is further configured to acquire a second image captured by a second camera; the processing unit is further configured to project the first point cloud data onto the second image to obtain fourth point cloud data corresponding to the second image, wherein the first point cloud data includes the fourth point cloud data; the processing unit is further configured to input the second image and the fourth point cloud data into a target network to obtain second target point cloud data; and the processing unit is further configured to stitch the first target point cloud data and the second target point cloud data together to obtain third target point cloud data.

[0036] Optionally, in one possible implementation, the first image is an image captured by a monocular camera.

[0037] Optionally, in one possible implementation, the first camera and the lidar have been calibrated.

[0038] A fourth aspect of this application provides a training apparatus, comprising: an acquisition unit and a processing unit; the acquisition unit is configured to acquire a training sample set, the training sample set including a first image captured by a camera, first point cloud data captured by a first lidar, and second point cloud data captured by a second lidar, wherein the number of beams of the first lidar is less than the number of beams of the second lidar; the processing unit is configured to input the first image and the first point cloud data into a network to be trained to obtain predicted point cloud data, wherein the network to be trained is configured to extract depth features from the first image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth features and the first feature to obtain first target point cloud data; the processing unit is further configured to acquire a first loss value based on the second point cloud data and the predicted point cloud data, the first loss value being used to describe the difference between the second point cloud data and the predicted point cloud data; the processing unit is further configured to update the parameters of the network to be trained based on the first loss value until the network training conditions are met to obtain a target network.

[0039] Optionally, in one possible implementation, the network to be trained includes a first network and a second network; the processing unit is further configured to input the first image into the first network to obtain a predicted depth image, the first network being used to predict the depth image based on a monocular image; the processing unit is further configured to input the predicted depth image and the first point cloud data into the second network to obtain the predicted point cloud data, the second network being used to extract depth features from the predicted depth image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data.

[0040] Optionally, in one possible implementation, the second network includes a third network and a fourth network; the processing unit is further configured to input the first point cloud data into the third network to obtain second point cloud data, wherein the number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data, and the third network is configured to perform feature extraction and upsampling on the first point cloud data to obtain the second point cloud data; the processing unit is further configured to input the predicted depth image and the second point cloud data into the fourth network to obtain the predicted point cloud data, wherein the fourth network is configured to extract depth features from the predicted depth image, perform feature extraction on the second point cloud data to obtain the first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data.

[0041] Optionally, in one possible implementation, the second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a stitching layer; the first set of network layers is used to extract and upsample the depth image to obtain a second feature; the second set of network layers is used to extract and upsample the first point cloud data to obtain the first feature; the stitching layer is used to stitch the first feature and the second feature to obtain a third feature; the third set of network layers is used to process the third feature to obtain the first target point cloud data.

[0042] Optionally, in one possible implementation, the network to be trained includes a first network and a second network; the processing unit is further configured to input the first image into the first network to obtain a feature map generated by the first network, the feature map including depth information in the first image, and the first network being used to predict a depth image based on the monocular image; the processing unit is further configured to input the feature map and the first point cloud data into the second network to obtain the predicted point cloud data, and the second network being used to predict the predicted point cloud data based on the feature map and the first point cloud data.

[0043] Optionally, in one possible implementation, the first image is an image captured by a monocular camera.

[0044] Optionally, in one possible implementation, the camera and the first lidar have been calibrated.

[0045] The fifth aspect of this application provides an autonomous driving device, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in any possible implementation of the first or second aspect is performed.

[0046] A sixth aspect of this application provides a computer-readable storage medium that may include computer-readable instructions that, when executed on a computer, cause the computer to perform the methods described in any possible implementation of the first or second aspect.

[0047] The seventh aspect of this application provides a computer program product that may include computer-readable instructions that, when executed on a computer, cause the computer to perform the methods described in any possible implementation of the first or second aspect.

[0048] The eighth aspect of this application provides an autonomous vehicle that may include processing circuitry and storage circuitry configured to perform the methods described in the first aspect or any possible implementation thereof.

[0049] The ninth aspect of this application provides a circuit system that may include processing circuitry configured to perform the methods described in the first aspect or any possible implementation thereof.

[0050] The tenth aspect of this application provides a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0051] The eleventh aspect of this application provides a chip system that may include a processor for supporting an autonomous driving device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system may further include a memory for storing program instructions and data necessary for a server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0052] For the specific implementation steps of the second to eleventh aspects of this application and the various possible implementation methods, as well as the beneficial effects brought about by each possible implementation method, please refer to the description of the various possible implementation methods in the first aspect, which will not be repeated here. Attached Figure Description

[0053] Figure 1a A schematic diagram of the structure of a vehicle provided in an embodiment of this application;

[0054] Figure 1b A schematic diagram of the structure of a computer system provided in an embodiment of this application;

[0055] Figure 2 A flowchart illustrating a point cloud data processing method 200 provided in an embodiment of this application;

[0056] Figure 3 This is a schematic diagram of a point cloud data processing flow provided in an embodiment of this application;

[0057] Figure 4a This application provides a schematic diagram of a second network processing flow for point cloud data.

[0058] Figure 4b This is a schematic diagram of another point cloud data processing flow provided in an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of another point cloud data processing flow provided in an embodiment of this application;

[0060] Figure 6 This is a schematic diagram of another point cloud data processing flow provided in an embodiment of this application;

[0061] Figure 7 This is a schematic diagram of another point cloud data processing flow provided in an embodiment of this application;

[0062] Figure 8 A schematic diagram of a point cloud data processing apparatus 800 provided in an embodiment of this application;

[0063] Figure 9 A schematic diagram of the structure of the training device 900 provided in the embodiments of this application;

[0064] Figure 10 A schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application;

[0065] Figure 11 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0066] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0067] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0068] To facilitate understanding of this solution, the embodiments of this application first combine... Figure 1a The structure of the vehicle is described, and the method provided in the embodiments of this application can be applied as follows: Figure 1a On the vehicle shown. See also Figure 1a , Figure 1a This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application.

[0069] In one embodiment, the autonomous vehicle 100 can be configured in a fully or partially autonomous driving mode. For example, the autonomous vehicle 100 can control itself while in autonomous driving mode, and can determine the current state of the vehicle and its surrounding environment through human intervention, determine the possible behaviors of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the probability of that other vehicle performing a possible behavior, and control the autonomous vehicle 100 based on the determined information. When the autonomous vehicle 100 is in autonomous driving mode, it can be configured to operate without human interaction.

[0070] The autonomous vehicle 100 may include various subsystems, such as a mobility system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 101, and a user interface 116. Optionally, the autonomous vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the autonomous vehicle 100 may be interconnected via wired or wireless means.

[0071] The propulsion system 102 may include components that provide powered motion for the autonomous vehicle 100. In one embodiment, the propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels 121. The engine 118 may be an internal combustion engine, an electric motor, an air-compressed engine, or other types of engine combinations, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air-compressed engine. The engine 118 converts the energy source 119 into mechanical energy.

[0072] Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. Energy source 119 can also provide power to other systems of the autonomous vehicle 100.

[0073] The transmission 120 can transmit mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other components, such as a clutch. The drive shaft may include one or more axles that can be coupled to one or more wheels 121.

[0074] Sensor system 104 may include several sensors for sensing information about the environment surrounding autonomous vehicle 100. For example, sensor system 104 may include a positioning system 122 (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU) 124, radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors monitoring the internal systems of autonomous vehicle 100 (e.g., in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). This detection and identification is a key function for the safe operation of autonomous vehicle 100.

[0075] The positioning system 122 can be used to estimate the geographic location of the autonomous vehicle 100. The IMU 124 is used to sense changes in the position and orientation of the autonomous vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope.

[0076] Radar 126 can use radio signals to sense objects in the surrounding environment of the autonomous vehicle 100. In some embodiments, in addition to sensing objects, radar 126 can also be used to sense the speed and / or direction of travel of objects.

[0077] The laser rangefinder 128 can use lasers to sense objects in the environment in which the autonomous vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0078] Camera 130 can be used to capture multiple images of the surrounding environment of the autonomous vehicle 100. Camera 130 can be a still camera or a video camera.

[0079] The control system 106 controls the operation of the autonomous vehicle 100 and its components. The control system 106 may include various elements, including a steering system 132, a throttle 134, a braking unit 136, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0080] The steering system 132 is operable to adjust the forward direction of the autonomous vehicle 100. For example, in one embodiment, it may be a steering wheel system.

[0081] Throttle 134 is used to control the operating speed of engine 118 and thus the speed of autonomous vehicle 100.

[0082] Braking unit 136 is used to control the deceleration of autonomous vehicle 100. Braking unit 136 can use friction to slow down wheel 121. In other embodiments, braking unit 136 can convert the kinetic energy of wheel 121 into electric current. Braking unit 136 may also take other forms to slow down the rotational speed of wheel 121 to control the speed of autonomous vehicle 100.

[0083] Computer vision system 140 is operable to process and analyze images captured by camera 130 to identify objects and / or features in the environment surrounding autonomous vehicle 100. Objects and / or features may include traffic signals, road boundaries, and obstacles. Computer vision system 140 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, computer vision system 140 may be used to map the environment, track objects, estimate object velocities, and so on.

[0084] The route control system 142 is used to determine the driving route of the autonomous vehicle 100. In some embodiments, the route control system 142 may combine data from the global positioning system 122 and one or more predetermined maps to determine the driving route for the autonomous vehicle 100.

[0085] The obstacle avoidance system 144 is used to identify, assess and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100.

[0086] Of course, in one instance, the control system 106 may include additional or alternative components besides those shown and described. Alternatively, some of the components shown above may be reduced.

[0087] The autonomous vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and / or a speaker 152.

[0088] In some embodiments, peripheral device 108 provides a means for a user of autonomous vehicle 100 to interact with user interface 116. For example, onboard computer 148 may provide information to the user of autonomous vehicle 100. User interface 116 may also operate onboard computer 148 to receive user input. Onboard computer 148 may be operated via touchscreen. In other cases, peripheral device 108 may provide a means for autonomous vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from the user of autonomous vehicle 100. Similarly, speaker 152 may output audio to the user of autonomous vehicle 100.

[0089] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication, such as LTE, or 5G cellular communication. The wireless communication system 146 can communicate using WiFi and a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 can communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are also possible. For example, the wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices that can enable public and / or private data communication between vehicles and / or roadside stations.

[0090] Power source 110 can provide power to various components of autonomous vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of autonomous vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, as is the case in some fully electric vehicles.

[0091] Some or all of the functions of the autonomous vehicle 100 are controlled by a computer system 101. The computer system 101 may include at least one processor 113, which executes instructions 115 stored in a non-transitory computer-readable medium such as a data storage device 114. The computer system 101 may also be multiple computing devices that control individual components or subsystems of the autonomous vehicle 100 in a distributed manner.

[0092] Processor 113 can be any conventional processor, such as a commercially available CPU. Alternatively, the processor can be a special-purpose device such as an ASIC or other hardware-based processor. Although Figure 1aThe processor, memory, and other elements of computer system 101 within the same block are functionally illustrated; however, those skilled in the art will understand that the processor, computer, or memory may or may not be stored in the same physical housing. For example, memory may be a hard disk drive or other storage media located in a housing different from that of computer system 101. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering and deceleration components, may each have their own processor, which performs calculations only relevant to the component's specific function.

[0093] In the various aspects described herein, the processor may be located remotely from the vehicle and communicate wirelessly with the vehicle. In other aspects, some of the processes described herein are executed on a processor located within the vehicle, while others are executed by a remote processor, including taking the necessary steps to perform a single operation.

[0094] In some embodiments, the data storage device 114 may include instructions 115 (e.g., program logic) that can be executed by the processor 113 to perform various functions of the autonomous vehicle 100, including those described above. The data storage device 114 may also include additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0095] In addition to instruction 115, data storage device 114 may also store data such as road maps, route information, vehicle position, direction, speed, and other vehicle data, as well as other information. This information can be used by autonomous vehicle 100 and computer system 101 during autonomous, semi-autonomous, and / or manual operation of autonomous vehicle 100.

[0096] User interface 116 is used to provide information to or receive information from users of autonomous vehicle 100. Optionally, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, vehicle-to-everything (V2X) computer 148, microphone 150, and speaker 152.

[0097] Computer system 101 can control the functions of autonomous vehicle 100 based on input received from various subsystems (e.g., mobility system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 101 can utilize input from control system 106 to control steering unit 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 101 is operable to provide control over many aspects of autonomous vehicle 100 and its subsystems.

[0098] Optionally, one or more of these components may be installed separately from or associated with the autonomous vehicle 100. For example, the data storage device 114 may exist partially or completely separately from the autonomous vehicle 100. The components may be communicatively coupled together in a wired and / or wireless manner.

[0099] Optionally, the components described above are merely examples. In actual applications, components in each of the above modules may be added or removed as needed. Figure 1a This should not be construed as a limitation on the embodiments of this application.

[0100] Autonomous vehicles traveling on roads, such as the autonomous vehicle 100 above, can identify objects in their surrounding environment to determine adjustments to their current speed. These objects can be other vehicles, traffic control equipment, or other types of objects. In some examples, each identified object can be considered independently, and based on the object's individual characteristics, such as its current speed, acceleration, and distance from the vehicle, the speed adjustment to be made by the autonomous vehicle can be determined.

[0101] Optionally, the autonomous vehicle 100 or a computing device associated with the autonomous vehicle 100 (such as...) Figure 1a The computer system 101, computer vision system 140, and data storage device 114 can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can be considered together to predict the behavior of a single identified object. The autonomous vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, the autonomous vehicle can determine what steady state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered in determining the speed of the autonomous vehicle 100, such as the lateral position of the autonomous vehicle 100 on the road, the curvature of the road, the proximity of static and dynamic objects, etc.

[0102] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the autonomous vehicle 100 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the autonomous vehicle (e.g., cars in adjacent lanes on the road).

[0103] The aforementioned autonomous vehicle 100 can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, etc., and this application embodiment does not impose any special limitations.

[0104] Please refer to Figure 1b , Figure 1b This is a schematic diagram of the structure of a computer system provided in an embodiment of this application. Figure 1b As shown, the computer system 101 includes a processor 103, which is coupled to a system bus 105. The processor 103 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to the system bus 105. The system bus 105 is coupled to an input / output (I / O) bus via a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with various I / O devices, such as input devices 117 (e.g., keyboard, mouse, touchscreen), media trays (e.g., CD-ROM, multimedia interfaces), transceivers 123 (which can send and / or receive radio communication signals), a camera 155 (which can capture dynamic digital video images), and an external USB port 125. Optionally, the interface connected to the I / O interface 115 can be a USB port.

[0105] The processor 103 can be any conventional processor, including a Reduced Instruction Set Computing (“RISC”) processor, a Complex Instruction Set Computing (“CISC”) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an Application-Specific Integrated Circuit (“ASIC”). Optionally, the processor 103 can be a neural network processor or a combination of a neural network processor and the conventional processors described above.

[0106] Optionally, in the various embodiments described herein, the computer system 101 may be located remotely from the autonomous vehicle and may communicate wirelessly with the autonomous vehicle 100. In other aspects, some of the processes described herein are executed on a processor located within the autonomous vehicle, while others are executed by a remote processor, including taking actions necessary to perform a single manipulation.

[0107] Computer 101 can communicate with software deployment server 149 via network interface 129. Network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a Virtual Private Network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0108] The hard disk drive interface is coupled to the system bus 105. The hardware drive interface is connected to the hard disk drive. The system memory 135 is coupled to the system bus 105. The data running in the system memory 135 may include the operating system 137 and applications 143 of the computer 101.

[0109] An operating system consists of a shell (139) and a kernel (141). The shell (139) is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. It manages the interaction between the user and the operating system: waiting for user input, interpreting user input for the operating system, and handling various types of operating system output.

[0110] The kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0111] Application 143 includes programs related to controlling autonomous driving, such as programs managing the interaction between the autonomous vehicle and obstacles on the road, programs controlling the autonomous vehicle's route or speed, and programs controlling the interaction between the autonomous vehicle and other autonomous vehicles on the road. Application 143 also exists on the system of deploying server 149. In one embodiment, when autonomous driving-related programs 147 need to be executed, computer system 101 can download application 143 from deploying server 14.

[0112] Sensor 153 is associated with computer system 101. Sensor 153 is used to detect the environment surrounding computer 101. For example, sensor 153 can detect animals, cars, obstacles, and pedestrian crossings, etc. Furthermore, the sensor can also detect the environment around the aforementioned animals, cars, obstacles, and pedestrian crossings, such as the environment around the animals, for example, other animals present in the vicinity, weather conditions, ambient light levels, etc. Optionally, if computer 101 is located on an autonomous vehicle, the sensor can be a camera, lidar, infrared sensor, chemical detector, microphone, etc.

[0113] Currently, in autonomous driving scenarios, LiDAR (Light Detection and Ranging) is commonly used to acquire three-dimensional (3D) information about the vehicle's surroundings to detect 3D targets. LiDAR is a crucial sensor for achieving environmental perception in autonomous driving and is suitable for perceiving complex traffic environments. The working principle of LiDAR is to emit a laser beam towards the target; after the laser beam contacts the target, it is reflected and received by the system, thereby determining the target's distance information.

[0114] Specifically, a lidar system mainly consists of four core components: a laser, a receiver, a signal processing unit, and a rotating mechanism. The laser is the laser emitting mechanism in the lidar system. During operation, the laser is activated in pulses, emitting laser light. After the emitted laser light illuminates an obstacle, it is reflected by the obstacle and then converged onto the receiver via a lens assembly. The signal processing unit is responsible for controlling the laser's emission and processing the signals received by the receiver, thereby calculating the distance to the target object. The rotating mechanism is responsible for rotating the aforementioned core components at a stable speed, thereby scanning the plane and generating real-time point cloud data.

[0115] In autonomous driving scenarios, multi-beam LiDAR is typically used to acquire information about the surrounding environment. Multi-beam LiDAR refers to a laser rotating ranging radar that simultaneously emits and receives multiple laser beams. It can identify the height information of objects and acquire a 3D scan map of the surrounding environment. Multi-beam LiDAR has multiple laser emitters distributed vertically, each corresponding to one beam. The scanning of multiple beams is achieved through the rotation of a rotating mechanism. Currently, multi-beam LiDARs on the market have 4, 8, 16, 32, 64, and 128 beams.

[0116] Generally speaking, the detection accuracy of 3D targets is closely related to the raw point cloud data provided by the LiDAR. For example, low-beam LiDAR (such as a 4-beam LiDAR) has fewer scanning beams, resulting in sparser point cloud data and lower 3D target detection accuracy; while high-beam LiDAR (such as a 128-beam LiDAR) acquires denser point cloud data, which can effectively improve the 3D target detection accuracy.

[0117] Currently, autonomous vehicles typically use high-beam LiDAR to acquire 3D information to ensure accurate 3D target detection. However, high-beam LiDAR is expensive, resulting in high costs for autonomous driving systems.

[0118] In view of this, embodiments of this application provide a method for processing point cloud data. This method acquires images captured by a camera and point cloud data collected by a low-beam LiDAR, and inputs the images and point cloud data into a fusion network to fuse the depth information of the image and the low-beam point cloud data to obtain high-beam point cloud data. This allows for the acquisition of high-beam point cloud data without deploying a high-beam LiDAR, reducing the cost of autonomous vehicles and ensuring the detection accuracy of 3D targets.

[0119] The point cloud data processing method provided in this application embodiment can be applied to... Figure 1a The autonomous vehicle shown may have an onboard computing platform deployed on it, which runs... Figure 1b The computer system shown is used to perform the processing of point cloud data provided in the embodiments of this application.

[0120] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0121] (1) Neural Network

[0122] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0123]

[0124] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0125] The function of each layer in a neural network can be expressed using mathematical expressions. To describe it: From a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are... The operation 4 is completed using +b, and the operation 5 is implemented using a(). The term "space" is used here because the objects being classified are not individual things, but a class of things; space refers to the set of all individuals within this class of things. Here, W is the weight vector, where each value represents the weight of a neuron in that layer of the neural network. This vector W determines the spatial transformation from the input space to the output space, as described above; that is, the weights W of each layer control how the space is transformed. The purpose of training the neural network is to ultimately obtain the weight matrix of all layers of the trained neural network (a weight matrix formed by the vectors W from many layers). Therefore, the training process of a neural network is essentially learning how to control the spatial transformation, more specifically, learning the weight matrix.

[0126] Because we want the output of the neural network to be as close as possible to the actual predicted value, we can compare the current network's prediction with the desired target value, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring the parameters of each layer in the neural network). For example, if the network's prediction is too high, the weight vector is adjusted to make it predict lower, and this adjustment is continued until the neural network can predict the actual target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so training the neural network becomes the process of minimizing this loss as much as possible.

[0127] (2) Backpropagation algorithm

[0128] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0129] The method provided in this application will be described below from the perspectives of both the application and training aspects of neural networks.

[0130] The neural network training method provided in this application involves the processing of image and point cloud data. Specifically, it can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data (such as the image and point cloud data in this application) to ultimately obtain a trained data processing model. Furthermore, the point cloud data processing method provided in this application can utilize the aforementioned trained data processing model to input input data (such as the image and point cloud data in this application) into the trained data processing model to obtain output data (such as the target point cloud data in this application). It should be noted that the data processing model training method and the point cloud data processing method provided in this application are inventions based on the same concept, and can also be understood as two parts of a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0131] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a point cloud data processing method 200 provided in an embodiment of this application. Figure 2 As shown, the method 200 includes the following steps.

[0132] Step 201: Acquire the first image captured by the camera.

[0133] Autonomous vehicles can be equipped with one or more cameras to capture images of the surrounding environment, enabling them to identify road conditions, obstacles, traffic signs, and other traffic elements. Generally, during operation, the cameras on the autonomous vehicle acquire images in real time and transmit them to the onboard computing platform. The platform then analyzes the acquired images and, based on the detected traffic elements, controls the autonomous vehicle to slow down, stop, or dodge obstacles.

[0134] In one possible embodiment, the first image may be an image captured by a monocular camera. Compared to a binocular camera, which can capture two images at a time, a monocular camera can only capture one image at a time. Because monocular cameras are simple in structure, small in size, and low in cost, multiple monocular cameras can be installed on autonomous vehicles to achieve multi-directional camera installation and ensure a wider field of view. Furthermore, the first image may also be an image captured by a binocular camera or a multi-camera system; this embodiment does not specifically limit the camera that captures the first image.

[0135] Step 202: Obtain the first point cloud data collected by the lidar.

[0136] During the operation of a lidar system, the lidar emits a laser beam, which illuminates the surface of an object and is reflected. Based on the received reflected laser light, the lidar can obtain information about the object illuminated by the laser. Specifically, the lidar can calculate the coordinates of each point on the surface of the object illuminated by the laser in three-dimensional space. The combination of these points constitutes the laser point cloud, and the coordinate information of these points is the point cloud data. In other words, this first point cloud data can be the point cloud data collected by the lidar in an autonomous vehicle during laser scanning of the surrounding environment.

[0137] In this embodiment, the lidar can be a low-cost low-beam lidar, such as the aforementioned 4-beam lidar, 8-beam lidar, or 16-beam lidar.

[0138] Step 203: Input the first image and the first point cloud data into the target network to obtain the first target point cloud data.

[0139] The target network is used to extract depth features from the first image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and fuse the depth features and the first feature to obtain the first target point cloud data. In simpler terms, this target network is used to predict the point cloud data collected by a high-beam lidar in the same scene based on the image and the point cloud data collected by the low-beam lidar. That is, the first target point cloud data is the predicted point cloud data collected by the high-beam lidar in the same scene; therefore, the number of beams corresponding to the first target point cloud data is greater than the number of beams corresponding to the first point cloud data.

[0140] It is understandable that the number of laser beams corresponding to the first point cloud data can refer to the number of laser beams emitted by the lidar used to collect the first point cloud data. For example, if the lidar collecting the first point cloud data is a 4-beam lidar, the number of laser beams corresponding to the first point cloud data can be 4. For the first target point cloud data, since it is not actually the data collected by the lidar, but rather the point cloud data corresponding to the high-beam lidar predicted by the target network, the number of laser beams corresponding to the first target point cloud data can be understood as the number of laser beams corresponding to the multi-beam lidar predicted by the target network. For example, if the target network is used to predict the point cloud data collected by a 64-beam lidar, the number of laser beams corresponding to the first target point cloud data can be 64; if the target network is used to predict the point cloud data collected by a 128-beam lidar, the number of laser beams corresponding to the first target point cloud data can be 128.

[0141] Since point cloud data is calculated by lidar based on reflected laser light, the more laser beams emitted by the lidar, the denser the point cloud data will be; conversely, the fewer laser beams emitted, the sparser the point cloud data will be. Compared to the first point cloud data, the first target point cloud data has more laser beams and contains a denser number of data points, meaning it contains more information.

[0142] It should be understood that since the first image is captured by a camera, it typically contains complete information about the scene captured by the camera. Compared to LiDAR, which obtains scene information by emitting laser beams, the information contained in the image is much denser. For example, suppose there is a pedestrian in front of the autonomous vehicle in the current driving scenario. The LiDAR on the autonomous vehicle rotates horizontally to obtain point cloud data obtained by laser beam scanning. Among the multiple laser beams emitted by the LiDAR, two laser beams scan past the pedestrian: one laser beam scans past the pedestrian's thigh, and the other laser beam scans past the pedestrian's neck. The point cloud data collected by the LiDAR includes the coordinate information of the pedestrian's thigh and neck. However, the LiDAR cannot collect the coordinate information of the pedestrian's body below the thigh, between the thigh and neck, and above the neck. However, for the camera on the autonomous vehicle, since the pedestrian is within the camera's field of view, the image captured by the camera contains information about the entire pedestrian. Based on this image, depth information of various parts of the pedestrian can be obtained. Since the information contained in the first point cloud data is the coordinate information of the object surface, i.e., depth information, when the image contains more dense depth feature information, by extracting the depth feature information from the image by the target network and fusing the depth feature information with the first point cloud data, a denser first target point cloud data, i.e., a first target point cloud data with more line bundles, can be obtained.

[0143] Understandably, after the target network extracts features from the first point cloud data, it obtains the features of the first point cloud data. The target network then upsamples the features of the first point cloud data to obtain a first feature, the resolution of which is higher than that of the features of the first point cloud data. Thus, after fusing the first feature with the depth features corresponding to the first image, a first target point cloud data with a higher resolution than the first point cloud data can be obtained; that is, the number of line bundles corresponding to the first target point cloud data is greater than the number of line bundles corresponding to the first point cloud data. Optionally, in one possible implementation, when the resolution of the first image is not high, the target network can also be used to extract the depth features of the first image, upsample the depth features to obtain a second feature, extract and upsample the features of the first point cloud data to obtain the first feature, and fuse the second feature and the first feature to obtain the first target point cloud data. That is, the target network also upsamples the extracted depth features of the first image to obtain a higher resolution feature, and then fuses it with the upsampled feature corresponding to the first point cloud data to obtain the target point cloud data. For example, if the depth image corresponding to the first image corresponds to the point cloud data collected by a 32-beam LiDAR, and the first point cloud data is 16-beam point cloud data, the target network can extract the depth features of the first image and upsample the depth features to obtain a second feature. This second feature corresponds to the features of the point cloud data collected by a 64-beam LiDAR. The target network also performs feature extraction and upsampling on the first point cloud data to obtain a first feature, which corresponds to the features of the point cloud data collected by the 64-beam LiDAR. Thus, by fusing the second feature and the first feature, the first target point cloud data can be obtained.

[0144] For example, see Figure 3 , Figure 3 This is a schematic diagram illustrating a point cloud data processing flow provided in an embodiment of this application. Figure 3 As shown, the camera 301 acquires a first image 302, and the low-beam lidar 303 acquires a first point cloud data 304. The first point cloud data 304 is low-beam point cloud data. Then, the first image 302 and the first point cloud data 304 are input into the target network 305 to obtain the first target point cloud data 306 output by the target network 305. The first target point cloud data 306 is high-beam point cloud data.

[0145] In one possible embodiment, the target network can be a trained neural network, such as a convolutional neural network. During the training process, images captured by a camera and point cloud data collected by a low-beam LiDAR can be used as training data, while point cloud data collected by a high-beam LiDAR can be used as the desired output of the target network. The low-beam and high-beam LiDAR collect point cloud data in the same scene. Specifically, during training, the images and point cloud data collected by the low-beam LiDAR are first input into the network to be trained to obtain the prediction result output by the network. Then, based on the point cloud data collected by the high-beam LiDAR and the prediction result, the value of the loss function is determined, and the parameters of the network to be trained are adjusted based on the value of the loss function. This training process is repeated until the loss function converges, thereby obtaining the target network.

[0146] In this embodiment, images captured by a camera and point cloud data collected by a low-beam LiDAR are acquired. These images and point cloud data are then input into a fusion network to fuse the depth information of the images and the low-beam point cloud data, resulting in high-beam point cloud data. This allows for the acquisition of high-beam point cloud data without deploying a high-beam LiDAR, reducing the cost of autonomous vehicles while ensuring the accuracy of 3D target detection.

[0147] In one possible embodiment, the target network may include a first network and a second network. In step 203 above, inputting the first image and the first point cloud data into the target network to obtain first target point cloud data specifically includes: inputting the first image into the first network to obtain a depth image, where the first network is used to predict the depth image based on a monocular image; inputting the depth image and the first point cloud data into the second network to obtain the first target point cloud data, where the second network is used to extract depth features from the depth image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data. That is, the vehicle-mounted computing platform may first predict the depth image corresponding to the first image using the first network, and then, based on the obtained depth image and low-beam point cloud data, fuse them through the second network to obtain high-beam point cloud data.

[0148] In predicting high-beam point cloud data based on images captured by a camera and low-beam point cloud data, it is necessary to extract depth feature information from the image. Therefore, in this embodiment, the target network can be divided into a first network for extracting depth feature information from the image, and a second network for fusing the depth feature information from the image and the low-beam point cloud data. By dividing the target network into two networks with different functions, each network can be trained separately to ensure the performance of each network, thereby improving the final prediction effect. In other words, by using an independent first network to extract the depth feature information from the image, the accuracy of the depth feature information extraction can be guaranteed, thereby ensuring the accuracy of subsequent predictions.

[0149] As a possible example, the first network could be a Deep Ordinal Regression Network (DORN) or other convolutional neural networks for depth estimation. Based on this DORN, monocular depth estimation can be performed to obtain a depth image corresponding to the monocular image. The second network could be a convolutional neural network capable of performing feature extraction, upsampling operations, and feature concatenation. Based on the second network, low-beam point cloud data can be upsampled and fused with the depth image to obtain high-beam point cloud data.

[0150] For example, see Figure 4a , Figure 4a This is a schematic diagram illustrating a second network processing flow for point cloud data, provided as an embodiment of this application. For example... Figure 4aAs shown, the second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a stitching layer. After inputting a depth image into the second network, the first set of network layers in the second network extracts and upsamples features from the depth image to obtain a first feature, which is the depth feature in the depth image. The first set of network layers may include multiple network layers. After inputting first point cloud data into the second network, the second set of network layers in the second network extracts and upsamples features from the first point cloud data to obtain a second feature, which is the upsampled point cloud data feature. The second set of network layers may include multiple network layers. Then, the stitching layer in the second network is used to stitch the first feature and the second feature together to obtain a stitched third feature. This stitching layer can use a concatenation function to stitch the first feature and the second feature together. Finally, the third set of network layers in the second network performs convolution processing on the third feature to obtain the first target point cloud data. The third set of network layers may include multiple network layers. The first set of network layers, the second set of network layers, and the third set of network layers may each include multiple network layers, including but not limited to convolutional layers, activation layers, pooling layers, deconvolutional layers, and / or scaling layers.

[0151] The training process for the first and second networks will be described below. Before training the networks, training data for both networks can be acquired. The training data for the first network includes images captured by the camera and their corresponding true depth images. The training data for the second network includes point cloud data acquired by both low-beam and high-beam LiDAR sensors. During training, the first network can be trained first, followed by the second network. Finally, the first and second networks can be connected and trained together.

[0152] Specifically, during the training process of the first network, the image captured by the camera is first input into the first network to obtain the prediction result output by the first network. Then, based on the real depth image corresponding to the input image and the prediction result, the value of the loss function is determined, and the parameters of the first network are adjusted based on the value of the loss function. The above training process is repeated until the loss function converges, thereby obtaining the trained first network.

[0153] During the training of the second network, the depth image output by the trained first network and the point cloud data collected by the low-beam LiDAR can be used as inputs to the second network to obtain the point cloud data output by the second network. Then, based on the point cloud data output by the second network and the point cloud data collected by the high-beam LiDAR, the value of the loss function is determined, and the parameters of the second network are adjusted based on the value of the loss function. This training process is repeated until the loss function converges, thus obtaining the trained second network. The loss function of the second network can be an L1 norm loss function, and the second network is optimized using stochastic gradient descent until it converges.

[0154] For example, the loss function of the second network can be as shown in Equation 1:

[0155]

[0156] Among them, L(y) g y p ) represents the loss function; y g Point cloud data acquired by a high-beam lidar; y p This represents the point cloud data output by the second network; c is a hyperparameter that can be set to 1.

[0157] After training the first and second networks, they can be connected together to form the target network. Finally, images captured by a camera and point cloud data collected by a low-beam lidar can be used as training data for the target network, while point cloud data collected by a high-beam lidar can be used as the desired output. This process trains the target network to obtain the trained target network.

[0158] For example, see Figure 4b , Figure 4b This is a schematic diagram illustrating another point cloud data processing flow provided in an embodiment of this application. For example... Figure 4b As shown, camera 301 acquires a first image 302, and low-beam lidar 303 acquires first point cloud data 304, which is low-beam point cloud data. Then, the first image 302 is input into a first network 3051 to obtain a depth image 307 output by the first network 3051. Finally, the first point cloud data 304 and the depth image 307 are input into a second network 3052 to obtain first target point cloud data 360 output by the second network 3052, which is high-beam point cloud data.

[0159] In one possible embodiment, the second network may further include a third network and a fourth network.

[0160] In the steps described above, inputting the depth image and the point cloud data into the second network to obtain the first target point cloud data may specifically include: inputting the first point cloud data into the third network to obtain second point cloud data, wherein the number of line bundles corresponding to the second point cloud data is greater than the number of line bundles corresponding to the first point cloud data; the third network is used to perform feature extraction and upsampling on the first point cloud data to obtain the second point cloud data. Inputting the depth image and the second point cloud data into the fourth network to obtain the first target point cloud data, the fourth network is used to extract depth features from the depth image, extract features from the second point cloud data to obtain the first feature, and fuse the depth feature and the first feature to obtain the first target point cloud data.

[0161] In simple terms, the second network is further divided into two independent networks (i.e., the third network and the fourth network). The third network is used to predict high-beam point cloud data based only on low-beam point cloud data, while the fourth network is used to predict the target point cloud data based on the depth image and the point cloud data. The number of beams in the target point cloud data predicted by the fourth network can be the same as the number of beams in the point cloud data input to the fourth network.

[0162] As a possible example, the third network could be a network such as U-Net, Super-Resolution Convolutional Neural Network (SRCNN), Very Deep Network for Super-Resolution (VDSR), or SRDenseNet. The fourth network could be a convolutional neural network capable of performing feature extraction and feature concatenation operations.

[0163] Understandably, image quality is often significantly affected by lighting conditions. Therefore, in some scenarios, the images captured by the camera may not contain effective depth information, thus impacting the prediction performance of high-beam point cloud data. For example, in a completely dark nighttime scene, due to low light intensity, the images captured by the camera may not include relevant information about the current scene. In this case, because the image lacks effective depth information, it is difficult to effectively fuse this image with low-beam point cloud data. Consequently, the resulting high-beam point cloud data differs considerably from the true point cloud data, affecting the prediction performance.

[0164] Based on this, in this embodiment, the second network is further divided into two independent networks. Before fusing the image depth information with the point cloud data, the input low-beam point cloud data is first predicted based on the third network to obtain high-beam point cloud data. Then, the predicted high-beam point cloud data is fused with the image depth information to obtain the fused point cloud data. Since the high-beam point cloud data is obtained by predicting the low-beam point cloud data before fusing the image depth information with the point cloud data, it can be ensured that a good point cloud data prediction effect can still be guaranteed even when effective image depth information cannot be obtained, thus improving the point cloud data prediction effect in various scenarios.

[0165] Similarly, in this embodiment, since the target network includes a first network, a third network, and a fourth network, the first network, the third network, and the fourth network can be trained separately. Finally, the trained first network, the third network, and the fourth network are connected together, and then the connected network is trained uniformly. The training process of the first network is similar to that described above; please refer to the above description for details, which will not be repeated here.

[0166] During the training of the third network, the point cloud data collected by the low-beam lidar can be used as the input of the third network to obtain the high-beam point cloud data output by the third network. Then, based on the point cloud data output by the third network and the point cloud data collected by the high-beam lidar, the value of the loss function is determined, and the parameters of the third network are adjusted based on the value of the loss function. The above training process is repeated until the loss function converges, thereby obtaining the trained third network.

[0167] During the training of the fourth network, the depth image output by the trained first network and the point cloud data output by the trained third network can be used as inputs to the fourth network to obtain the point cloud data output by the fourth network. Then, based on the point cloud data output by the fourth network and the point cloud data collected by the high-beam lidar, the value of the loss function is determined, and the parameters of the fourth network are adjusted based on the value of the loss function. The above training process is repeated until the loss function converges, thereby obtaining the trained fourth network.

[0168] After training the first, third, and fourth networks, these three networks can be connected together to form the target network. Finally, images captured by a camera and point cloud data collected by a low-beam lidar can be used as training data for the target network, while point cloud data collected by a high-beam lidar can be used as the desired output. This process trains the target network to obtain the trained target network.

[0169] For example, see Figure 5 , Figure 5 This is a schematic diagram illustrating another point cloud data processing flow provided in an embodiment of this application. For example... Figure 5 As shown, camera 301 acquires a first image 302, and low-beam lidar 303 acquires first point cloud data 304, which is low-beam point cloud data. Then, the first image 302 is input into a first network 3051 to obtain a depth image 307 output by the first network 3051. The first point cloud data 304 is input into a third network 30521 to obtain a second point cloud data 308 output by the third network 30521, where the first point cloud data is low-beam point cloud data and the second point cloud data 308 is high-beam point cloud data. Finally, the second point cloud data 308 and the depth image 307 are input into a fourth network 30522 to obtain a first target point cloud data 306 output by the fourth network 30522, which is point cloud data fused with depth feature information from the depth image.

[0170] The above embodiments describe the process of predicting the target point cloud data based on the depth image corresponding to the image obtained by the first network and then predicting the target point cloud data based on the depth image and point cloud data. The following will describe the process of predicting the target point cloud data based on the feature map corresponding to the image obtained by the first network and then predicting the target point cloud data based on the feature map and point cloud data.

[0171] In another possible embodiment, the target network may include the first network and the second network as described above. Specifically, inputting the first image and the point cloud data into the target network to obtain first target point cloud data may include: inputting the first image into the first network and obtaining a feature map generated by the first network, the feature map including depth information from the first image, and the first network being used to predict a depth image based on the monocular image. For example, when the first network is used to predict the depth image, after inputting the first image into the first network, a feature map generated by the penultimate layer of the first network can be obtained, and the size of this feature map is the same as that of the first image.

[0172] Then, the feature map and the first point cloud data are input into the second network to obtain the first target point cloud data. The second network is used to predict the first target point cloud data based on the feature map and the first point cloud data. Since the feature map actually includes the depth feature information in the first image, the second network can fuse the depth feature information in the feature map with the first point cloud data to predict the first target point cloud data, thus eliminating the need to extract the depth feature information from the depth image.

[0173] It is understood that in this embodiment, the training process of the first network and the second network is similar to the training process described above, and will not be repeated here.

[0174] For example, see Figure 6 , Figure 6 This is a schematic diagram illustrating another point cloud data processing flow provided in an embodiment of this application. For example... Figure 6 As shown, camera 301 acquires a first image 302, and low-beam lidar 303 acquires first point cloud data 304, which is low-beam point cloud data. Then, the first image 302 is input into a first network 3051 to obtain a feature map 309 generated by the first network 3051. Finally, the first point cloud data 304 and the feature map 309 are input into a second network 3052 to obtain first target point cloud data 360 output by the second network 3052, which is high-beam point cloud data.

[0175] Similarly, for Figure 5 In the embodiment shown, in addition to obtaining the depth image corresponding to the first image through the first network, it is also possible to obtain the feature map corresponding to the first image through the first network, and then input the obtained feature map and the second point cloud data output by the third network into the fourth network to obtain the first target point cloud data output by the fourth network.

[0176] Understandably, because LiDAR includes a rotating mechanism that can rotate the laser 360°, the laser beam can rotate 360° to acquire point cloud data of the environment within a 360° range for autonomous vehicles. However, due to the limited field of view of cameras, the viewing angle (the angle between the line of sight and the camera's perpendicular direction) of a single image captured by a camera is often around 10° to 90°, meaning a single image cannot cover the entire scene within a 360° range.

[0177] Therefore, before fusing point cloud data with the depth information of the image, the portion of the point cloud data corresponding to the current image can be determined. Then, point cloud data prediction is performed based on this portion of the point cloud data and the current image. Specifically, the portion of the point cloud data corresponding to the current image covers the same field of view as the current image, meaning the scene scanned by this portion of the point cloud data is the same as the scene captured by the current image.

[0178] In one possible embodiment, the method 200 described above may further include: projecting the first point cloud data onto the first image to obtain third point cloud data corresponding to the first image, wherein the first point cloud data includes the third point cloud data. Specifically, the process of projecting the first point cloud data onto the first image may involve performing a coordinate system transformation on the first point cloud data, that is, transforming the coordinate system of the first point cloud data to the same coordinate system as the first image, thereby transforming the coordinates of the first point cloud data into coordinates under the coordinate system of the first image; then, based on the coordinates corresponding to each pixel in the first image, determining the points in the first point cloud data with the same coordinates, thereby obtaining the third point cloud data corresponding to the first image. For example, assuming the viewing angle range of the first image is 0° to 30°, the first point cloud data can be projected onto the first image to determine the portion of the first point cloud data with a viewing angle range of 0° to 30°, and this portion of point cloud data can be determined as the third point cloud data corresponding to the first image.

[0179] After obtaining the third point cloud data corresponding to the first image, the first image and the third point cloud data can be input into the target network to obtain the first target point cloud data. The viewing angle range covered by the first target point cloud data is the same as that covered by the first image.

[0180] It is understood that the first and second cameras and the LiDAR mentioned above may have already been calibrated, meaning that the cameras and LiDAR can be calibrated before executing the point cloud data processing method provided in this embodiment. Calibrating the cameras and LiDAR may involve calculating the coordinate relationship between them, so that the coordinates of the point cloud data collected by the LiDAR can be switched to the coordinates of the image collected by the camera.

[0181] After the first camera and the lidar are calibrated, there is a transformation matrix p between the coordinate systems of the lidar and the first camera. t Suppose x i y i The x-coordinate represents the coordinates of a point in the projected point cloud data within the image coordinate system. p y p , z p This represents the coordinates of a point in the point cloud data before projection, within the point cloud coordinate system. After projection, the points whose coordinates lie within the image can be selected based on these coordinates, thus obtaining the corresponding point cloud data for the image. For example, the transformation of points from the point cloud coordinate system to the image coordinate system can be shown in Formula 2:

[0182]

[0183] It should be understood that in order to ensure that autonomous vehicles can acquire information about their surrounding environment, they are typically equipped with multiple cameras, each covering a different field of view. Through the combination of multiple cameras, autonomous vehicles can acquire environmental information with a 360° field of view.

[0184] Based on this, autonomous vehicles can acquire images from multiple cameras and project the point cloud data collected by LiDAR onto different images to obtain point cloud data corresponding to each image. Based on these images and their corresponding point cloud data, autonomous vehicles can fuse multiple target point cloud data sets with different viewing angles. Finally, these multiple target point cloud data sets with different viewing angles are stitched together to obtain target point cloud data with a larger viewing angle.

[0185] For example, the method 200 described above may further include: acquiring a second image captured by a second camera, wherein the first camera and the second camera are installed at different angles, i.e., the viewing angles covered by the first camera and the second camera are different. For example, the viewing angle range of the first camera is 0° to 30°, and the viewing angle range of the second camera is 30° to 80°, i.e., scene information within the range of 0° to 80° can be obtained through the first camera and the second camera. Projecting the first point cloud data onto the second image to obtain fourth point cloud data corresponding to the second image, wherein the first point cloud data includes the fourth point cloud data. The operation of projecting the first point cloud data onto the second image is similar to the operation of projecting the first point cloud data onto the first image, and can be referred to the above description for details, which will not be repeated here. Inputting the second image and the fourth point cloud data into a target network to obtain second target point cloud data. Stitching the first target point cloud data and the second target point cloud data together to obtain third target point cloud data, which is the point cloud data corresponding to the first image and the second image.

[0186] Understandably, during the stitching process of the first target point cloud data and the second target point cloud data, the stitching can be performed based on the coordinate point information in both the first and second target point cloud data. If adjacent coordinate point information exists in the first and second target point cloud data, they can be directly stitched together, meaning the first and second target point cloud data are adjacent. If no adjacent coordinate point information exists in the first and second target point cloud data, the first and second target point cloud data can be replaced with the corresponding point cloud data in the first point cloud data, respectively, thus obtaining the stitched point cloud data.

[0187] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating another point cloud data processing flow provided in an embodiment of this application. For example... Figure 7 As shown, camera 301 acquires a first image 302 and a second image 310, and low-beam lidar 303 acquires first point cloud data 304, which is low-beam point cloud data. Then, the first image 302 and the second image 310 are input into a first network 3051 to obtain a depth image 307 and a depth image 311 output by the first network 3051. The first point cloud data 304 is projected onto the first image 302 to obtain third point cloud data 3041; the first point cloud data 304 is projected onto the second image 310 to obtain fourth point cloud data 3042. Then, the third point cloud data 3041 and the depth image 307 are input into a second network 3052 to obtain first target point cloud data 306 output by the second network 3052. This first target point cloud data 306 is point cloud data that incorporates depth feature information from the depth images. The fourth point cloud data 3042 and the depth image 311 are input into the second network 3052 to obtain the second target point cloud data 312 output by the second network 3052. This second target point cloud data 312 is also point cloud data that incorporates depth feature information from the depth image. Finally, the first target point cloud data 306 and the second target point cloud data 312 are stitched together to obtain the stitched third target point cloud data 313.

[0188] It is worth noting that the above example illustrates the process of stitching together two point cloud datasets. In practical applications, the point cloud datasets corresponding to the images collected by multiple cameras can be determined based on the number of cameras installed in the autonomous vehicle. These point cloud datasets can then be processed predictively and stitched together to obtain point cloud datasets that can cover a wider field of view.

[0189] exist Figures 2 to 7 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 8 , Figure 8This is a schematic diagram of a point cloud data processing device 800 provided in an embodiment of this application. The point cloud data processing device 800 may include an acquisition unit 801 and a processing unit 802. The acquisition unit 801 is used to acquire a first image captured by a first camera; the acquisition unit 801 is also used to acquire first point cloud data captured by a lidar; the processing unit 802 is used to input the first image and the first point cloud data into a target network to obtain first target point cloud data; wherein the number of line bundles corresponding to the first target point cloud data is greater than the number of line bundles corresponding to the first point cloud data, the target network is used to extract depth features from the first image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data.

[0190] Optionally, in one possible implementation, the target network includes a first network and a second network; the processing unit 802 is further configured to input the first image into the first network to obtain a depth image, the first network being configured to predict the depth image based on the monocular image; the processing unit 802 is further configured to input the depth image and the first point cloud data into the second network to obtain the first target point cloud data, the second network being configured to extract depth features from the depth image, perform feature extraction and upsampling on the first point cloud data to obtain a first feature, and perform fusion processing on the depth feature and the first feature to obtain the first target point cloud data.

[0191] Optionally, in one possible implementation, the second network includes a third network and a fourth network; the processing unit 802 is further configured to input the first point cloud data into the third network to obtain second point cloud data, wherein the number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data, and the third network is configured to perform feature extraction and upsampling on the first point cloud data to obtain the second point cloud data; the processing unit 802 is further configured to input the depth image and the second point cloud data into the fourth network to obtain the first target point cloud data, wherein the fourth network is configured to extract the depth features of the depth image, perform feature extraction on the second point cloud data to obtain the first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data.

[0192] Optionally, in one possible implementation, the second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a stitching layer; the first set of network layers is used to extract and upsample the depth image to obtain a second feature; the second set of network layers is used to extract and upsample the first point cloud data to obtain a first feature; the stitching layer is used to stitch the first feature and the second feature to obtain a third feature; the third set of network layers is used to process the third feature to obtain the first target point cloud data.

[0193] Optionally, in one possible implementation, the target network includes a first network and a second network; the processing unit 802 is further configured to input the first image into the first network and obtain a feature map generated by the first network, the feature map including depth information in the first image, the first network being used to predict a depth image based on the monocular image; the processing unit 802 is further configured to input the feature map and the first point cloud data into the second network to obtain the first target point cloud data, the second network being used to predict the first target point cloud data based on the feature map and the first point cloud data.

[0194] Optionally, in one possible implementation, the processing unit 802 is further configured to project the first point cloud data onto the first image to obtain third point cloud data corresponding to the first image, wherein the first point cloud data includes the third point cloud data; the processing unit 802 is further configured to input the first image and the third point cloud data into a target network to obtain first target point cloud data.

[0195] Optionally, in one possible implementation, the acquisition unit 801 is further configured to acquire a second image captured by a second camera; the processing unit 802 is further configured to project the first point cloud data onto the second image to obtain fourth point cloud data corresponding to the second image, wherein the first point cloud data includes the fourth point cloud data; the processing unit 802 is further configured to input the second image and the fourth point cloud data into a target network to obtain second target point cloud data; the processing unit 802 is further configured to perform stitching processing on the first target point cloud data and the second target point cloud data to obtain third target point cloud data.

[0196] Optionally, in one possible implementation, the first image is an image captured by a monocular camera.

[0197] Optionally, in one possible implementation, the first camera and the lidar have been calibrated.

[0198] Please refer to Figure 9 ,Figure 9 This is a schematic diagram of a training device 900 provided in an embodiment of this application. The training device 900 may include an acquisition unit 901 and a processing unit 902. The acquisition unit 901 is used to acquire a training sample set, which includes a first image captured by a camera, first point cloud data captured by a first lidar, and second point cloud data captured by a second lidar, wherein the number of beams of the first lidar is less than the number of beams of the second lidar. The processing unit 902 is used to input the first image and the first point cloud data into a network to be trained to obtain predicted point cloud data, wherein the network to be trained is used to extract depth features from the first image and predict the predicted point cloud data based on the depth features of the first image and the first point cloud data. The processing unit 902 is further used to acquire a first loss value based on the second point cloud data and the predicted point cloud data, the first loss value being used to describe the difference between the second point cloud data and the predicted point cloud data. The processing unit 902 is further used to update the parameters of the network to be trained based on the first loss value until the network training conditions are met, thereby obtaining a target network.

[0199] Optionally, in one possible implementation, the network to be trained includes a first network and a second network; the processing unit 902 is further configured to input the first image into the first network to obtain a predicted depth image, wherein the first network is used to predict the depth image based on the monocular image; the processing unit 902 is further configured to input the predicted depth image and the first point cloud data into the second network to obtain the predicted point cloud data, wherein the second network is used to extract the depth features of the predicted depth image and predict the predicted point cloud data based on the depth features of the depth image and the first point cloud data.

[0200] Optionally, in one possible implementation, the second network includes a third network and a fourth network; the processing unit 902 is further configured to input the first point cloud data into the third network to obtain second point cloud data, wherein the number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data; the processing unit 902 is further configured to input the predicted depth image and the second point cloud data into the fourth network to obtain the predicted point cloud data, wherein the fourth network is configured to extract the depth features of the predicted depth image and predict the predicted point cloud data based on the depth features of the predicted depth image and the second point cloud data.

[0201] Optionally, in one possible implementation, the second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a stitching layer; the first set of network layers is used to extract and upsample the depth image to obtain a second feature; the second set of network layers is used to extract and upsample the first point cloud data to obtain a first feature; the stitching layer is used to stitch the first feature and the second feature to obtain a third feature; the third set of network layers is used to process the third feature to obtain the first target point cloud data.

[0202] Optionally, in one possible implementation, the network to be trained includes a first network and a second network; the processing unit 902 is further configured to input the first image into the first network to obtain a feature map generated by the first network, the feature map including depth information in the first image, and the first network is used to predict a depth image based on the monocular image; the processing unit 902 is further configured to input the feature map and the first point cloud data into the second network to obtain the predicted point cloud data, and the second network is used to predict the predicted point cloud data based on the feature map and the first point cloud data.

[0203] Optionally, in one possible implementation, the first image is an image captured by a monocular camera.

[0204] Optionally, in one possible implementation, the camera and the first lidar have been calibrated.

[0205] It should be noted that the information interaction and execution process between the units in the point cloud data processing device 800 and the training device 900 are different from those in this application. Figures 2 to 7 The various method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0206] This application also provides an autonomous driving vehicle, in conjunction with the above-described embodiments. Figure 1a For a description, please refer to Figure 10 , Figure 10 This is a schematic diagram of the structure of an autonomous vehicle provided in an embodiment of this application, wherein the autonomous vehicle 100 may be equipped with... Figure 8 The point cloud data processing device described in the corresponding embodiment is used to implement Figures 2 to 7 The corresponding embodiments describe the functions of the autonomous vehicle. Since in some embodiments, the autonomous vehicle 100 may also include communication functions, the autonomous vehicle 100 includes, in addition to... Figure 1aThe components shown may further include a receiver 1001 and a transmitter 1002, wherein the processor 113 may include an application processor 1131 and a communication processor 1132. In some embodiments of this application, the receiver 1001, transmitter 1002, processor 113, and data storage device 114 may be connected via a bus or other means.

[0207] Processor 113 controls the operation of the autonomous vehicle. In specific applications, the various components of the autonomous vehicle 100 are coupled together through a bus system, which may include a data bus, as well as a power bus, control bus, and status signal bus. However, for clarity, all buses are referred to as the bus system in the diagram.

[0208] Receiver 1001 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the autonomous vehicle. Transmitter 1002 can be used to output digital or character information through the first interface; transmitter 1002 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1002 may also include a display device such as a display screen.

[0209] In this embodiment of the application, the application processor 1131 is used to execute... Figure 2 The method for processing point cloud data executed by the autonomous vehicle in the corresponding embodiment.

[0210] It should be noted that the specific implementation method of the point cloud data processing method executed by application processor 1131 and the beneficial effects thereof can be found in [reference needed]. Figures 2 to 7 The descriptions in the corresponding method embodiments will not be repeated here.

[0211] This application also provides a computer-readable storage medium storing a program for performing autonomous driving, which, when executed on a computer, causes the computer to perform the aforementioned actions. Figures 2 to 7 The steps performed by the autonomous vehicle (or autonomous driving device) in the method described in the illustrated embodiment.

[0212] This application also provides a computer program product that, when executed on a computer, causes the computer to perform as described above. Figures 2 to 7 The steps performed by the autonomous vehicle in the method described in the illustrated embodiment.

[0213] This application embodiment also provides a circuit system, the circuit system including a processing circuit, the processing circuit being configured to perform the aforementioned... Figures 2 to 7 The steps performed by the autonomous vehicle in the method described in the illustrated embodiment.

[0214] The autonomous driving device or vehicle provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the server to perform the aforementioned operations. Figures 2 to 7 The illustrated embodiment describes a method for processing point cloud data. Optionally, the storage unit can be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit can be an external storage unit located within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0215] For details, please refer to Figure 11 , Figure 11 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 110. The NPU 110 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1103, which is controlled by the controller 1104 to extract matrix data from the memory and perform multiplication operations.

[0216] In some implementations, the arithmetic circuit 1103 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1103 is a two-dimensional pulsating array. The arithmetic circuit 1103 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1103 is a general-purpose matrix processor.

[0217] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1102 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1101 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1108.

[0218] Unified memory 1106 is used to store input and output data. Weight data is directly transferred to weight memory 1102 via direct memory access controller (DMAC) 1105. Input data is also transferred to unified memory 1106 via DMAC.

[0219] The bus interface unit (BIU) 1110 is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1109.

[0220] BIU1110 is used by instruction fetch memory 1109 to fetch instructions from external memory, and also by memory cell access controller 1105 to fetch the original data of input matrix A or weight matrix B from external memory.

[0221] The DMAC is mainly used to move input data from external memory DDR to unified memory 1106, or to weight data to weight memory 1102, or to input data to input memory 1101.

[0222] The vector computation unit 1107 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0223] In some implementations, the vector computation unit 1107 can store the processed output vector in the unified memory 1106. For example, the vector computation unit 1107 can apply linear and / or nonlinear functions to the output of the computation circuit 1103, such as linear interpolation of feature planes extracted by convolutional layers, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1107 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to the computation circuit 1103, for example, for use in subsequent layers of the neural network.

[0224] The instruction fetch buffer 1109 connected to the controller 1104 is used to store the instructions used by the controller 1104.

[0225] Unified memory 1106, input memory 1101, weight memory 1102, and instruction fetch memory 1109 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0226] The operations of each layer in the recurrent neural network can be performed by the arithmetic circuit 1103 or the vector calculation unit 1107.

[0227] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of a program in the first aspect of the method.

[0228] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, and of course, it can also be implemented by special hardware including application-specific integrated circuits, special-purpose CLUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0230] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0231] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Claims

1. A method for processing point cloud data, characterized in that, include: Acquire the first image captured by the first camera; Acquire first point cloud data collected by lidar, the first point cloud data including third point cloud data corresponding to the first image, the third point cloud data being obtained by projecting the first point cloud data onto the first image; The first image and the first point cloud data are input into the target network to obtain the first target point cloud data; Wherein, the number of wire bundles corresponding to the first target point cloud data is greater than the number of wire bundles corresponding to the first point cloud data. The target network is used to extract the depth features of the first image, perform feature extraction and upsampling on the first point cloud data to obtain the first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data. The target network includes a first network and a second network; The step of inputting the first image and the first point cloud data into the target network to obtain the first target point cloud data includes: The first image is input into the first network to obtain a depth image. The first network is used to predict the depth image based on the monocular image. The depth image and the first point cloud data are input into the second network to obtain the first target point cloud data. The second network is used to extract the depth features of the depth image, perform feature extraction and upsampling on the first point cloud data to obtain the first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data.

2. The method according to claim 1, characterized in that, The second network includes a third network and a fourth network; The step of inputting the depth image and the point cloud data into the second network to obtain the first target point cloud data includes: The first point cloud data is input into the third network to obtain the second point cloud data. The number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data. The third network is used to extract features and upsample the first point cloud data to obtain the second point cloud data. The depth image and the second point cloud data are input into the fourth network to obtain the first target point cloud data. The fourth network is used to extract the depth features of the depth image, extract features from the second point cloud data to obtain the first feature, and fuse the depth features and the first feature to obtain the first target point cloud data.

3. The method according to claim 1 or 2, characterized in that, The second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a splicing layer; The first set of network layers is used to extract and upsample the depth image to obtain the second feature; The second set of network layers is used to extract and upsample the first point cloud data to obtain the first feature; The splicing layer is used to splice the first feature and the second feature to obtain the third feature; The third network layer set is used to process the third feature to obtain the first target point cloud data.

4. The method according to claim 1, characterized in that, The target network includes a first network and a second network; The step of inputting the first image and the point cloud data into the target network to obtain the first target point cloud data includes: The first image is input into the first network, and the feature map generated by the first network is obtained. The feature map includes depth information in the first image. The first network is used to predict a depth image based on a monocular image. The feature map and the first point cloud data are input into the second network to obtain the first target point cloud data. The second network is used to predict the first target point cloud data based on the feature map and the first point cloud data.

5. The method according to claim 1 or 2, characterized in that, The method further includes: The first point cloud data is projected onto the first image to obtain the third point cloud data corresponding to the first image, wherein the first point cloud data includes the third point cloud data; The first image and the first point cloud data are input into the target network to obtain the first target point cloud data, including: The first image and the third point cloud data are input into the target network to obtain the first target point cloud data.

6. The method according to claim 5, characterized in that, The method further includes: Acquire the second image captured by the second camera; The first point cloud data is projected onto the second image to obtain the fourth point cloud data corresponding to the second image, wherein the first point cloud data includes the fourth point cloud data; The second image and the fourth point cloud data are input into the target network to obtain the second target point cloud data; The first target point cloud data and the second target point cloud data are stitched together to obtain the third target point cloud data.

7. The method according to claim 1 or 2, characterized in that, The first image is an image captured by a monocular camera.

8. The method according to claim 1 or 2, characterized in that, The first camera and the lidar have been calibrated.

9. A method for training a network, characterized in that, include: A training sample set is obtained, which includes a first image captured by a camera, first point cloud data captured by a first lidar and second point cloud data captured by a second lidar. The number of beams of the first lidar is less than the number of beams of the second lidar. The first point cloud data includes third point cloud data corresponding to the first image. The third point cloud data is obtained by projecting the first point cloud data onto the first image. The first image and the first point cloud data are input into the network to be trained to obtain predicted point cloud data. The network to be trained is used to extract the depth features of the first image, perform feature extraction and upsampling on the first point cloud data to obtain the first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data. Based on the second point cloud data and the predicted point cloud data, a first loss value is obtained, which is used to describe the difference between the second point cloud data and the predicted point cloud data. The parameters of the network to be trained are updated based on the first loss value until the network training conditions are met, and the target network is obtained. The network to be trained includes a first network and a second network; The step of inputting the first image and the first point cloud data into the network to be trained includes: The first image is input into the first network to obtain a predicted depth image. The first network is used to predict the depth image based on the monocular image. The predicted depth image and the first point cloud data are input into the second network to obtain the predicted point cloud data. The second network is used to extract the depth features of the predicted depth image, perform feature extraction and upsampling on the first point cloud data to obtain the first feature, and perform fusion processing on the depth features and the first feature to obtain the first target point cloud data.

10. The method according to claim 9, characterized in that, The second network includes a third network and a fourth network; The step of inputting the predicted depth image and the first point cloud data into the second network to obtain the predicted point cloud data includes: The first point cloud data is input into the third network to obtain the second point cloud data. The number of wire bundles corresponding to the second point cloud data is greater than the number of wire bundles corresponding to the first point cloud data. The third network is used to extract features and upsample the first point cloud data to obtain the second point cloud data. The predicted depth image and the second point cloud data are input into the fourth network to obtain the predicted point cloud data. The fourth network is used to extract the depth features of the predicted depth image, extract features from the second point cloud data to obtain the first feature, and fuse the depth features and the first feature to obtain the first target point cloud data.

11. The method according to claim 9 or 10, characterized in that, The second network includes a first set of network layers, a second set of network layers, a third set of network layers, and a splicing layer; The first set of network layers is used to extract and upsample the depth image to obtain the second feature; The second set of network layers is used to extract and upsample the first point cloud data to obtain the first feature; The splicing layer is used to splice the first feature and the second feature to obtain the third feature; The third network layer set is used to process the third feature to obtain the first target point cloud data.

12. The method according to claim 9, characterized in that, The network to be trained includes a first network and a second network; The step of inputting the first image and the first point cloud data into the network to be trained includes: The first image is input into the first network to obtain a feature map generated by the first network. The feature map includes depth information in the first image. The first network is used to predict a depth image based on a monocular image. The feature map and the first point cloud data are input into the second network to obtain the predicted point cloud data. The second network is used to predict the predicted point cloud data based on the feature map and the first point cloud data.

13. The method according to claim 9 or 10, characterized in that, The first image is an image captured by a monocular camera.

14. The method according to claim 9 or 10, characterized in that, The camera and the first lidar have been calibrated.

15. An automatic driving device, characterized in that, The method includes a processor coupled to a memory storing program instructions that, when executed by the processor, implement the method of any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 14.

17. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 14.

18. An autonomous vehicle, characterized in that, The autonomous vehicle includes a processing circuit and a storage circuit, the processing circuit and the storage circuit being configured to perform the method as described in any one of claims 1 to 14.

Images