Point cloud acquisition method and related device

Abstract

Disclosed in the present application are a point cloud acquisition method and a related device, which can reduce hardware costs required for point cloud acquisition and facilitate implementation in some cost-limited scenarios. The method in the present application comprises: when it is necessary to acquire a four-dimensional point cloud, first, acquiring a first pose of a camera and a second pose of the camera, wherein the first pose is temporally associated with the second pose; next, inputting the first pose into a first neural network model, and then processing the first pose by means of the first neural network model, so as to obtain a first image captured by the camera, and converting the first image, so as to obtain a first three-dimensional point cloud; then, further inputting the second pose into the first neural network model, and then processing the second pose by means of the first neural network model, so as to obtain a second image captured by the camera, and converting the second image, so as to obtain a second three-dimensional point cloud; and finally, fusing the first three-dimensional point cloud with the second three-dimensional point cloud, so as to obtain a first four-dimensional point cloud.

Description

A point cloud acquisition method and related device thereof

[0001] The present application claims priority to the Chinese Patent Application No. 202410465284.5, filed on April 17, 2024, and entitled "A point cloud acquisition method and related device thereof", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular, to a point cloud acquisition method and related device thereof. BACKGROUND

[0003] In the field of autonomous driving, a vehicle usually needs to utilize a general object detection (GOD) model to perceive and understand the environment around the vehicle in order to improve the safety of autonomous driving of the vehicle. Four-dimensional point cloud (which can also be referred to as 4D semantic occupancy (4D-SO)) as training data of the GOD model plays a crucial role in the performance of the GOD model.

[0004] In the related art, in order to acquire the four-dimensional point cloud, three-dimensional point clouds are acquired by using devices such as lidar, and then the four-dimensional point cloud that can be used to train the GOD model is obtained by using the three-dimensional point clouds. Specifically, the surrounding environment can be data-acquired at different times by using the lidar, so as to obtain three-dimensional point clouds at different times. Then, the three-dimensional point clouds at different times can be fused to obtain the four-dimensional point cloud. In this way, the GOD model can be trained by using the obtained four-dimensional point cloud.

[0005] Since the above-mentioned method of acquiring the four-dimensional point cloud depends on the lidar and needs the assistance of devices such as inertial measurement unit (IMU), the cost of the devices required by the method is too high, which is not conducive to implementation in some cost-limited model training scenarios. SUMMARY

[0006] Embodiments of the present application provide a point cloud acquisition method and related device thereof, which can reduce the hardware cost required for point cloud acquisition and are conducive to implementation in some cost-limited scenarios.

[0007] A first aspect of embodiments of the present application provides a point cloud acquisition method, which comprises:

[0008] When a four-dimensional point cloud needs to be obtained, a first pose of the camera at a first time and a second pose of the camera at a second time can be obtained. The first time and the second time are two different times, and thus the first pose and the second pose are two poses associated in time.

[0009] Then, the first pose can be input to the first neural network model to process the first pose by the first neural network model, so as to simulate a first image captured by the camera in the first pose. Then, the first image captured by the camera can be converted to obtain a first three-dimensional point cloud.

[0010] Then, the second pose can be input to the first neural network model to process the second pose by the first neural network model, so as to simulate a second image captured by the camera in the second pose. Then, the second image captured by the camera can be converted to obtain a second three-dimensional point cloud.

[0011] Finally, the first three-dimensional point cloud and the second three-dimensional point cloud can be fused to obtain a first four-dimensional point cloud. Then, the first four-dimensional point cloud can be used as data for subsequent model training.

[0012] As can be seen from the above method, when a four-dimensional point cloud needs to be obtained, a first pose of the camera and a second pose of the camera can be obtained, and the first pose and the second pose are associated in time. Then, the first pose can be input to the first neural network model to process the first pose by the first neural network model to obtain a first image captured by the camera, and the first image can be converted to obtain a first three-dimensional point cloud. Then, the second pose can also be input to the first neural network model to process the second pose by the first neural network model to obtain a second image captured by the camera, and the second image can be converted to obtain a second three-dimensional point cloud. Finally, the first three-dimensional point cloud and the second three-dimensional point cloud can be fused to obtain a first four-dimensional point cloud. In the foregoing process, the neural network model (i.e., the first neural network model described above, for example, a neural implicit field) can be used to process different poses (i.e., the first pose and the second pose described above) of the camera at different times to obtain three-dimensional point clouds corresponding to different poses (which can also be understood as three-dimensional point clouds at different times, i.e., the first three-dimensional point cloud and the second three-dimensional point cloud described above), and then the three-dimensional point clouds corresponding to different poses are used to obtain a four-dimensional point cloud (i.e., the first four-dimensional point cloud described above). This way of obtaining a four-dimensional point cloud does not need to rely on a laser radar, but only needs a camera, which can reduce the hardware cost required for obtaining a point cloud, and is conducive to implementing this way of obtaining a four-dimensional point cloud in some cost-limited scenarios.

[0013] In a possible implementation, the first image can include a first red / green / blue (RGB) image captured by the camera, a first semantic image corresponding to the first RGB image, and a first depth image corresponding to the first RGB image, where the first RGB image includes a plurality of pixels (i.e., the first RGB image includes colors of the plurality of pixels), the first semantic image includes a category of each pixel in the first RGB image, and the first depth image includes a depth of each pixel in the first RGB image.

[0014] In a possible implementation, the second image can include a second RGB image captured by the camera, a second semantic image corresponding to the second RGB image, and a second depth image corresponding to the second RGB image, where the second RGB image includes a plurality of pixels (i.e., the second RGB image includes colors of the plurality of pixels), the second semantic image includes a category of each pixel in the second RGB image, and the second depth image includes a depth of each pixel in the second RGB image.

[0015] In a possible implementation manner, the first RGB image includes a first pixel point, the first pixel point is any one of the pixels in the first RGB image, and the first pose is processed by the first neural network model to obtain the first image captured by the camera, including: obtaining, by the first neural network model, a first ray sent to the first pixel point by the camera in the first pose; sampling, by the first neural network model, the first ray to obtain a plurality of first sampling points; processing, by the first neural network model, the plurality of first sampling points to obtain colors of the plurality of first sampling points, categories of the plurality of first sampling points, depths of the plurality of first sampling points, and transparencies of the plurality of first sampling points; and obtaining, based on the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points, a color of the first pixel point, a category of the first pixel point, and a depth of the first pixel point. In the foregoing implementation manner, for the convenience of description, any one of the pixels in the first RGB image, that is, the first pixel point, is introduced illustratively. After the first pose of the camera is obtained, the first pose can be input to the first neural network model. Then, the first neural network model can construct the first ray sent to the first pixel point by the camera in the first pose. Then, the first neural network model can sample the first ray to obtain a plurality of first sampling points on the first ray. After the plurality of first sampling points are obtained, the first neural network model can perform a series of processing on the plurality of first sampling points to obtain the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points. Subsequently, the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points can be calculated to obtain the color of the first pixel point, the category of the first pixel point, and the depth of the first pixel point. Then, the same operation as performed on the first pixel point can be performed on the remaining pixels in the first RGB image except the first pixel point, and finally the colors of all the pixels in the first RGB image, the categories of all the pixels in the first RGB image, and the depths of all the pixels in the first RGB image can be obtained, which is equivalent to obtaining the first RGB image, the first semantic image, and the first depth image.

[0016] In a possible implementation manner, the second RGB image includes a second pixel point, the second pixel point being any one of the pixels in the second RGB image, and the processing of the second pose by the first neural network model to obtain the second image captured by the camera includes: obtaining, by the first neural network model, a second ray sent to the second pixel point by the camera in the second pose; sampling, by the first neural network model, the second ray to obtain a plurality of second sampling points; processing, by the first neural network model, the plurality of second sampling points to obtain colors of the plurality of second sampling points, categories of the plurality of second sampling points, depths of the plurality of second sampling points, and transparencies of the plurality of second sampling points; and obtaining, based on the colors of the plurality of second sampling points, the categories of the plurality of second sampling points, the depths of the plurality of second sampling points, and the transparencies of the plurality of second sampling points, a color of the second pixel point, a category of the second pixel point, and a depth of the second pixel point. In the foregoing implementation manner, for the convenience of description, any one of the pixels in the second RGB image, i.e., the second pixel point, is used for illustrative introduction. After obtaining the second pose of the camera, the second pose can be input to the second neural network model. Then, the second neural network model can construct a second ray sent to the second pixel point by the camera in the second pose. Then, the second neural network model can sample the second ray to obtain a plurality of second sampling points on the second ray. After obtaining the plurality of second sampling points, the second neural network model can perform a series of processing on the plurality of second sampling points to obtain colors of the plurality of second sampling points, categories of the plurality of second sampling points, depths of the plurality of second sampling points, and transparencies of the plurality of second sampling points. Subsequently, the colors of the plurality of second sampling points, the categories of the plurality of second sampling points, the depths of the plurality of second sampling points, and the transparencies of the plurality of second sampling points can be calculated to obtain a color of the second pixel point, a category of the second pixel point, and a depth of the second pixel point. Then, for the remaining pixels in the second RGB image except the second pixel point, the same operations performed on the second pixel point can also be performed on the remaining pixels, and therefore, the colors of all the pixels in the second RGB image, the categories of all the pixels in the second RGB image, and the depths of all the pixels in the second RGB image can be finally obtained, which is equivalent to obtaining the second RGB image, the second semantic image, and the second depth image.

[0017] In a possible implementation, the color, the category, and the depth of the first pixel point are obtained based on the colors, the categories, the depths, and the transparencies of the first sampling points, including: calculating the colors and the transparencies of the first sampling points to obtain the color of the first pixel point; calculating the categories and the transparencies of the first sampling points to obtain the category of the first pixel point; and calculating the depths and the transparencies of the first sampling points to obtain the depth of the first pixel point. In the foregoing implementation, after the colors, the categories, the depths, and the transparencies of the first sampling points are obtained, the colors, the transparencies, and the distances between the first sampling points can be calculated to obtain the color of the first pixel point. Meanwhile, the categories, the transparencies, and the distances between the first sampling points can be calculated to obtain the category of the first pixel point. Meanwhile, the depths, the transparencies, and the distances between the first sampling points can be calculated to obtain the depth of the first pixel point.

[0018] In a possible implementation, the color, the category, and the depth of the second pixel point are obtained based on the colors, the categories, the depths, and the transparencies of the second sampling points, including: calculating the colors and the transparencies of the second sampling points to obtain the color of the second pixel point; calculating the categories and the transparencies of the second sampling points to obtain the category of the second pixel point; and calculating the depths and the transparencies of the second sampling points to obtain the depth of the second pixel point. In the foregoing implementation, after the colors, the categories, the depths, and the transparencies of the second sampling points are obtained, the colors, the transparencies, and the distances between the second sampling points can be calculated to obtain the color of the second pixel point. Meanwhile, the categories, the transparencies, and the distances between the second sampling points can be calculated to obtain the category of the second pixel point. Meanwhile, the depths, the transparencies, and the distances between the second sampling points can be calculated to obtain the depth of the second pixel point.

[0019] In a possible implementation manner, the method further comprises: calculating the depths of the plurality of first sampling points and the depth of the first pixel point to obtain an evaluation value of the first pixel point; calculating the depths of the plurality of second sampling points and the depth of the second pixel point to obtain an evaluation value of the second pixel point; and fusing the first three-dimensional point cloud and the second three-dimensional point cloud to obtain the first four-dimensional point cloud, comprising: taking the evaluation value of each pixel point in the first RGB image as the evaluation value of each spatial point in the first three-dimensional point cloud, and removing, from the first three-dimensional point cloud, spatial points with an evaluation value less than an evaluation threshold to obtain a denoised first three-dimensional point cloud; taking the evaluation value of each pixel point in the second RGB image as the evaluation value of each spatial point in the second three-dimensional point cloud, and removing, from the second three-dimensional point cloud, spatial points with an evaluation value less than the evaluation threshold to obtain a denoised second three-dimensional point cloud; and fusing the denoised first three-dimensional point cloud and the denoised second three-dimensional point cloud to obtain the first four-dimensional point cloud. In the foregoing implementation manner, after obtaining the depth of the first pixel point, the depths of the plurality of first sampling points and the depth of the first pixel point are calculated to obtain the evaluation value of the first pixel point, and the evaluation value of the first pixel point is used to indicate the quality of the first pixel point. Similarly, for the remaining pixel points of the first RGB image except the first pixel point, the same operation as performed on the first pixel point can also be performed on the remaining pixel points, and therefore the evaluation values of the pixel points in the first RGB image can be finally obtained. Similarly, the evaluation values of the pixel points in the second RGB image can also be obtained. Then, after projecting the first RGB image, the first semantic image and the first depth image to obtain the first three-dimensional point cloud, since each pixel point in the first RGB image is in one-to-one correspondence with each spatial point in the first three-dimensional point cloud, the evaluation values of the pixel points in the first RGB image can be taken as the evaluation values of the spatial points in the first three-dimensional point cloud, and spatial points with an evaluation value lower than the evaluation threshold can be removed from the first three-dimensional point cloud to obtain the denoised first three-dimensional point cloud. Similarly, the denoised second three-dimensional point cloud can also be obtained. In this way, the denoised first three-dimensional point cloud and the denoised second three-dimensional point cloud can be fused to obtain the first four-dimensional point cloud. As can be seen, by obtaining the evaluation values of the pixel points in the RGB image and taking the evaluation values of the pixel points as the evaluation values of the spatial points in the three-dimensional point cloud, the noise points in the three-dimensional point cloud can be effectively removed, and therefore the four-dimensional point cloud obtained based on the denoised three-dimensional point cloud has better quality.

[0020] In a possible implementation, the method further includes: processing the first image and the second image by a second neural network model to obtain a second four-dimensional point cloud; and optimizing the first four-dimensional point cloud based on the second four-dimensional point cloud to obtain an optimized first four-dimensional point cloud. In the foregoing implementation, after the first image and the second image are obtained, the first image and the second image can be input into the second neural network model, so that the first image and the second image are processed by the second neural network model in a series of processes, thereby obtaining the second four-dimensional point cloud. After the second four-dimensional point cloud is obtained, the first four-dimensional point cloud and the second four-dimensional point cloud can be fused, thereby obtaining the optimized first four-dimensional point cloud. As can be seen, for the four-dimensional point cloud obtained based on the three-dimensional point cloud, the model-predicted four-dimensional point cloud can also be obtained, and the model-predicted four-dimensional point cloud can be used to optimize the four-dimensional point cloud obtained based on the three-dimensional point cloud, thereby obtaining a four-dimensional point cloud with better quality.

[0021] In a possible implementation, the first neural network model is a neural implicit field.

[0022] The second aspect of the embodiment of the present application provides a model training method, which includes: processing a pose of a camera by a to-be-trained model to obtain an image captured by the camera, the image being used to obtain a three-dimensional point cloud, and the three-dimensional point cloud being used to obtain a four-dimensional point cloud; obtaining a target loss based on the image and a real image corresponding to the image, the target loss being used to indicate a difference between the image and the real image; updating parameters of the to-be-trained model based on the target loss until a model training condition is met, and obtaining a first neural network model.

[0023] In a possible implementation, the image includes an RGB image captured by the camera, a semantic image corresponding to the RGB image, and a depth image corresponding to the RGB image, where the semantic image includes categories of each pixel point in the RGB image, and the depth image includes depths of each pixel point in the RGB image.

[0024] In a possible implementation, the RGB image includes a target pixel point, the target pixel point being any one pixel point in the RGB image, and processing the pose by the first neural network model to obtain the image captured by the camera includes: obtaining, by the to-be-trained model, a ray sent to the target pixel point by the camera in the pose; sampling, by the to-be-trained model, the ray to obtain a plurality of sampling points; processing, by the to-be-trained model, the plurality of sampling points to obtain colors of the plurality of sampling points, categories of the plurality of sampling points, depths of the plurality of sampling points, and transparencies of the plurality of sampling points; and obtaining, based on the colors of the plurality of sampling points, the categories of the plurality of sampling points, the depths of the plurality of sampling points, and the transparencies of the plurality of sampling points, a color of the target pixel point, a category of the target pixel point, and a depth of the target pixel point.

[0025] In a possible implementation, the color, the category, and the depth of the target pixel point are obtained based on the colors of the plurality of sampling points, the categories of the plurality of sampling points, the depths of the plurality of sampling points, and the transparencies of the plurality of sampling points, including: calculating the colors of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the color of the target pixel point; calculating the categories of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the category of the target pixel point; and calculating the depths of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the depth of the target pixel point.

[0026] In a possible implementation, the real image includes a real RGB image, a real semantic image, and a real depth image, the real RGB image includes a real color of the target pixel point, the real semantic image includes a real category of the target pixel point, and the real depth image includes a real depth of the target pixel point; and the target loss is obtained based on the image and the real image corresponding to the image, including: obtaining a first loss based on the color of the target pixel point and the real color of the target pixel point, the first loss being used to indicate a difference between the color of the target pixel point and the real color of the target pixel point; obtaining a second loss based on the category of the target pixel point and the real category of the target pixel point, the second loss being used to indicate a difference between the category of the target pixel point and the real category of the target pixel point; obtaining a third loss based on the depth of the target pixel point and the real depth of the target pixel point, the third loss being used to indicate a difference between the depth of the target pixel point and the real depth of the target pixel point; and obtaining the target loss based on the first loss, the second loss, and the third loss.

[0027] In a possible implementation, the target loss is obtained based on the image and the real image corresponding to the image, further including: obtaining a fourth loss based on the transparency distribution of the plurality of sampling points and a real transparency distribution of the plurality of sampling points; and the target loss is obtained based on the first loss, the second loss, and the third loss, including: obtaining the target loss based on the first loss, the second loss, the third loss, and the fourth loss.

[0028] In a possible implementation, the first neural network model is a neural implicit field.

[0029] The third aspect of the embodiment of the present application provides a point cloud acquisition device, which comprises: an acquisition module, configured to acquire a first pose of a camera and a second pose of the camera, the first pose being associated with the second pose in time; a first processing module, configured to process the first pose by using a first neural network model to obtain a first image captured by the camera, and convert the first image to obtain a first three-dimensional point cloud; a second processing module, configured to process the second pose by using the first neural network model to obtain a second image captured by the camera, and convert the second image to obtain a second three-dimensional point cloud; and a fusion module, configured to fuse the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first four-dimensional point cloud.

[0030] In a possible implementation manner, the first image comprises a first RGB image captured by the camera, a first semantic image corresponding to the first RGB image, and a first depth image corresponding to the first RGB image, wherein the first semantic image comprises a category of each pixel point in the first RGB image, and the first depth image comprises a depth of each pixel point in the first RGB image.

[0031] In a possible implementation manner, the second image comprises a second RGB image captured by the camera, a second semantic image corresponding to the second RGB image, and a second depth image corresponding to the second RGB image, wherein the second semantic image comprises a category of each pixel point in the second RGB image, and the second depth image comprises a depth of each pixel point in the second RGB image.

[0032] In a possible implementation manner, the first RGB image comprises a first pixel point, the first pixel point being any one of the pixel points in the first RGB image, and the first processing module is configured to: acquire, by using the first neural network model, a first ray sent to the first pixel point by the camera in the first pose; sample, by using the first neural network model, the first ray to obtain a plurality of first sampling points; process, by using the first neural network model, the plurality of first sampling points to obtain colors of the plurality of first sampling points, categories of the plurality of first sampling points, depths of the plurality of first sampling points, and transparencies of the plurality of first sampling points; and acquire, based on the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points, a color of the first pixel point, a category of the first pixel point, and a depth of the first pixel point.

[0033] In a possible implementation, the second RGB image includes a second pixel point, the second pixel point being any one of the pixel points in the second RGB image, and the second processing module is configured to: acquire, by using the first neural network model, a second ray sent to the second pixel point by the camera in the second pose; sample, by using the first neural network model, the second ray to obtain a plurality of second sampling points; process, by using the first neural network model, the plurality of second sampling points to obtain colors of the plurality of second sampling points, categories of the plurality of second sampling points, depths of the plurality of second sampling points, and transparencies of the plurality of second sampling points; and acquire, based on the colors of the plurality of second sampling points, the categories of the plurality of second sampling points, the depths of the plurality of second sampling points, and the transparencies of the plurality of second sampling points, a color of the second pixel point, a category of the second pixel point, and a depth of the second pixel point.

[0034] In a possible implementation, based on the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points, the first processing module is configured to: calculate the colors of the plurality of first sampling points and the transparencies of the plurality of first sampling points to obtain a color of the first pixel point; calculate the categories of the plurality of first sampling points and the transparencies of the plurality of first sampling points to obtain a category of the first pixel point; and calculate the depths of the plurality of first sampling points and the transparencies of the plurality of first sampling points to obtain a depth of the first pixel point.

[0035] In a possible implementation, based on the colors of the plurality of second sampling points, the categories of the plurality of second sampling points, the depths of the plurality of second sampling points, and the transparencies of the plurality of second sampling points, the second processing module is configured to: calculate the colors of the plurality of second sampling points and the transparencies of the plurality of second sampling points to obtain a color of the second pixel point; calculate the categories of the plurality of second sampling points and the transparencies of the plurality of second sampling points to obtain a category of the second pixel point; and calculate the depths of the plurality of second sampling points and the transparencies of the plurality of second sampling points to obtain a depth of the second pixel point.

[0036] In a possible implementation, the apparatus further includes: a first evaluation module, configured to calculate the depth of the plurality of first sampling points and the depth of the first pixel point to obtain an evaluation value of the first pixel point; a second evaluation module, configured to calculate the depth of the plurality of second sampling points and the depth of the second pixel point to obtain an evaluation value of the second pixel point; and a fusion module, configured to: take the evaluation value of each pixel point in the first RGB image as an evaluation value of each spatial point in the first three-dimensional point cloud, remove spatial points with an evaluation value less than an evaluation threshold from the first three-dimensional point cloud to obtain a first denoised three-dimensional point cloud; take the evaluation value of each pixel point in the second RGB image as an evaluation value of each spatial point in the second three-dimensional point cloud, remove spatial points with an evaluation value less than the evaluation threshold from the second three-dimensional point cloud to obtain a second denoised three-dimensional point cloud; and fuse the first denoised three-dimensional point cloud and the second denoised three-dimensional point cloud to obtain the first four-dimensional point cloud.

[0037] In a possible implementation, the apparatus further includes an optimization module, configured to: process the first image and the second image by using the second neural network model to obtain a second four-dimensional point cloud; and optimize the first four-dimensional point cloud based on the second four-dimensional point cloud to obtain an optimized first four-dimensional point cloud.

[0038] In a possible implementation, the first neural network model is a neural implicit field.

[0039] A fourth aspect of the embodiment of the present application provides a model training apparatus, which includes: a processing module, configured to process a pose of a camera by using a to-be-trained model to obtain an image captured by the camera, the image being used to obtain a three-dimensional point cloud, and the three-dimensional point cloud being used to obtain a four-dimensional point cloud; an obtaining module, configured to obtain a target loss based on the image and a real image corresponding to the image, the target loss being used to indicate a difference between the image and the real image; and a training module, configured to update a parameter of the to-be-trained model based on the target loss until a model training condition is met to obtain a first neural network model.

[0040] In a possible implementation, the image includes an RGB image captured by the camera, a semantic image corresponding to the RGB image, and a depth image corresponding to the RGB image, where the semantic image includes a category of each pixel point in the RGB image, and the depth image includes a depth of each pixel point in the RGB image.

[0041] In a possible implementation, the RGB image contains a target pixel point, the target pixel point being any one of the pixels in the RGB image, and the processing module is configured to: acquire, by using the to-be-trained model, a ray sent by the camera in the pose to the target pixel point; sample, by using the to-be-trained model, the ray to obtain a plurality of sampling points; process, by using the to-be-trained model, the plurality of sampling points to obtain colors of the plurality of sampling points, categories of the plurality of sampling points, depths of the plurality of sampling points, and transparencies of the plurality of sampling points; and acquire, based on the colors of the plurality of sampling points, the categories of the plurality of sampling points, the depths of the plurality of sampling points, and the transparencies of the plurality of sampling points, a color of the target pixel point, a category of the target pixel point, and a depth of the target pixel point.

[0042] In a possible implementation, the processing module is configured to: calculate the colors of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the color of the target pixel point; calculate the categories of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the category of the target pixel point; and calculate the depths of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the depth of the target pixel point.

[0043] In a possible implementation, the real image contains a real RGB image, a real semantic image, and a real depth image, the real RGB image containing a real color of the target pixel point, the real semantic image containing a real category of the target pixel point, and the real depth image containing a real depth of the target pixel point; and the acquisition module is configured to: acquire, based on the color of the target pixel point and the real color of the target pixel point, a first loss, the first loss being used to indicate a difference between the color of the target pixel point and the real color of the target pixel point; acquire, based on the category of the target pixel point and the real category of the target pixel point, a second loss, the second loss being used to indicate a difference between the category of the target pixel point and the real category of the target pixel point; acquire, based on the depth of the target pixel point and the real depth of the target pixel point, a third loss, the third loss being used to indicate a difference between the depth of the target pixel point and the real depth of the target pixel point; and acquire, based on the first loss, the second loss, and the third loss, a target loss.

[0044] In a possible implementation, the acquisition module is further configured to acquire, based on a transparency distribution of the plurality of sampling points and a real transparency distribution of the plurality of sampling points, a fourth loss; and the acquisition module is configured to acquire, based on the first loss, the second loss, the third loss, and the fourth loss, the target loss.

[0045] In a possible implementation, the first neural network model is a neural implicit field.

[0046] In a fifth aspect, an embodiment of the present application provides a point cloud acquisition apparatus, comprising a memory and a processor; the memory stores codes, and the processor is configured to execute the codes, when the codes are executed, the point cloud acquisition apparatus performs the method in the first aspect or any possible implementation manner of the first aspect.

[0047] In a sixth aspect, an embodiment of the present application provides a model training apparatus, comprising a memory and a processor; the memory stores codes, and the processor is configured to execute the codes, when the codes are executed, the model training apparatus performs the method in the second aspect or any possible implementation manner of the second aspect.

[0048] In a seventh aspect, an embodiment of the present application provides circuitry, comprising a processing circuit configured to perform the method in the first aspect, any possible implementation manner of the first aspect, the second aspect or any possible implementation manner of the second aspect.

[0049] In an eighth aspect, an embodiment of the present application provides a chip system, comprising a processor, configured to invoke a computer program or computer instruction stored in a memory, so that the processor performs the method in the first aspect, any possible implementation manner of the first aspect, the second aspect or any possible implementation manner of the second aspect.

[0050] In a possible implementation manner, the processor is coupled with the memory through an interface.

[0051] In a possible implementation manner, the chip system further comprises the memory, and the memory stores the computer program or computer instruction.

[0052] In a ninth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program, and the program, when executed by a computer, causes the computer to implement the method in the first aspect, any possible implementation manner of the first aspect, the second aspect or any possible implementation manner of the second aspect.

[0053] In a tenth aspect, an embodiment of the present application provides a computer program product, which stores instructions, and the instructions, when executed by a computer, causes the computer to implement the method in the first aspect, any possible implementation manner of the first aspect, the second aspect or any possible implementation manner of the second aspect.

[0054] In the embodiments of the present application, when a four-dimensional point cloud needs to be acquired, a first pose of the camera and a second pose of the camera can be acquired first, and the first pose and the second pose are associated in time. Then, the first pose can be input into the first neural network model to process the first pose through the first neural network model, to obtain a first image captured by the camera, and to convert the first image to obtain a first three-dimensional point cloud. Then, the second pose can also be input into the first neural network model to process the second pose through the first neural network model, to obtain a second image captured by the camera, and to convert the second image to obtain a second three-dimensional point cloud. Finally, the first three-dimensional point cloud and the second three-dimensional point cloud can be fused to obtain a first four-dimensional point cloud. In the foregoing process, the neural network model (i.e., the first neural network model described above, for example, a neural implicit field) can be used to process different poses (i.e., the first pose and the second pose described above) of the camera at different times to obtain three-dimensional point clouds corresponding to different poses (which can also be understood as three-dimensional point clouds at different times, i.e., the first three-dimensional point cloud and the second three-dimensional point cloud described above), and then the three-dimensional point clouds corresponding to different poses are used to acquire a four-dimensional point cloud (i.e., the first four-dimensional point cloud described above). This way of acquiring a four-dimensional point cloud does not need to rely on a laser radar, but only needs a camera, which can reduce the hardware cost required for acquiring a point cloud, and is conducive to the implementation of this way of acquiring a four-dimensional point cloud in some cost-limited scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0055] FIG. 1 is a structural schematic diagram of an artificial intelligence main body framework;

[0056] FIG. 2a is a structural schematic diagram of a fault prediction system provided by an embodiment of the present application;

[0057] FIG. 2b is another structural schematic diagram of a fault prediction system provided by an embodiment of the present application;

[0058] FIG. 2c is a schematic diagram of a device related to fault prediction provided by an embodiment of the present application;

[0059] FIG. 3 is a schematic diagram of a system 100 architecture provided by an embodiment of the present application;

[0060] FIG. 4 is a flow schematic diagram of a point cloud acquisition method provided by an embodiment of the present application;

[0061] FIG. 5 is a schematic diagram of image acquisition provided by an embodiment of the present application;

[0062] FIG. 6 is a schematic diagram of point cloud acquisition provided by an embodiment of the present application;

[0063] FIG. 7 is a schematic diagram of point cloud optimization provided by an embodiment of the present application;

[0064] FIG. 8 is a schematic diagram of an image in the field of autonomous driving according to an embodiment of the present application;

[0065] FIG. 9 is a schematic diagram of a three-dimensional point cloud according to an embodiment of the present application;

[0066] FIG. 10 is a schematic diagram of a four-dimensional point cloud according to an embodiment of the present application;

[0067] FIG. 11 is a schematic diagram of a model training method according to an embodiment of the present application;

[0068] FIG. 12 is a schematic diagram of a point cloud acquisition device according to an embodiment of the present application;

[0069] FIG. 13 is a schematic diagram of a model training device according to an embodiment of the present application;

[0070] FIG. 14 is a schematic diagram of an execution device according to an embodiment of the present application;

[0071] FIG. 15 is a schematic diagram of a training device according to an embodiment of the present application;

[0072] FIG. 16 is a schematic diagram of a chip according to an embodiment of the present application. DETAILED DESCRIPTION

[0073] The point cloud acquisition method and related devices provided in the embodiments of the present application can reduce the hardware cost required for point cloud acquisition, and are beneficial to implementation in some cost-limited scenarios.

[0074] The terms "first", "second", etc. in the specification and claims of the present application and in the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the terms thus used can be interchanged under appropriate circumstances, and are merely a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attributes. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device containing a series of units does not have to be limited to those units, but can include other units not clearly listed or inherent to the process, method, product or device.

[0075] In the field of autonomous driving, vehicles usually need to use a GOD model to perceive and understand the environment around the vehicle, so as to improve the safety of vehicle autonomous driving. Four-dimensional point cloud as training data of the GOD model plays a crucial role in the performance of the GOD model.

[0076] In the related art, in order to collect a four-dimensional point cloud, a three-dimensional point cloud is collected by using a device such as a laser radar, and the four-dimensional point cloud that can be used to train a GOD model is obtained by using the three-dimensional point cloud. Specifically, the three-dimensional point cloud at different times can be obtained by using the laser radar to collect data on the surrounding environment at different times. Then, the three-dimensional point cloud at different times can be fused to obtain the four-dimensional point cloud. In this way, the GOD model can be trained by using the obtained four-dimensional point cloud.

[0077] Since the above-mentioned method of obtaining a four-dimensional point cloud depends on a laser radar and also needs the assistance of devices such as an IMU, the cost of the devices required by the method is too high, which is not conducive to the implementation in some cost-limited model training scenarios.

[0078] Further, the laser radar will fail when dealing with extreme weather or low-reflectivity objects, and it cannot successfully collect a three-dimensional point cloud, and thus cannot successfully obtain a four-dimensional point cloud. It can be seen that the method is also difficult to implement in some scenarios with special conditions.

[0079] In order to solve the above-mentioned problems, an embodiment of the present application provides a scene perception method, which can be implemented in combination with artificial intelligence (AI) technology. AI technology is a technology discipline that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence. AI technology obtains the best results by perceiving the environment, acquiring knowledge and using knowledge. In other words, artificial intelligence technology is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Data processing using artificial intelligence is a common application of artificial intelligence.

[0080] First, the overall workflow of the artificial intelligence system is described. Please refer to FIG. 1, which is a structural schematic diagram of an artificial intelligence main body framework. The above-mentioned artificial intelligence theme framework is described below from two dimensions of “intelligent information chain” (horizontal axis) and “IT value chain” (vertical axis). Among them, the “intelligent information chain” reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of “data-information-knowledge-wisdom”. The “IT value chain” reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0081] (1) Infrastructure

[0082] Infrastructure provides computing power support for artificial intelligence systems, realizes communication with the external world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0083] (2) Data

[0084] The data of the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data relates to graphics, images, voice, text, and also relates to the Internet of Things data of traditional devices, including the business data of existing systems and the sensing data of force, displacement, liquid level, temperature, humidity, etc.

[0085] (3) Data processing

[0086] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision, etc.

[0087] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. Symbolic and formalized intelligent information.

[0088] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0089] Decision-making refers to the process of decision-making after intelligent information is reasoned, which usually provides functions such as classification, sorting, prediction, etc.

[0090] (4) General capabilities

[0091] After the data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0092] (5) Intelligent products and industry applications

[0093] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are the packaging of the overall solution of artificial intelligence, and realize the application of intelligent information decision-making products. Its application fields mainly include: intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0094] Next, several application scenarios of the present application are introduced.

[0095] FIG. 2a is a structural schematic diagram of a point cloud acquisition system provided by an embodiment of the present application, which includes a user device and a data processing device. The user device includes a smart terminal such as a mobile phone used by a user, an on-board computer in a vehicle driven by the user, etc. The user device is the initiator of point cloud acquisition, and serves as the initiator of a point cloud acquisition request, which is usually initiated by the user through the user device.

[0096] The data processing device can be a cloud server, a network server, an application server, a management server, etc. with a data processing function. The data processing device receives a point cloud acquisition request from the smart terminal through an interactive interface, and then performs point cloud acquisition in the manner of machine learning, deep learning, searching, reasoning, decision making, etc. through a storage memory and a processor for data processing. The storage memory in the data processing device can be a general term, including a local storage and a database for storing historical data, which can be on the data processing device or on other network servers.

[0097] In the point cloud acquisition system shown in FIG. 2a, the user device can acquire the poses of the camera of the user device at different times, and then initiate a request to the data processing device, so that the data processing device performs point cloud acquisition processing on the poses of the camera of the user device at different times, thereby obtaining a four-dimensional point cloud. For example, when the user triggers the user device, the user device can acquire the poses of the camera of the user device at different times, and then the user device can initiate a point cloud acquisition request to the data processing device, so that the data processing device performs a series of processing on the poses of the camera of the user device at different times based on the point cloud acquisition request, thereby obtaining a four-dimensional point cloud, i.e. 4D semantic occupancy data, which can be used for subsequent neural network model (e.g. GOD model, etc.) training.

[0098] In FIG. 2a, the data processing device can perform the point cloud acquisition method of an embodiment of the present application.

[0099] FIG. 2b is another structural schematic diagram of a point cloud acquisition system provided by an embodiment of the present application. In FIG. 2b, the user device directly serves as the data processing device, which can directly acquire input from the user and directly process by the hardware of the user device itself. The specific process is similar to that of FIG. 2a, and reference can be made to the above description, which will not be repeated here.

[0100] In the point cloud acquisition system shown in FIG. 2b, when the user triggers the user device, the user device can acquire the poses of the camera of the user device at different time points, and then the user device can perform a series of processing on the poses of the camera of the user device at different time points, so as to obtain four-dimensional point cloud, that is, 4D semantic occupancy data, which can be used for subsequent neural network model (for example, GOD model and the like) training.

[0101] In FIG. 2b, the user device itself can perform the point cloud acquisition method of the embodiments of the present application.

[0102] FIG. 2c is a schematic diagram of a related device for point cloud acquisition provided by the embodiments of the present application.

[0103] The user device in FIG. 2a and FIG. 2b can be the local device 301 or the local device 302 in FIG. 2c, and the data processing device in FIG. 2a can be the execution device 210 in FIG. 2c, wherein the data storage system 250 can store the to-be-processed data of the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or can be arranged on a cloud or other network server.

[0104] The processor in FIG. 2a and FIG. 2b can perform data training / machine learning / deep learning through a neural network model or other model (for example, a model based on a support vector machine), and finally train or learn a model through data, and perform point cloud acquisition application on the image through the model, so as to obtain a corresponding processing result.

[0105] FIG. 3 is a schematic diagram of a system 100 architecture provided by the embodiments of the present application. In FIG. 3, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140, and the input data can include various to-be-scheduled tasks, callable resources and other parameters in the embodiments of the present application.

[0106] During the pre-processing of the input data by the execution device 110, or during the processing (such as the implementation of the functions of the neural network in the present application) performed by the computing module 111 of the execution device 110, the execution device 110 can call the data, code and the like in the data storage system 150 for corresponding processing, or store the data, instructions and the like obtained through corresponding processing into the data storage system 150.

[0107] Finally, the I / O interface 112 returns the processing result to the client device 140, thereby providing the user.

[0108] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different targets or different tasks, and the corresponding target models / rules can be used to achieve the above targets or complete the above tasks, thereby providing the required results for the user. The training data can be stored in the database 130 and obtained from the training samples collected by the data collection device 160.

[0109] In the case shown in FIG. 3, the user can manually give input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112, and if the user's authorization is required to enable the client device 140 to automatically send input data, the user can set the corresponding permission in the client device 140. The user can view the results output by the execution device 110 in the client device 140, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 140 can also act as a data collection end to collect the input data input into the I / O interface 112 and the output results output from the I / O interface 112 as new sample data and store them into the database 130. Of course, the input data input into the I / O interface 112 and the output results output from the I / O interface 112 can also be directly stored into the database 130 as new sample data by the I / O interface 112 without being collected through the client device 140.

[0110] It is worth noting that FIG. 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 3, the data storage system 150 is an external memory relative to the execution device 110, and in other cases, the data storage system 150 can also be placed in the execution device 110. As shown in FIG. 3, the neural network can be trained by the training device 120.

[0111] The chip provided by an embodiment of the present application includes a neural network processor NPU. The chip can be arranged in the execution device 110 shown in FIG. 3 to complete the calculation work of the calculation module 111. The chip can also be arranged in the training device 120 shown in FIG. 3 to complete the training work of the training device 120 and output the target model / rules.

[0112] The neural network processor NPU is mounted on a host central processing unit (CPU) (host CPU) as a co-processor and is assigned tasks by the host CPU. The core part of the NPU is an operation circuit, and the controller controls the operation circuit to extract data in the memory (weight memory or input memory) and perform operations.

[0113] In some implementations, the arithmetic circuit includes a plurality of processing engines (PEs) inside. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

[0114] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory and buffers them on each PE of the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory and performs matrix operation with matrix B, and the partial result or final result of the matrix is saved in an accumulator.

[0115] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computation unit can be used for network computation of non-convolution / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0116] In some implementations, the vector computation unit can store the processed output vector to the unified buffer. For example, the vector computation unit can apply a non-linear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit, such as for use in a subsequent layer in a neural network.

[0117] The unified memory is used to store input data and output data.

[0118] The weight data is transferred from the external memory to the input memory and / or the unified memory, from the external memory to the weight memory, and from the unified memory to the external memory by a direct memory access controller (DMAC).

[0119] The bus interface unit (BIU) is used to interact between the main CPU, the DMAC, and the instruction memory through a bus.

[0120] An instruction fetch buffer connected to the controller, configured to store instructions used by the controller;

[0121] A controller configured to invoke the instructions cached in the instruction fetch buffer to control the operation of the operation accelerator.

[0122] Generally, the unified memory, the input memory, the weight memory and the instruction fetch buffer are on-chip memories, and the external memory is a memory external to the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0123] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0124] (1) Neural network

[0125] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs and intercept 1 as inputs. The output of the operation unit can be:

[0126] wherein s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is an activation function of the neural unit, which is used to introduce a nonlinear characteristic into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0127] The work of each layer in the neural network can be described by the mathematical expression y = a(Wx + b): from a physical perspective, the work of each layer in the neural network can be understood as completing the transformation of the input space (a set of input vectors) to the output space (i.e., the row space of the matrix to the column space) through five operations on the input space, which include: 1, dimensionality increase / decrease; 2, magnification / reduction; 3, rotation; 4, translation; 5, "bending". Among them, the operations of 1, 2, and 3 are completed by Wx, the operation of 4 is completed by +b, and the operation of 5 is completed by a(). The reason why "space" is used here is that the objects to be classified are not single things, but a class of things, and the space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network. The vector W determines the spatial transformation of the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training the neural network is to obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by the vectors W of many layers). Therefore, the training process of the neural network is essentially learning the way to control the spatial transformation, more specifically, learning the weight matrix.

[0128] Because the output of the neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network with the target value that is truly intended to be predicted, and then adjusting the weight vector according to the difference between the two, for example, if the predicted value of the network is too high, the weight vector is adjusted to make it predict lower, and the adjustment is continuously made until the neural network can predict the target value that is truly intended to be predicted. Therefore, it is necessary to define "how to compare the difference between the predicted value and the target value" in advance, which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the neural network becomes a process of trying to minimize this loss.

[0129] (2) Backpropagation algorithm

[0130] The neural network can adopt a back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal will generate an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal neural network model parameters, such as a weight matrix.

[0131] (3) Bird eye view

[0132] The bird eye view (BEV) graph is a natural and direct candidate view and can be used as a unified representation. Compared with the front view or perspective view widely studied in the field of two-dimensional vision, the BEV representation has some inherent advantages. First, the BEV does not have the occlusion and scale problems that exist universally in two-dimensional tasks. In the field of autonomous driving, the problem of identifying vehicles with occlusion or crossing traffic can be better solved by using the BEV. In addition, representing objects or road elements in the form of BEV will be beneficial to the development and deployment of subsequent modules (such as planning and control).

[0133] The method provided in the present application is described below from the training side of the neural network and the application side of the neural network.

[0134] The model training method provided in the embodiments of the present application involves processing of data sequences and can be applied to data training, machine learning, deep learning and the like. Symbolic and formal intelligent information modeling, extraction, preprocessing, training and the like are performed on training data (for example, the pose of the camera in the model training method provided in the embodiments of the present application), and finally a trained neural network (for example, the first neural network model in the model training method provided in the embodiments of the present application) is obtained. In addition, the point cloud acquisition method provided in the embodiments of the present application can use the trained neural network, input the input data (for example, the first pose of the camera and the second pose of the camera in the point cloud acquisition method provided in the embodiments of the present application) into the trained neural network, and obtain the output data (for example, the first four-dimensional point cloud in the embodiments of the present application). It should be noted that the model training method and the point cloud acquisition method provided in the embodiments of the present application are based on the same concept and can be understood as two parts of a system or two stages of an overall process, such as a model training stage and a model application stage.

[0135] The four-dimensional point cloud data obtained by the point cloud acquisition method provided in the embodiments of the present application can be applied not only to model training required by object detection and object segmentation in the field of automatic driving, but also to model training required by more scenarios such as robot route planning in the field of logistics transportation, which is not limited herein. In order to understand the process of the embodiments of the present application, the process is introduced below in combination with FIG. 4, which is a flowchart of a point cloud acquisition method provided in the embodiments of the present application, as shown in FIG. 4, the method comprises:

[0136] 401, obtaining a first pose of a camera and a second pose of the camera, the first pose and the second pose being associated in time.

[0137] In the embodiments of the present application, when it is necessary to obtain four-dimensional point cloud, the first pose of the camera at the first time and the second pose of the camera at the second time can be obtained first. Since the first time and the second time can be regarded as two different times (for example, the first time and the second time are adjacent two times), the first pose and the second pose are two poses associated in time.

[0138] 402, processing the first pose by a first neural network model to obtain a first image shot by the camera, and converting the first image to obtain a first three-dimensional point cloud.

[0139] After obtaining the first pose, the first pose can be input to the first neural network model to perform a series of processing on the first pose by the first neural network model, so as to simulate the first image shot by the camera in the first pose. Then, the first image shot by the camera can be converted (for example, three-dimensional projection and the like), so as to obtain the first three-dimensional point cloud.

[0140] Specifically, the first neural network model can be a neural implicit field (that is, a fully connected neural network model), or can be other neural network models, which are not limited herein.

[0141] More specifically, the first image can include a first red / green / blue (RGB) image shot by the camera, a first semantic image corresponding to the first RGB image, a first depth image corresponding to the first RGB image, and a first mask corresponding to the first RGB image. The first RGB image can include a plurality of pixel points, the first semantic image can include the categories of the pixel points in the first RGB image, the first depth image can include the depths of the pixel points in the first RGB image, and the first mask can include evaluation values of the pixel points in the first RGB image.

[0142] More specifically, the first image can be obtained by the following way:

[0143] Since the first RGB image contains a plurality of pixel points, the operations performed by the embodiment on each pixel point are similar. For the convenience of description, the following will be described by taking any one pixel point contained in the first RGB image as an example, and the pixel point will be referred to as a first pixel point. Then, the information acquisition process of the first pixel point includes:

[0144] (1) After obtaining the first pose of the camera, the first pose can be input to the first neural network model. Then, the first neural network model can construct a first ray sent from the camera in the first pose to the first pixel point, wherein the first ray can have the camera in the first pose as the starting point and the first pixel point as the ending point, or the first ray can also have the camera in the first pose as the starting point and pass through the first pixel point, which is not limited here.

[0145] For example, as shown in FIG. 5 (FIG. 5 is a schematic diagram of image acquisition provided by an embodiment of the present application), after determining the pose of the camera at a certain time, the pose can be input to the neural implicit field, and the neural implicit field can determine a certain pixel point (u, v) of the RGB image (simulated) photographed by the camera in the pose, and construct a ray emitted by the camera in the pose to (u, v). Wherein u is the horizontal coordinate of (u, v) in the RGB image, and v is the vertical coordinate of (u, v) in the RGB image.

[0146] (2) After obtaining the first ray, the first neural network model can sample the first ray, thereby obtaining a plurality of first sampling points on the first ray.

[0147] Still as the above example, after constructing the ray emitted by the camera in the pose to (u, v), the neural implicit field can sample the ray, and N sampling points can be obtained

[0148] (3) After obtaining a plurality of first sampling points, the first neural network model can perform a series of processing on the plurality of first sampling points, thereby respectively obtaining the color of each first sampling point in the plurality of first sampling points, the category of each first sampling point in the plurality of first sampling points, the depth of each first sampling point in the plurality of first sampling points, and the transparency of each first sampling point in the plurality of first sampling points (which can also be referred to as the probability density of each first sampling point in the plurality of first sampling points).

[0149] Still as the above example, among the N sampling points, for the i-th sampling point p i , the neural implicit field can process p i , thereby outputting the color c(p i ) of p i , the category s(p i ) of p i , the depth d(p i ) of p i , and the transparency t(p i ) of p i .depth d(p i ) and probability density σ(p i ) of p i ). For the rest of the N sampling points, the neural implicit field can also output the color of the rest of the sampling points, the class of the rest of the sampling points, the depth of the rest of the sampling points, and the probability density of the rest of the sampling points, which are not described here. In this way, the color of each of the N sampling points, the class of each of the N sampling points, the depth of each of the N sampling points, and the probability density of each of the N sampling points can be obtained.

[0150] (4) After obtaining the color of each of the plurality of first sampling points, the class of each of the plurality of first sampling points, the depth of each of the plurality of first sampling points, and the transparency of each of the plurality of first sampling points, the color of the first pixel point, the class of the first pixel point, and the depth of the first pixel point can be obtained based on the color of each of the plurality of first sampling points, the class of each of the plurality of first sampling points, the depth of each of the plurality of first sampling points, and the transparency of each of the plurality of first sampling points.

[0151] Then, for the rest of the pixel points in the first RGB image other than the first pixel point, the same operation as performed on the first pixel point can be performed on the rest of the pixel points, so that the color of each of the pixel points in the first RGB image, the class of each of the pixel points in the first RGB image, and the depth of each of the pixel points in the first RGB image can be finally obtained, which is equivalent to successfully obtaining the first RGB image, the first semantic image, and the first depth image.

[0152] Still as in the above example, after obtaining the color of the N sampling points, the class of the N sampling points, the depth of the N sampling points, and the probability density of the N sampling points, the color r(u, v) of (u, v), the class s(u, v) of (u, v), and the depth d(u, v) of (u, v) can be obtained by calculation.

[0153] Then, for the rest of the pixel points in the RGB image taken by the camera in the pose other than (u, v), the same operation as performed on (u, v) can be performed on the rest of the pixel points, so that the color, the class, and the depth of each of the pixel points in the RGB image taken by the camera in the pose can be obtained, which is equivalent to obtaining the RGB image taken by the camera in the pose, the semantic image corresponding to the RGB image, and the depth image corresponding to the RGB image.

[0154] More specifically, the color of the first pixel point, the class of the first pixel point, the depth of the first pixel point, and the transparency of the first pixel point can also be obtained by the following way:

[0155] (4.1) After obtaining the color of each first sampling point in the plurality of first sampling points, the category of each first sampling point in the plurality of first sampling points, the depth of each first sampling point in the plurality of first sampling points, and the transparency of each first sampling point in the plurality of first sampling points, the color of each first sampling point in the plurality of first sampling points and the transparency of each first sampling point in the plurality of first sampling points may be calculated to obtain the color of the first pixel.

[0156] Still as in the above example, the color, category, depth and probability density of each sampling point in these N sampling points can be obtained. The following formula can be used to calculate r(u,v) of (u,v):

[0157] In the above formula, c(p i ) is the i-th sampling point p i Color, s(p i ) is p i Category, d(p i ) is p i The depth, σ(p i ) is p i The probability density of i =d(p i+1 )-d(p i ), is the distance between the i+1th sampling point and the i-th sampling point.

[0158] (4.2) After obtaining the color of each of the plurality of first sampling points, the category of each of the plurality of first sampling points, the depth of each of the plurality of first sampling points, and the transparency of each of the plurality of first sampling points, the category of each of the plurality of first sampling points and the transparency of each of the plurality of first sampling points may be further calculated to obtain the category of the first pixel point.

[0159] Still as in the above example, the color, category, depth and probability density of each sampling point in these N sampling points can be obtained. The following formula can be used to calculate s(u,v) of (u,v):

[0160] (4.3) After obtaining the color of each of the plurality of first sampling points, the category of each of the plurality of first sampling points, the depth of each of the plurality of first sampling points, and the transparency of each of the plurality of first sampling points, the depth of each of the plurality of first sampling points and the transparency of each of the plurality of first sampling points may be further calculated to obtain the depth of the first pixel.

[0161] Still as the above example, after obtaining the color of each sampling point in the N sampling points, the category of each sampling point, the depth of each sampling point, and the probability density of each sampling point, the d(u, v) of (u, v) can be calculated by the following formula:

[0162] More specifically, the evaluation value of the first pixel point can also be obtained by the following way:

[0163] After obtaining the color of each first sampling point in the plurality of first sampling points, the category of each first sampling point in the plurality of first sampling points, the depth of each first sampling point in the plurality of first sampling points, the transparency of each first sampling point in the plurality of first sampling points, and the depth of the first pixel point, the depth of each first sampling point in the plurality of first sampling points and the depth of the first pixel point can be calculated, so as to obtain the evaluation value of the first pixel point (which can also be referred to as the uncertainty of the first pixel point), which is used to indicate the quality of the first pixel point. The higher the evaluation value is, the lower the quality of the first pixel point is. The lower the evaluation value is, the higher the quality of the first pixel point is.

[0164] Then, for the remaining pixel points in the first RGB image except the first pixel point, the operations performed on the first pixel point can also be performed on the remaining pixel points, so that the evaluation value of each pixel point in the first RGB image can be finally obtained, that is, the first mask is obtained.

[0165] Still as the above example, after obtaining the color of each sampling point in the N sampling points, the category of each sampling point, the depth of each sampling point, and the probability density of each sampling point, the evaluation value u(u, v) of (u, v) can be calculated by the following formula:

[0166] In the above formula, θ is a preset parameter. Then, for the remaining pixel points in the RGB image captured by the camera in the pose except (u, v), the operations performed on (u, v) can also be performed on the remaining pixel points, so that the evaluation value of each pixel point in the RGB image captured by the camera in the pose can be obtained, that is, the mask corresponding to the RGB image captured by the camera in the pose is obtained.

[0167] 403、processing the second pose by the first neural network model to obtain a second image captured by the camera, and converting the second image to obtain a second three-dimensional point cloud.

[0168] After obtaining the second pose, the second pose can be input into the first neural network model to perform a series of processing on the second pose through the first neural network model, so as to simulate a second image captured by the camera in the second pose. Then, the second image captured by the camera can be converted to obtain a second three-dimensional point cloud.

[0169] Specifically, the second image can include a second RGB image captured by the camera, a second semantic image corresponding to the second RGB image, a second depth image corresponding to the second RGB image, and a second mask corresponding to the second RGB image. The second RGB image can include a plurality of pixel points, the second semantic image can include the category of each pixel point in the second RGB image, the second depth image can include the depth of each pixel point in the second RGB image, and the second mask can include the evaluation value of each pixel point in the second RGB image.

[0170] More specifically, the second image can be obtained in the following manner:

[0171] Since the second RGB image includes a plurality of pixel points, the operations performed by the embodiment on each pixel point are similar. For the convenience of description, the following will be introduced in an illustrative manner with respect to any one of the pixel points included in the second RGB image, which will be referred to as a second pixel point. Then, the information acquisition process of the second pixel point includes:

[0172] (1) After obtaining the second pose of the camera, the second pose can be input into the first neural network model. Then, the first neural network model can construct a second ray sent from the camera in the second pose to the second pixel point, wherein the second ray can have the camera in the second pose as the starting point and the second pixel point as the ending point, or the second ray can have the camera in the second pose as the starting point and pass through the second pixel point, which is not limited here.

[0173] (2) After obtaining the second ray, the second neural network model can sample the second ray to obtain a plurality of second sampling points on the second ray.

[0174] (3) After obtaining the plurality of second sampling points, the second neural network model can perform a series of processing on the plurality of second sampling points to obtain the color of each second sampling point in the plurality of second sampling points, the category of each second sampling point in the plurality of second sampling points, the depth of each second sampling point in the plurality of second sampling points, and the transparency of each second sampling point in the plurality of second sampling points (which can also be referred to as the probability density of each second sampling point in the plurality of second sampling points).

[0175] (4) After obtaining the color of each of the plurality of second sampling points, the category of each of the plurality of second sampling points, the depth of each of the plurality of second sampling points, and the transparency of each of the plurality of second sampling points, the color of the second pixel point, the category of the second pixel point, and the depth of the second pixel point can be obtained based on the color of each of the plurality of second sampling points, the category of each of the plurality of second sampling points, the depth of each of the plurality of second sampling points, and the transparency of each of the plurality of second sampling points.

[0176] Then, for the remaining pixel points in the second RGB image except for the second pixel point, the same operation as performed on the second pixel point can also be performed on the remaining pixel points, and thus the color of each of the pixel points in the second RGB image, the category of each of the pixel points in the second RGB image, and the depth of each of the pixel points in the second RGB image can be finally obtained, which is equivalent to successfully obtaining the second RGB image, the second semantic image, and the second depth image.

[0177] More specifically, the color of the second pixel point, the category of the second pixel point, the depth of the second pixel point, and the transparency of the second pixel point can also be obtained in the following manner:

[0178] (4.1) After obtaining the color of each of the plurality of second sampling points, the category of each of the plurality of second sampling points, the depth of each of the plurality of second sampling points, and the transparency of each of the plurality of second sampling points, the color of the second pixel point can be obtained by calculating the color of each of the plurality of second sampling points and the transparency of each of the plurality of second sampling points.

[0179] (4.2) After obtaining the color of each of the plurality of second sampling points, the category of each of the plurality of second sampling points, the depth of each of the plurality of second sampling points, and the transparency of each of the plurality of second sampling points, the category of the second pixel point can be obtained by calculating the category of each of the plurality of second sampling points and the transparency of each of the plurality of second sampling points.

[0180] (4.3) After obtaining the color of each of the plurality of second sampling points, the category of each of the plurality of second sampling points, the depth of each of the plurality of second sampling points, and the transparency of each of the plurality of second sampling points, the depth of the second pixel point can be obtained by calculating the depth of each of the plurality of second sampling points and the transparency of each of the plurality of second sampling points.

[0181] More specifically, the evaluation value of the second pixel point can also be obtained by the following manner:

[0182] After obtaining the color of each second sampling point, the category of each second sampling point, the depth of each second sampling point, the transparency of each second sampling point, and the depth of the second pixel point, the depth of each second sampling point and the depth of the second pixel point can also be calculated, so as to obtain the evaluation value (which can also be referred to as the uncertainty) of the second pixel point, which is used to indicate the quality of the second pixel point. The higher the evaluation value is, the lower the quality of the second pixel point is. The lower the evaluation value is, the higher the quality of the second pixel point is.

[0183] Then, for the remaining pixel points in the second RGB except the second pixel point, the operations performed on the second pixel point can also be performed on the remaining pixel points. Therefore, the evaluation values of the pixel points in the second RGB can be finally obtained, which is equivalent to obtaining the second mask.

[0184] 404, fuse the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first four-dimensional point cloud.

[0185] After obtaining the first three-dimensional point cloud and the second three-dimensional point cloud, the first three-dimensional point cloud and the second three-dimensional point cloud can be fused (for example, feature matching, splicing, coordinate system conversion, and voxelization, etc.), so as to obtain the first four-dimensional point cloud. Then, the first four-dimensional point cloud can be used as data for subsequent model training.

[0186] Specifically, the first four-dimensional point cloud can be obtained by the following manner:

[0187] After obtaining the first three-dimensional point cloud, since the plurality of space points contained in the first three-dimensional point cloud are one-to-one corresponding to the plurality of pixel points contained in the first RGB (that is, the first three-dimensional point cloud is obtained by performing three-dimensional projection on each pixel point in the first RGB by using the parameters of the camera, the first semantic map, and the first depth map), the evaluation value of each pixel point in the first RGB can be used as the evaluation value of each space point in the first three-dimensional point cloud. From the first three-dimensional point cloud, the space points with evaluation values less than an evaluation threshold (the size of the threshold can be set according to actual needs, which is not limited here) are removed, so as to obtain the denoised first three-dimensional point cloud.

[0188] Similarly, after obtaining the second three-dimensional point cloud, since the plurality of spatial points contained in the second three-dimensional point cloud are in one-to-one correspondence with the plurality of pixel points contained in the second RGB image (that is, the second three-dimensional point cloud is obtained by performing three-dimensional projection on each pixel point in the second RGB image by using the parameters of the camera, the second semantic image, and the second depth image), the evaluation value of each pixel point in the second RGB image can be taken as the evaluation value of each spatial point in the second three-dimensional point cloud, and the spatial points with evaluation values less than the evaluation threshold value are removed from the second three-dimensional point cloud, thereby obtaining the second three-dimensional point cloud after denoising.

[0189] After obtaining the first three-dimensional point cloud after denoising and the second three-dimensional point cloud after denoising, the first three-dimensional point cloud after denoising and the second three-dimensional point cloud after denoising can be fused (for example, feature matching, splicing, and voxelization, etc.), thereby obtaining the first four-dimensional point cloud.

[0190] Still as in the above example, as shown in FIG. 6 (FIG. 6 is a schematic diagram of obtaining a point cloud provided by an embodiment of the present application, and FIG. 6 is obtained on the basis of FIG. 5), after obtaining the RGB image, the semantic image, the depth image, and the mask film captured by the camera in the pose, projection can be completed by using the RGB image, the semantic image, and the depth image captured by the camera in the pose, thereby obtaining the three-dimensional point cloud corresponding to the pose, and then the three-dimensional point cloud corresponding to the pose is denoised by using the mask film, thereby obtaining the three-dimensional point cloud after denoising corresponding to the pose.

[0191] Similarly, based on the neural implicit field, the RGB image, the semantic image, the depth image, and the mask film captured by the camera in the remaining poses can also be obtained, and projection can be completed by using these images, thereby obtaining the three-dimensional point cloud corresponding to the remaining poses, and then the three-dimensional point cloud corresponding to the remaining poses is denoised by using the mask film, thereby obtaining the three-dimensional point cloud after denoising corresponding to the remaining poses.

[0192] Then, the three-dimensional point cloud after denoising corresponding to the pose and the three-dimensional point cloud after denoising corresponding to the remaining poses can be subjected to feature matching and splicing, thereby obtaining the three-dimensional point cloud in the camera coordinate system, and then the three-dimensional point cloud in the camera coordinate system is converted into the three-dimensional point cloud in the world coordinate system, and then the three-dimensional point cloud in the world coordinate system is voxelized, thereby obtaining the four-dimensional point cloud obtained by voxelization.

[0193] More specifically, the first three-dimensional point cloud can also be optimized by the following method:

[0194] After obtaining the first image and the second image, the first image and the second image can also be input into the second neural network model, so that the first image and the second image are subjected to a series of processing by the second neural network model (for example, a GOD model, etc.), thereby obtaining the second four-dimensional point cloud.

[0195] After obtaining the second four-dimensional point cloud, the first four-dimensional point cloud can be optimized (e.g., the first four-dimensional point cloud and the second four-dimensional point cloud are fused) by using the second four-dimensional point cloud, so as to obtain an optimized first four-dimensional point cloud.

[0196] Still as the above example, as shown in FIG. 7 (FIG. 7 is a schematic diagram of point cloud optimization provided by an embodiment of the present application, and FIG. 7 is obtained based on FIG. 6), after obtaining the denoised three-dimensional point cloud corresponding to the pose and the denoised three-dimensional point clouds corresponding to the rest of the poses, the point clouds can be subjected to feature matching and splicing, so as to obtain a three-dimensional point cloud in a camera coordinate system. Then, the three-dimensional point cloud in the camera coordinate system can be converted into a three-dimensional point cloud in a world coordinate system. Then, the three-dimensional point cloud in the world coordinate system can be voxelized, so as to obtain a four-dimensional point cloud obtained by voxelization.

[0197] In addition, the RGB image, the semantic image and the depth image captured by the camera in the pose, and the RGB image, the semantic image and the depth image captured by the cameras in the rest of the poses can be input into the GOD model, so as to obtain a four-dimensional point cloud predicted by the model. Then, the four-dimensional point cloud obtained by voxelization and the four-dimensional point cloud predicted by the model can be fused, so as to obtain an optimized four-dimensional point cloud.

[0198] It should be understood that, in the embodiment, only the denoising operation is illustratively introduced as being performed before the point cloud fusion operation, and the denoising operation can also be performed in the process of the point cloud fusion operation. For example, after obtaining the first three-dimensional point cloud and the second three-dimensional point cloud, the two three-dimensional point clouds can be subjected to feature matching and splicing first, and then the three-dimensional point cloud in the camera coordinate system is denoised to obtain a denoised three-dimensional point cloud in the camera coordinate system, and then the coordinate system conversion and voxelization are performed, so as to obtain the first four-dimensional point cloud.

[0199] Further, the first neural network model provided by the embodiments of the present application can realize the function of volume rendering, which can simulate the RGB image, semantic image and depth image taken by the camera in different poses, that is, the RGB image, semantic image and depth image in different perspectives, and the subsequent model trained based on the four-dimensional point cloud obtained from these images can more accurately understand the content taken by the camera, thereby having better performance. For example, as shown in FIG. 8 (FIG. 8 is a schematic diagram of an image in the field of autonomous driving provided by the embodiments of the present application), the RGB image, semantic image and depth image in different perspectives obtained by using the first neural network model (implicit neural field), wherein the semantic image (semantic segmentation) and the depth image (depth estimation) are used to obtain the four-dimensional point cloud to train the model in the autonomous driving system, can help the model of the autonomous driving system to more accurately understand the surrounding environment, thereby better planning the driving route and avoiding potential dangers. Specifically, semantic segmentation can assign each pixel in the RGB image to different semantic categories, such as roads, vehicles, pedestrians, etc., thereby helping the model of the autonomous driving system to better understand the road and the surrounding environment. Depth estimation can estimate the distance from each pixel to the camera, thereby helping the autonomous driving system to better understand the three-dimensional structure and distance relationship of the scene. These information can be used to generate high-precision maps, plan safer and more efficient driving routes, and when encountering obstacles or other dangerous situations, the model of the autonomous driving system can quickly respond and take appropriate measures to ensure the driving safety of the vehicle.

[0200] Further, the embodiments of the present application can be compared with related technologies. The three-dimensional point cloud generated by related technologies has certain noise points, and the embodiments of the present application can use the uncertainty (that is, the evaluation value mentioned above) obtained by volume rendering to denoise the obtained three-dimensional point cloud, thereby obtaining the denoised three-dimensional point cloud, which is conducive to constructing a four-dimensional point cloud with better quality to train a model with better performance. For example, as shown in FIG. 9 (FIG. 9 is a schematic diagram of a three-dimensional point cloud provided by the embodiments of the present application), in the three-dimensional point cloud obtained by related technologies (see the left half of FIG. 9), there are many noise points, and the embodiments of the present application can effectively remove these noise points, thereby obtaining a three-dimensional point cloud with better quality (see the right half of FIG. 9).

[0201] Further, the four-dimensional point cloud obtained by the embodiments of the present application can also be compared with the four-dimensional point cloud obtained by related technologies (for example, using a laser radar device), as shown in FIG. 10 (FIG. 10 is a schematic diagram of a four-dimensional point cloud provided by the embodiments of the present application), the three-dimensional scene reflected by the four-dimensional point clouds obtained by the two has comparable accuracy, and the three-dimensional overlap precision (Precision) of the two is 74%, the recall rate (Recall) is 84%, and the intersection of union (IoU) is 64%.

[0202] In the embodiments of the present application, when a four-dimensional point cloud needs to be obtained, a first pose of a camera and a second pose of the camera can be obtained first, and the first pose and the second pose are associated in time. Then, the first pose can be input into a first neural network model to process the first pose by the first neural network model, to obtain a first image captured by the camera, and to convert the first image to obtain a first three-dimensional point cloud. Then, the second pose can also be input into the first neural network model to process the second pose by the first neural network model, to obtain a second image captured by the camera, and to convert the second image to obtain a second three-dimensional point cloud. Finally, the first three-dimensional point cloud and the second three-dimensional point cloud can be fused to obtain a first four-dimensional point cloud. In the foregoing process, the neural network model (i.e., the first neural network model described above, for example, a neural implicit field) can be used to process different poses of the camera at different times (i.e., the first pose and the second pose described above) to obtain three-dimensional point clouds corresponding to different poses (which can also be understood as three-dimensional point clouds at different times, i.e., the first three-dimensional point cloud and the second three-dimensional point cloud described above), and then the three-dimensional point clouds corresponding to different poses are used to obtain a four-dimensional point cloud (i.e., the first four-dimensional point cloud described above). This way of obtaining a four-dimensional point cloud does not need to rely on a lidar, but only needs a camera, which can reduce the hardware cost required for obtaining a point cloud, and is conducive to the implementation of this way of obtaining a four-dimensional point cloud in some cost-limited scenarios.

[0203] Further, in the embodiments of the present application, based on the processing of the pose of the camera at a certain time by the neural network model, the RGB image captured by the camera at the pose, the corresponding semantic image and depth image can be simulated. Since these images have better dense texture features, even if the content in the image is an extreme weather or a low-reflectivity object, various features of these situations can be reflected, so that a high-quality three-dimensional point cloud is successfully obtained based on these images, and then a four-dimensional point cloud is obtained. It can be seen that this way of obtaining a four-dimensional point cloud can be implemented in some scenarios with special conditions.

[0204] Further, in the embodiments of the present application, based on the processing of the pose of the camera at a certain time by the neural network model, the RGB image captured by the camera at the pose, the corresponding semantic image and depth image can be simulated. Since these images have better dense texture features, even if the content in the image is an extreme weather or a low-reflectivity object, various features of these situations can be reflected, so that a high-quality three-dimensional point cloud is successfully obtained based on these images, and then a four-dimensional point cloud is obtained. It can be seen that this way of obtaining a four-dimensional point cloud can be implemented in some scenarios with special conditions.

[0205] Further, in the embodiments of the present application, after obtaining the four-dimensional point cloud, the RGB image, the corresponding semantic image and the depth image can be processed by another neural network model (i.e., the aforementioned second neural network model, such as the GOD model, etc.) to obtain a four-dimensional point cloud predicted by the model, so as to optimize the four-dimensional point cloud obtained based on the three-dimensional point cloud, thereby obtaining a four-dimensional point cloud with better quality.

[0206] The above is a detailed description of the point cloud acquisition method provided in the embodiments of the present application. The model training method provided in the embodiments of the present application will be introduced below. FIG. 11 is a flowchart of a model training method provided in the embodiments of the present application. As shown in FIG. 11, the method comprises the following steps:

[0207] 1101. The pose of the camera is processed by the to-be-trained model to obtain an image captured by the camera, the image being used to acquire a three-dimensional point cloud, and the three-dimensional point cloud being used to acquire a four-dimensional point cloud.

[0208] In the embodiments, when the to-be-trained model needs to be trained, the to-be-trained model (e.g., a to-be-trained neural implicit field) can be acquired first. Then, a batch of training data can be acquired, the batch of training data containing the pose of the camera and a real image captured by the camera in the pose, the real image captured by the camera can contain a real RGB image captured by the camera, a real semantic image corresponding to the real RGB image, and a real depth image corresponding to the real RGB image.

[0209] Then, the pose of the camera can be input to the to-be-trained model to process the pose of the camera by the to-be-trained model, so as to simulate a (predicted) image captured by the camera in the pose, the image being used to acquire a three-dimensional point cloud, and the three-dimensional point cloud being used to acquire a four-dimensional point cloud.

[0210] In a possible implementation manner, the image captured by the camera contains a (predicted) RGB image captured by the camera, a (predicted) semantic image corresponding to the RGB image, and a (predicted) depth image corresponding to the RGB image, wherein the RGB image contains each pixel point, that is, contains the (predicted) color of each pixel point, the semantic image contains the (predicted) category of each pixel point in the RGB image, and the depth image contains the (predicted) depth of each pixel point in the RGB image.

[0211] In a possible implementation, the RGB image contains a target pixel point, the target pixel point being any one of the pixels in the RGB image, and the camera pose is processed by the first neural network model to obtain the image captured by the camera, including: obtaining, by the to-be-trained model, a ray sent to the target pixel point by the camera in the pose; sampling, by the to-be-trained model, the ray to obtain a plurality of sampling points; processing, by the to-be-trained model, the plurality of sampling points to obtain (predicted) colors of the plurality of sampling points, (predicted) categories of the plurality of sampling points, (predicted) depths of the plurality of sampling points, and (predicted) transparencies of the plurality of sampling points; and obtaining, based on the colors of the plurality of sampling points, the categories of the plurality of sampling points, the depths of the plurality of sampling points, and the transparencies of the plurality of sampling points, a color of the target pixel point, a category of the target pixel point, and a depth of the target pixel point.

[0212] In a possible implementation, the obtaining, based on the colors of the plurality of sampling points, the categories of the plurality of sampling points, the depths of the plurality of sampling points, and the transparencies of the plurality of sampling points, the color of the target pixel point, the category of the target pixel point, and the depth of the target pixel point includes: calculating the colors of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the color of the target pixel point; calculating the categories of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the category of the target pixel point; and calculating the depths of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the depth of the target pixel point.

[0213] It should be noted that the description of step 1101 can refer to the related description of step 402 or step 403 in the foregoing embodiments, which will not be repeated here.

[0214] 1102. Obtain a target loss based on the image and the real image corresponding to the image, the target loss being used to indicate a difference between the image and the real image.

[0215] After obtaining the image captured by the camera, since the real image captured by the camera is known, the image captured by the camera and the real image captured by the camera can be processed to obtain a target loss, which can be used to indicate a difference between the image captured by the camera and the real image captured by the camera.

[0216] Specifically, after obtaining the RGB image captured by the camera, the semantic image, and the depth image, since the real RGB image, the real semantic image, and the real depth image are known, the target loss can be obtained based on these images. In the following, any one of the (predicted) RGB image captured by the camera is taken as an example, which is referred to as a target pixel point, the real RGB image contains a real color of the target pixel point, the real semantic image contains a real category of the target pixel point, and the real depth image contains a real depth of the target pixel point. Then, the target loss can be obtained in the following way:

[0217] The color of the target pixel point and the real color of the target pixel point are calculated by the first loss function, so as to obtain a first loss, the first loss being used to indicate the difference between the color of the target pixel point and the real color of the target pixel point.

[0218] The category of the target pixel point and the real category of the target pixel point are calculated by the second loss function, so as to obtain a second loss, the second loss being used to indicate the difference between the category of the target pixel point and the real category of the target pixel point.

[0219] The depth of the target pixel point and the real depth of the target pixel point are calculated by the third loss function, so as to obtain a third loss, the third loss being used to indicate the difference between the depth of the target pixel point and the real depth of the target pixel point.

[0220] The transparency distribution of the plurality of sampling points and the real transparency distribution (also known) of the plurality of sampling points are calculated by the fourth loss function, so as to obtain a fourth loss.

[0221] The first loss, the second loss, the third loss and the fourth loss are superimposed, so as to obtain a target loss.

[0222] For example, assuming that there is a neural implicit field to be trained, after inputting the pose of the camera at a certain moment to the neural implicit field to be trained, a predicted RGB image, a predicted semantic image and a predicted depth image photographed by the camera in the pose can be obtained. Since the real RGB image, the real semantic image and the real depth image photographed by the camera in the pose are known, the target loss can be calculated by the formula:

[0223] In the above formula, (u, v) is a certain pixel point in the predicted RGB image, p i is a certain sampling point on the ray sent by the camera in the pose to (u, v), u(u, v) is the evaluation value of (u, v), r(u, v) is the predicted color of (u, v), r_gt(u, v) is the real color of (u, v) (obtained from the real RGB image), s(u, v) is the predicted category of (u, v), s_gt(u, v) is the real category of (u, v) (obtained from the real semantic image), d(u, v) is the predicted depth of (u, v), and d_gt(u, v) is the real depth of (u, v) (obtained from the real depth image). P(p i ) is the probability density distribution (i.e. the transparency distribution) of the N sampling points, and Q(p i ) is the target probability density distribution (i.e. the real transparency distribution, for example, Gaussian probability distribution, etc.) of the N sampling points.

[0224] After obtaining the four losses of RL(u, v), DL(u, v), SL(u, v), and UL(u, v), the target loss can be obtained by superimposing the four losses.

[0225] 1103、Based on the target loss, the parameters of the to-be-trained model are updated until the model training condition is met, and the first neural network model is obtained.

[0226] After obtaining the target loss, the parameters of the to-be-trained model can be updated using the target loss, and the to-be-trained model with updated parameters can be further trained using the next batch of training data until the model training condition (for example, the target loss converges, etc.) is met. The first neural network model (for example, a trained neural implicit field) in the embodiment shown in FIG. 4 can be obtained.

[0227] In addition, the embodiments of the present application can also be compared with related technologies. The main comparison is the training speed of the neural network model and the accuracy of the depth estimation performed by the trained model. The comparison results are shown in Table 1 and Table 2:

[0228] Table 1

[0229] Table 2

[0230] Based on Table 1 and Table 2, it can be known that the training speed and performance of the model trained by the embodiments of the present application are superior to related technologies.

[0231] The above is a detailed description of the point cloud acquisition method and the model training method provided by the embodiments of the present application. In the following, the point cloud acquisition device and the model training device provided by the embodiments of the present application will be introduced. FIG. 12 is a structural schematic diagram of a point cloud acquisition device provided by an embodiment of the present application. As shown in FIG. 12, the point cloud acquisition device comprises:

[0232] The acquisition module 1201 is configured to acquire a first pose of a camera and a second pose of the camera, the first pose and the second pose being associated in time.

[0233] The first processing module 1202 is configured to process the first pose by using the first neural network model to obtain a first image captured by the camera, and convert the first image to obtain a first three-dimensional point cloud.

[0234] The second processing module 1203 is configured to process the second pose by using the first neural network model to obtain a second image captured by the camera, and convert the second image to obtain a second three-dimensional point cloud.

[0235] The fusion module 1204 is configured to fuse the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first four-dimensional point cloud.

[0236] In a possible implementation, the first image includes a first RGB image captured by the camera, a first semantic map corresponding to the first RGB image, and a first depth map corresponding to the first RGB image, where the first semantic map includes a category of each pixel point in the first RGB image, and the first depth map includes a depth of each pixel point in the first RGB image.

[0237] In a possible implementation, the second image includes a second RGB image captured by the camera, a second semantic map corresponding to the second RGB image, and a second depth map corresponding to the second RGB image, where the second semantic map includes a category of each pixel point in the second RGB image, and the second depth map includes a depth of each pixel point in the second RGB image.

[0238] In a possible implementation, the first RGB image includes a first pixel point, the first pixel point being any one of the pixel points in the first RGB image, and the first processing module 1202 is configured to: acquire, by using the first neural network model, a first ray sent to the first pixel point by the camera in the first pose; sample, by using the first neural network model, the first ray to obtain a plurality of first sampling points; process, by using the first neural network model, the plurality of first sampling points to obtain colors of the plurality of first sampling points, categories of the plurality of first sampling points, depths of the pluralityity of first sampling points, and transparencies of the plurality of first sampling points; and acquire, based on the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points, a color of the first pixel point, a category of the first pixel point, and a depth of the first pixel point.

[0239] In a possible implementation, the second RGB image includes a second pixel point, the second pixel point being any one of the pixel points in the second RGB image, and the second processing module 1203 is configured to: acquire, by using the first neural network model, a second ray sent to the second pixel point by the camera in the second pose; sample, by using the first neural network model, the second ray to obtain a plurality of second sampling points; process, by using the first neural network model, the plurality of second sampling points to obtain colors of the plurality of second sampling points, categories of the plurality of second sampling points, depths of the plurality of second sampling points, and transparencies of the plurality of second sampling points; and acquire, based on the colors of the plurality of second sampling points, the categories of the plurality of second sampling points, the depths of the plurality of second sampling points, and the transparencies of the plurality of second sampling points, a color of the second pixel point, a category of the second pixel point, and a depth of the second pixel point.

[0240] In a possible implementation, the first processing module 1202 is configured to: calculate the color of the first pixel point based on the color of the first sampling points, the category of the first sampling points, the depth of the first sampling points and the transparency of the first sampling points; calculate the category of the first pixel point based on the category of the first sampling points and the transparency of the first sampling points; and calculate the depth of the first pixel point based on the depth of the first sampling points and the transparency of the first sampling points.

[0241] In a possible implementation, the second processing module 1203 is configured to: calculate the color of the second pixel point based on the color of the second sampling points, the category of the second sampling points, the depth of the second sampling points and the transparency of the second sampling points; calculate the category of the second pixel point based on the category of the second sampling points and the transparency of the second sampling points; and calculate the depth of the second pixel point based on the depth of the second sampling points and the transparency of the second sampling points.

[0242] In a possible implementation, the apparatus further includes: a first evaluation module configured to calculate the depth of the first sampling points and the depth of the first pixel point to obtain an evaluation value of the first pixel point; a second evaluation module configured to calculate the depth of the second sampling points and the depth of the second pixel point to obtain an evaluation value of the second pixel point; and a fusion module 1204 configured to: take the evaluation value of each pixel point in the first RGB image as an evaluation value of each spatial point in the first three-dimensional point cloud, remove spatial points with an evaluation value less than an evaluation threshold from the first three-dimensional point cloud to obtain a denoised first three-dimensional point cloud; take the evaluation value of each pixel point in the second RGB image as an evaluation value of each spatial point in the second three-dimensional point cloud, remove spatial points with an evaluation value less than the evaluation threshold from the second three-dimensional point cloud to obtain a denoised second three-dimensional point cloud; and fuse the denoised first three-dimensional point cloud and the denoised second three-dimensional point cloud to obtain the first four-dimensional point cloud.

[0243] In a possible implementation, the apparatus further includes an optimization module configured to: process the first image and the second image by using a second neural network model to obtain a second four-dimensional point cloud; and optimize the first four-dimensional point cloud based on the second four-dimensional point cloud to obtain an optimized first four-dimensional point cloud.

[0244] In a possible implementation, the first neural network model is a neural implicit field.

[0245] FIG. 13 is a structural schematic diagram of a model training apparatus provided by an embodiment of the present application, as shown in FIG. 13, the model training apparatus includes:

[0246] The processing module 1301 is configured to process the pose of the camera by using the to-be-trained model, to obtain an image captured by the camera, the image being used to acquire a three-dimensional point cloud, and the three-dimensional point cloud being used to acquire a four-dimensional point cloud.

[0247] The acquisition module 1302 is configured to acquire a target loss based on the image and a real image corresponding to the image, the target loss being used to indicate a difference between the image and the real image.

[0248] The training module 1303 is configured to update parameters of the to-be-trained model based on the target loss until a model training condition is met, to obtain a first neural network model.

[0249] In a possible implementation, the image includes an RGB image captured by the camera, a semantic image corresponding to the RGB image, and a depth image corresponding to the RGB image, where the semantic image includes categories of respective pixel points in the RGB image, and the depth image includes depths of the respective pixel points in the RGB image.

[0250] In a possible implementation, the RGB image includes a target pixel point, and the target pixel point is any one of pixel points in the RGB image. The processing module 1301 is configured to: acquire, by using the to-be-trained model, a ray sent to the target pixel point by the camera in the pose; sample, by using the to-be-trained model, the ray to obtain a plurality of sampling points; process, by using the to-be-trained model, the plurality of sampling points to obtain colors of the plurality of sampling points, categories of the plurality of sampling points, depths of the plurality of sampling points, and transparencies of the plurality of sampling points; and acquire a color of the target pixel point, a category of the target pixel point, and a depth of the target pixel point based on the colors of the plurality of sampling points, the categories of the plurality of sampling points, the depths of the plurality of sampling points, and the transparencies of the plurality of sampling points.

[0251] In a possible implementation, the processing module 1301 is configured to: calculate the colors of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the color of the target pixel point; calculate the categories of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the category of the target pixel point; and calculate the depths of the plurality of sampling points and the transparencies of the plurality of sampling points to obtain the depth of the target pixel point.

[0252] In a possible implementation, the real image includes a real RGB image, a real semantic image, and a real depth image, the real RGB image includes real colors of the target pixels, the real semantic image includes real categories of the target pixels, and the real depth image includes real depths of the target pixels; the obtaining module 1302 is configured to: obtain a first loss based on the color of the target pixel and the real color of the target pixel, the first loss being used to indicate a difference between the color of the target pixel and the real color of the target pixel; obtain a second loss based on the category of the target pixel and the real category of the target pixel, the second loss being used to indicate a difference between the category of the target pixel and the real category of the target pixel; and obtain a third loss based on the depth of the target pixel and the real depth of the target pixel, the third loss being used to indicate a difference between the depth of the target pixel and the real depth of the target pixel; and obtain the target loss based on the first loss, the second loss, and the third loss.

[0253] In a possible implementation, the obtaining module 1302 is further configured to obtain a fourth loss based on the transparency distribution of the plurality of sampling points and the real transparency distribution of the plurality of sampling points; and the obtaining module 1302 is configured to obtain the target loss based on the first loss, the second loss, the third loss, and the fourth loss.

[0254] In a possible implementation, the first neural network model is a neural implicit field.

[0255] It should be noted that the information interaction and execution process between the modules / units of the apparatus are based on the same consideration as the method embodiments of the present application, and the technical effects brought by the same are the same as those of the method embodiments of the present application. For details, refer to the description of the method embodiments of the foregoing application embodiments.

[0256] The embodiments of the present application also relate to an execution device. FIG. 14 is a structural schematic diagram of an execution device according to an embodiment of the present application. As shown in FIG. 14, the execution device 1400 can specifically be a mobile phone, a tablet, a notebook computer, a smart wearable device, a server, etc., which is not limited here. The execution device 1400 can be deployed with the point cloud obtaining apparatus described in the embodiment corresponding to FIG. 12, to implement the function of obtaining a point cloud in the embodiment corresponding to FIG. 4. Specifically, the execution device 1400 includes a receiver 1401, a transmitter 1402, a processor 1403, and a memory 1404 (wherein the number of processors 1403 in the execution device 1400 can be one or more, and one processor is taken as an example in FIG. 14), wherein the processor 1403 can include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403, and the memory 1404 can be connected through a bus or other means.

[0257] The memory 1404 can include read-only memory and random access memory, and provide the processor 1403 with instructions and data. A portion of the memory 1404 can also include non-volatile random access memory (NVRAM). The memory 1404 stores processor and operating instructions, executable modules, or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions can include various operating instructions for implementing various operations.

[0258] The processor 1403 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, all the buses are referred to as a bus system in the figure.

[0259] The method disclosed in the above embodiments of the present application can be applied in the processor 1403 or implemented by the processor 1403. The processor 1403 can be an integrated circuit chip with a processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 1403 or the instruction in the form of software. The processor 1403 mentioned above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1403 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1404, and the processor 1403 reads the information in the memory 1404 and combines the hardware to complete the steps of the above method.

[0260] The receiver 1401 can be configured to receive inputted digital or character information, and to generate signal input related to relevant settings and function control of the execution device. The transmitter 1402 can be configured to output digital or character information through the first interface; the transmitter 1402 can also be configured to send instructions to the disk group through the first interface to modify data in the disk group; and the transmitter 1402 can further include a display device such as a display screen.

[0261] In an embodiment of the present application, in one case, the processor 1403 is configured to acquire an image captured by a camera by using the first neural network model in the corresponding embodiment of FIG. 4, to acquire a three-dimensional point cloud, and to further acquire a four-dimensional point cloud.

[0262] An embodiment of the present application also relates to a training device, and FIG. 15 is a structural schematic diagram of a training device provided by an embodiment of the present application. As shown in FIG. 15, the training device 1500 is implemented by one or more servers, and the training device 1500 can have relatively large differences due to different configurations or performances, and can include one or more central processing units (CPUs) 1515 (for example, one or more processors) and a memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. The memory 1532 and the storage media 1530 can be temporary storage or persistent storage. The programs stored in the storage media 1530 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the training device. Furthermore, the central processing unit 1515 can be configured to communicate with the storage media 1530, and to execute the series of instruction operations in the storage media 1530 on the training device 1500.

[0263] The training device 1500 can also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, and one or more input and output interfaces 1558; or one or more operating systems 1541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0264] Specifically, the training device can execute the model training method in the corresponding embodiment of FIG. 11.

[0265] An embodiment of the present application also relates to a computer storage medium, and the computer readable storage medium stores a program for signal processing, and when the program runs on a computer, causes the computer to execute the steps performed by the foregoing execution device, or causes the computer to execute the steps performed by the foregoing training device.

[0266] The embodiments of the present application also relate to a computer program product, which stores instructions, which, when executed by a computer, cause the computer to perform the steps performed by the aforementioned execution device, or cause the computer to perform the steps performed by the aforementioned training device.

[0267] The execution device, the training device or the terminal device provided by the embodiments of the present application can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device performs the data processing method described in the above embodiments, or so that the chip in the training device performs the data processing method described in the above embodiments. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0268] Specifically, please refer to FIG. 16, which is a structural schematic diagram of a chip provided by the embodiments of the present application. The chip can be a neural network processor (NPU) 1600, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1603, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1604.

[0269] In some implementations, the operation circuit 1603 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1603 is a two-dimensional systolic array. The operation circuit 1603 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1603 is a general-purpose matrix processor.

[0270] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from a weight memory 1602 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from an input memory 1601 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1608.

[0271] The unified memory 1606 is used to store input data and output data. The weight data is transferred to the weight memory 1602 through a Direct Memory Access Controller (DMAC) 1605. The input data is also transferred to the unified memory 1606 through the DMAC.

[0272] The BIU is a Bus Interface Unit 1614 for the interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1609.

[0273] The BIU is a Bus Interface Unit 1614 for the interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1609.

[0274] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1606 or to transfer the weight data to the weight memory 1602 or to transfer the input data to the input memory 1601.

[0275] The vector computation unit 1607 includes a plurality of operation processing units, which further process the output of the operation circuit 1603 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network computation in neural networks, such as Batch Normalization, pixel-level summation, upsampling of the predicted label plane, etc.

[0276] In some implementations, the vector computation unit 1607 can store the processed output vector to the unified memory 1606. For example, the vector computation unit 1607 can apply a linear function; or, a non-linear function to the output of the operation circuit 1603, such as linear interpolation to the predicted label plane extracted from the convolutional layer, and further, a vector of accumulated values to generate activation values. In some implementations, the vector computation unit 1607 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as activation input to the operation circuit 1603, such as for use in subsequent layers in the neural network.

[0277] The controller 1604 is connected to an instruction fetch buffer 1609 for storing instructions used by the controller 1604;

[0278] The unified memory 1606, the input memory 1601, the weight memory 1602, and the instruction memory 1609 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0279] Any processor mentioned in the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of the above programs.

[0280] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0281] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0282] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0283] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A point cloud acquisition method, characterized in that, The method comprises: obtaining a first pose of a camera and a second pose of the camera, the first pose being associated in time with the second pose; processing the first pose by a first neural network model to obtain a first image captured by the camera, and converting the first image to obtain a first three-dimensional point cloud; processing the second pose by the first neural network model to obtain a second image captured by the camera, and converting the second image to obtain a second three-dimensional point cloud; fusing the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first four-dimensional point cloud.

2. The method of claim 1, wherein, The first image comprises a first red-green-blue (RGB) image captured by the camera, a first semantic image corresponding to the first RGB image, and a first depth image corresponding to the first RGB image, wherein the first semantic image comprises a category of each pixel point in the first RGB image, and the first depth image comprises a depth of each pixel point in the first RGB image.

3. The method of claim 2, wherein, The second image comprises a second RGB image captured by the camera, a second semantic image corresponding to the second RGB image, and a second depth image corresponding to the second RGB image, wherein the second semantic image comprises a category of each pixel point in the second RGB image, and the second depth image comprises a depth of each pixel point in the second RGB image.

4. The method of claim 3, wherein, The first RGB image comprises a first pixel point, the first pixel point being any one of the pixel points in the first RGB image, and the processing of the first pose by the first neural network model to obtain the first image captured by the camera comprises: obtaining, by the first neural network model, a first ray sent to the first pixel point by the camera in the first pose; sampling, by the first neural network model, the first ray to obtain a plurality of first sampling points; processing, by the first neural network model, the plurality of first sampling points to obtain colors of the plurality of first sampling points, categories of the plurality of first sampling points, depths of the plurality of first sampling points, and transparencies of the plurality of first sampling points; based on the colors of the plurality of first sampling points, the categories of the plurality of first sampling points, the depths of the plurality of first sampling points, and the transparencies of the plurality of first sampling points, obtaining a color of the first pixel point, a category of the first pixel point, and a depth of the first pixel point.

5. The method of claim 4, wherein, The second RGB image comprises a second pixel point, the second pixel point being any one of the pixel points in the second RGB image, and the processing of the second pose by the first neural network model to obtain the second image captured by the camera comprises: obtaining, by the first neural network model, a second ray sent to the second pixel point by the camera in the second pose; sampling, by the first neural network model, the second ray to obtain a plurality of second sampling points; obtaining the color, the category, the depth and the transparency of the second sampling points by processing the second sampling points through the first neural network model; obtaining the color, the category and the depth of the second pixel point based on the color, the category, the depth and the transparency of the second sampling points.

6. The method of claim 4, wherein, The obtaining the color, the category and the depth of the first pixel point based on the color, the category, the depth and the transparency of the first sampling points comprises: calculating the color and the transparency of the first sampling points to obtain the color of the first pixel point; calculating the category and the transparency of the first sampling points to obtain the category of the first pixel point; calculating the depth and the transparency of the first sampling points to obtain the depth of the first pixel point.

7. The method according to claim 4 or 5, characterized in that, The obtaining the color, the category and the depth of the second pixel point based on the color, the category, the depth and the transparency of the second sampling points comprises: calculating the color and the transparency of the second sampling points to obtain the color of the second pixel point; calculating the category and the transparency of the second sampling points to obtain the category of the second pixel point; calculating the depth and the transparency of the second sampling points to obtain the depth of the second pixel point.

8. The method according to any one of claims 5 to 7, characterized in that, The method further comprises: calculating the depth of the first sampling points and the depth of the first pixel point to obtain the evaluation value of the first pixel point; calculating the depth of the second sampling points and the depth of the second pixel point to obtain the evaluation value of the second pixel point; The fusing the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first four-dimensional point cloud comprises: taking the evaluation value of each pixel point in the first RGB image as the evaluation value of each spatial point in the first three-dimensional point cloud, removing spatial points with evaluation values less than an evaluation threshold from the first three-dimensional point cloud to obtain a denoised first three-dimensional point cloud; taking the evaluation value of each pixel point in the second RGB image as the evaluation value of each spatial point in the second three-dimensional point cloud, removing spatial points with evaluation values less than the evaluation threshold from the second three-dimensional point cloud to obtain a denoised second three-dimensional point cloud; fusing the denoised first three-dimensional point cloud and the denoised second three-dimensional point cloud to obtain a first four-dimensional point cloud.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: obtaining the second four-dimensional point cloud by processing the first image and the second image through a second neural network model; optimizing the first four-dimensional point cloud based on the second four-dimensional point cloud to obtain an optimized first four-dimensional point cloud.

10. The method according to any one of claims 1 to 9, characterized in that, The first neural network model is a neural implicit field.

11. A model training method, comprising: The method comprises: processing the pose of the camera through a to-be-trained model to obtain an image captured by the camera, the image being used to obtain a three-dimensional point cloud, and the three-dimensional point cloud being used to obtain a four-dimensional point cloud; obtaining a target loss based on the image and a real image corresponding to the image, the target loss being used to indicate the difference between the image and the real image; updating the parameters of the to-be-trained model based on the target loss until a model training condition is met to obtain a first neural network model.

12. The method of claim 11, wherein, The image comprises an RGB image captured by the camera, a semantic image corresponding to the RGB image, and a depth image corresponding to the RGB image, wherein the semantic image comprises the category of each pixel point in the RGB image, and the depth image comprises the depth of each pixel point in the RGB image.

13. The method of claim 12, wherein, The RGB image comprises a target pixel point, and the target pixel point is any one of the pixel points in the RGB image. The processing of the pose through the first neural network model to obtain the image captured by the camera comprises: obtaining a ray sent by the camera in the pose to the target pixel point through a to-be-trained model; sampling the ray through the to-be-trained model to obtain a plurality of sampling points; processing the plurality of sampling points through the to-be-trained model to obtain the color, category, depth, and transparency of the plurality of sampling points; 14. The method of claim 13, wherein, obtaining the color, category, and depth of the target pixel point based on the color, category, depth, and transparency of the plurality of sampling points. The obtaining of the color, category, and depth of the target pixel point based on the color, category, depth, and transparency of the plurality of sampling points comprises: calculating the color and transparency of the plurality of sampling points to obtain the color of the target pixel point; calculating the category and transparency of the plurality of sampling points to obtain the category of the target pixel point; 15. The method according to claim 13 or 14, characterized in that, calculating the depth and transparency of the plurality of sampling points to obtain the depth of the target pixel point. The real image comprises a real RGB image, a real semantic image, and a real depth image, the real RGB image comprising the real color of the target pixel point, the real semantic image comprising the real category of the target pixel point, and the real depth image comprising the real depth of the target pixel point. The obtaining of the target loss based on the image and the real image corresponding to the image comprises: obtaining a first loss based on the color of the target pixel point and the real color of the target pixel point, the first loss being used to indicate a difference between the color of the target pixel point and the real color of the target pixel point; obtaining a second loss based on the category of the target pixel point and the real category of the target pixel point, the second loss being used to indicate a difference between the category of the target pixel point and the real category of the target pixel point; obtaining a third loss based on the depth of the target pixel point and the real depth of the target pixel point, the third loss being used to indicate a difference between the depth of the target pixel point and the real depth of the target pixel point; obtaining a target loss based on the first loss, the second loss and the third loss.

16. The method of claim 15, wherein, The obtaining the target loss based on the image and the real image corresponding to the image further includes: obtaining a fourth loss based on the transparency distribution of the plurality of sampling points and the real transparency distribution of the plurality of sampling points; The obtaining the target loss based on the first loss, the second loss and the third loss includes: obtaining the target loss based on the first loss, the second loss, the third loss and the fourth loss.

17. The method according to any one of claims 11 to 16, characterized in that, The first neural network model is a neural implicit field.

18. A point cloud acquisition apparatus, characterized by comprising: The device includes: an obtaining module, configured to obtain a first pose of a camera and a second pose of the camera, the first pose being associated with the second pose in time; a first processing module, configured to process the first pose by a first neural network model to obtain a first image captured by the camera, and convert the first image to obtain a first three-dimensional point cloud; a second processing module, configured to process the second pose by the first neural network model to obtain a second image captured by the camera, and convert the second image to obtain a second three-dimensional point cloud; a fusion module, configured to fuse the first three-dimensional point cloud and the second three-dimensional point cloud to obtain a first four-dimensional point cloud.

19. A model training apparatus, comprising: The device includes: a processing module, configured to process a pose of a camera by a to-be-trained model to obtain an image captured by the camera, the image being used to obtain a three-dimensional point cloud, and the three-dimensional point cloud being used to obtain a four-dimensional point cloud; an obtaining module, configured to obtain a target loss based on the image and a real image corresponding to the image, the target loss being used to indicate a difference between the image and the real image; a training module, configured to update parameters of the to-be-trained model based on the target loss until a model training condition is met to obtain a first neural network model.

20. A point cloud acquisition apparatus, comprising: The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, when the code is executed, the point cloud obtaining device executes the method in any one of claims 1 to 17.

21. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the method in any one of claims 1 to 17.

22. A computer program product, characterised in that, The computer program product stores instructions which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 17.

Images