Three-dimensional scene perception method and device, electronic equipment and storage medium

Abstract

The invention discloses a three-dimensional scene perception method and device, electronic equipment and a storage medium, and relates to the technical field of automatic driving, and the three-dimensional scene perception method comprises the steps: obtaining a plurality of panoramic images of the surrounding environment of a vehicle and a first query coordinate point of a target region, the target region belongs to a region included by the plurality of panoramic images, and the first query coordinate point belongs to a region included by the plurality of panoramic images; and inputting the plurality of panoramic images and the first query coordinate point into the target three-dimensional perception model to obtain an SDF value and a semantic category of the first query coordinate point, and obtaining a three-dimensional scene corresponding to the first query coordinate point based on the SDF value and the semantic category. Therefore, dense three-dimensional reconstruction is carried out in real time based on the SDF value and the semantic category, a continuous three-dimensional scene perception result is obtained, and the accuracy of three-dimensional scene reconstruction is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a three-dimensional scene perception method, device, electronic device, and storage medium. Background Art

[0002] Currently, 3D scenes are represented by voxel grids (e.g. Figure 1 As shown in the figure, the three-dimensional space is divided into many small blocks. Through multiple camera images, it is analyzed whether each block is occupied, so that obstacles of any shape and their movement can be accurately judged around it, and then a reasonable route is planned for the autonomous vehicle to ensure its driving safety.

[0003] In related technologies, a voxel-based neural network model is used to determine whether each voxel is occupied and the voxel category. Voxel-based neural network models follow an occupancy grid prediction paradigm, requiring a predefined spatial grid resolution, and the prediction results must be output at that resolution, making it impossible to output continuous three-dimensional scene perception results. For example, the voxel grid that can be output is around 40 cm, which is not accurate enough for crowded, narrow roads, or low-speed scenes. Summary of the Invention

[0004] The present disclosure provides a three-dimensional scene perception method, device, electronic device and storage medium to at least solve the technical problems in the related art of being unable to output continuous three-dimensional scene perception results and having low accuracy.

[0005] According to one aspect of the present disclosure, a three-dimensional scene perception method is provided, the method comprising:

[0006] Acquire a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area, wherein the target area belongs to an area included in the plurality of surrounding images;

[0007] Inputting the plurality of surrounding images and the first query coordinate point into a target three-dimensional perception model to obtain an SDF value and a semantic category of the first query coordinate point;

[0008] A three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category.

[0009] Optionally, before inputting the plurality of panoramic images and the first query coordinate point into the target three-dimensional perception model to obtain the SDF value and semantic category of the first query coordinate point, the method further includes:

[0010] Acquire a plurality of sample surrounding images, a lidar point cloud image, an occupancy annotation image, and a second query coordinate point of a sample area of ​​the vehicle's surrounding environment, wherein the sample area belongs to an area included in the plurality of sample surrounding images;

[0011] The three-dimensional perception model to be trained is trained based on the multiple sample surrounding images, the lidar point cloud images, the occupancy annotation images and the second query coordinate points to obtain a target perception model.

[0012] Optionally, the 3D perception model to be trained includes an encoder to be trained and an implicit prediction model to be trained, and the training of the 3D perception model to be trained based on the multiple sample surrounding images, the lidar point cloud images, the occupancy annotation images, and the second query coordinate points to obtain a target perception model includes:

[0013] Extracting voxel features of the plurality of sample surrounding images using the encoder to be trained to obtain three-dimensional voxel features of the sample surrounding images;

[0014] Inputting the three-dimensional voxel feature and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category of the second query coordinate point;

[0015] Based on the predicted SDF value, predicted semantic category, lidar point cloud image, and occupancy annotation image of the second query coordinate point, a loss value of the to-be-trained 3D perception model is obtained through a loss function;

[0016] The parameters of the training encoder and the implicit prediction model to be trained are updated based on the loss value until the three-dimensional perception model to be trained converges to obtain a target perception model.

[0017] Optionally, inputting the three-dimensional voxel feature and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category of the second query coordinate point includes:

[0018] Based on the second query coordinate point, interpolate the three-dimensional voxel feature onto the second query coordinate point using a cubic linear interpolation method to obtain a feature vector of the second query coordinate point;

[0019] Converting the second query coordinate point into a coordinate vector, and concatenating the coordinate vector and the feature vector to obtain a target vector;

[0020] The target vector is input into the SDF head network to be trained and the semantic head network to be trained respectively to obtain the predicted SDF value and predicted semantic category of the second query coordinate point.

[0021] Optionally, the loss function comprises an Eikonal loss function, wherein supervision is performed based on a lidar point cloud image and an occupancy annotated image;

[0022] Constrain the true SDF value of the point at the laser point coordinate in the lidar point cloud image to 0;

[0023] Constrain the minimum SDF value of each point in each grid marked as occupied in the annotation image to be less than 0;

[0024] The SDF values ​​of points occupying the grid marked as traversable in the labeled image are constrained to be greater than 0.

[0025] Optionally, obtaining the three-dimensional scene corresponding to the query coordinate point based on the SDF value and the semantic category includes:

[0026] Based on the SDF value, a reconstructed triangular mesh is obtained by a marching cube algorithm, wherein the vertex coordinates of each mesh are reconstructed using a semantic category corresponding to the vertex coordinates of each mesh.

[0027] Optionally, the method further includes:

[0028] Obtain vehicle trajectory and positioning information;

[0029] Based on the vehicle trajectory and positioning information, a three-dimensional reconstruction result of the driving trajectory is obtained through the target three-dimensional perception model.

[0030] According to another aspect of the present disclosure, a three-dimensional scene perception device is provided, the device comprising:

[0031] an acquisition module, configured to acquire a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area, wherein the target area belongs to an area included in the plurality of surrounding images;

[0032] a processing module, configured to input the plurality of surrounding images and the first query coordinate point into a target three-dimensional perception model to obtain an SDF value and a semantic category of the query coordinate point;

[0033] A reconstruction module obtains a three-dimensional scene corresponding to the query coordinate point based on the SDF value and the semantic category.

[0034] Optionally, the device is further used to:

[0035] Acquire a plurality of sample surrounding images, a lidar point cloud image, an occupancy annotation image, and a second query coordinate point of a sample area of ​​the vehicle's surrounding environment, wherein the sample area belongs to an area included in the plurality of sample surrounding images;

[0036] The three-dimensional perception model to be trained is trained based on the multiple sample surrounding images, the lidar point cloud images, the occupancy annotation images and the second query coordinate points to obtain a target perception model.

[0037] Optionally, the three-dimensional perception model to be trained includes an encoder to be trained and an implicit prediction model to be trained; and the apparatus is further configured to:

[0038] Extracting voxel features of the plurality of sample surrounding images using the encoder to be trained to obtain three-dimensional voxel features of the sample surrounding images;

[0039] Inputting the three-dimensional voxel feature and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category of the second query coordinate point;

[0040] Based on the predicted SDF value, predicted semantic category, lidar point cloud image, and occupancy annotation image of the second query coordinate point, a loss value of the to-be-trained 3D perception model is obtained through a loss function;

[0041] The parameters of the training encoder and the implicit prediction model to be trained are updated based on the loss value until the three-dimensional perception model to be trained converges to obtain a target perception model.

[0042] Optionally, the device is further used to:

[0043] Based on the second query coordinate point, interpolate the three-dimensional voxel feature onto the second query coordinate point using a cubic linear interpolation method to obtain a feature vector of the second query coordinate point;

[0044] Converting the second query coordinate point into a coordinate vector, and concatenating the coordinate vector and the feature vector to obtain a target vector;

[0045] The target vector is input into the SDF head network to be trained and the semantic head network to be trained respectively to obtain the predicted SDF value and predicted semantic category of the second query coordinate point.

[0046] Optionally, the loss function comprises an Eikonal loss function, wherein supervision is performed based on a lidar point cloud image and an occupancy annotated image;

[0047] Constrain the true SDF value of the point at the laser point coordinate in the lidar point cloud image to 0;

[0048] Constrain the minimum SDF value of each point in each grid marked as occupied in the annotation image to be less than 0;

[0049] The SDF values ​​of points occupying the grid marked as traversable in the labeled image are constrained to be greater than 0.

[0050] Optionally, the reconstruction module is specifically configured to:

[0051] Based on the SDF value, a reconstructed triangular mesh is obtained by a marching cube algorithm, wherein the vertex coordinates of each mesh are reconstructed using a semantic category corresponding to the vertex coordinates of each mesh.

[0052] Optionally, the device is further used to:

[0053] Obtain vehicle trajectory and positioning information;

[0054] Based on the vehicle trajectory and positioning information, a three-dimensional reconstruction result of the driving trajectory is obtained through the target three-dimensional perception model.

[0055] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0056] at least one processor; and

[0057] a memory communicatively connected to the at least one processor; wherein,

[0058] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods described in the above aspect.

[0059] According to another aspect of the present disclosure, a vehicle is provided, comprising the above-mentioned electronic device or connected to the above-mentioned electronic device.

[0060] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the aforementioned aspect.

[0061] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of the aforementioned aspects.

[0062] In one or more embodiments of the present disclosure, a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area are obtained, wherein the target area belongs to an area included in the plurality of surrounding images, the plurality of surrounding images and the first query coordinate point are input into a target three-dimensional perception model, an SDF value and a semantic category of the first query coordinate point are obtained, and a three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category. Thus, the present disclosure uses a lidar point cloud image and an occupancy annotation image as supervisory signals, and adopts Eikonal constraints to enable the target three-dimensional perception model to obtain a valid SDF value, so that dense three-dimensional reconstruction can be performed in real time based on the SDF value and the semantic category to obtain a continuous three-dimensional scene perception result. At the same time, the resolution of the three-dimensional scene corresponding to the first query coordinate point in the present disclosure can be arbitrary, that is, a three-dimensional scene of any resolution can be generated, thereby improving the accuracy of three-dimensional scene reconstruction.

[0063] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0065] Figure 1 A schematic diagram showing a voxel grid representing a 3D scene provided by one embodiment of the present disclosure is shown;

[0066] Figure 2 A schematic diagram illustrating a flow chart of a three-dimensional scene perception method provided by another embodiment of the present disclosure;

[0067] Figure 3 A schematic diagram showing a three-dimensional scene provided by an embodiment of the present disclosure;

[0068] Figure 4 A schematic diagram showing a comparison of the prediction accuracy of different three-dimensional scene perception methods provided by another embodiment of the present disclosure;

[0069] Figure 5 A schematic diagram showing a comparison of reconstruction effects of different three-dimensional scene perception methods provided by another embodiment of the present disclosure;

[0070] Figure 6 A schematic diagram illustrating a flow chart of a three-dimensional scene perception method provided by one embodiment of the present disclosure is shown;

[0071] Figure 7 A schematic diagram illustrating a flow chart of a three-dimensional scene perception method provided by one embodiment of the present disclosure is shown;

[0072] Figure 8A schematic diagram illustrating a flow chart of a three-dimensional scene perception method provided by one embodiment of the present disclosure is shown;

[0073] Figure 9a A schematic diagram showing a three-dimensional reconstruction result of a driving trajectory provided by one embodiment of the present disclosure;

[0074] Figure 9b The figure shows the three-dimensional reconstruction result of the driving trajectory provided by one embodiment of the present disclosure;

[0075] Figure 10 A schematic diagram illustrating a flow chart of a three-dimensional scene perception method provided by one embodiment of the present disclosure is shown;

[0076] Figure 11 A schematic structural diagram of a three-dimensional scene perception device provided by one embodiment of the present disclosure is shown;

[0077] Figure 12 3D scene perception method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0079] Figure 2 FIG. 1 is a flow chart of a three-dimensional scene perception method proposed in an embodiment of the present disclosure. Figure 2 The three-dimensional scene perception method includes:

[0080] S201 , obtaining a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area.

[0081] It should be noted that the executor of the three-dimensional scene perception method of this embodiment is a three-dimensional scene perception device, which can be implemented by software and / or hardware. The three-dimensional scene perception device in this embodiment can be configured in an electronic device.

[0082] In this exemplary embodiment, the electronic device may include a terminal device, a server, and other devices, and this embodiment does not limit the electronic device.

[0083] In some embodiments, a plurality of surrounding images of the vehicle's surrounding environment may be acquired by a camera disposed on the vehicle.

[0084] In some embodiments, the target area is an area encompassed by multiple surrounding images. Specifically, the surrounding environment can be selected to obtain multiple first query coordinate points within the target area, and the multiple first query coordinate points are used to generate a three-dimensional scene of the target area. For example, the target area can be within a range of 100 meters in front and behind, 50 meters in the left and right, and 10 meters in height of the surrounding environment. The three-dimensional scene of the target area can also be used to determine the driving trajectory of the autonomous driving system.

[0085] In some embodiments, the resolution of the three-dimensional scene corresponding to the above-mentioned first query coordinate point can be arbitrary, without the need to predefine a fixed resolution, thereby generating a three-dimensional scene with higher accuracy, and then accurately planning a reasonable route for the autonomous driving vehicle, thereby ensuring the driving safety of the autonomous driving vehicle.

[0086] S202: Input a plurality of surrounding images and a first query coordinate point into a target three-dimensional perception model to obtain an SDF (Signed Distance Field) value and a semantic category of the first query coordinate point.

[0087] In some embodiments, after obtaining multiple surrounding images and a first query coordinate point through the above steps, the multiple surrounding images and the first query coordinate point can be input into a target three-dimensional perception model, and the SDF value and semantic category corresponding to the first query coordinate point can be obtained through the target three-dimensional perception model, so that the three-dimensional scene corresponding to the first query coordinate point can be obtained subsequently based on the SDF value and the semantic category.

[0088] In some embodiments, before inputting multiple surrounding images and a first query coordinate point into the target 3D perception model and obtaining the SDF value and semantic category of the first query coordinate point, the method may further include: obtaining multiple sample surrounding images, lidar point cloud images, occupancy annotated images, and a second query coordinate point of a sample area of ​​the vehicle's surrounding environment; and training the to-be-trained 3D perception model based on the multiple sample surrounding images, lidar point cloud images, occupancy annotated images, and the second query coordinate point to obtain the target perception model, wherein the sample area belongs to the area encompassed by the multiple sample surrounding images. This content will be described in detail in subsequent embodiments and is not further elaborated on in the present disclosed embodiments.

[0089] S203: Obtain a three-dimensional scene corresponding to the first query coordinate point based on the SDF value and the semantic category.

[0090] In some embodiments, after obtaining the SDF values ​​and semantic categories corresponding to the plurality of first query points respectively through the above steps, the three-dimensional scene corresponding to the first query coordinate point can be obtained based on the SDF values ​​and semantic categories.

[0091] Specifically, in some embodiments, a triangular mesh can be reconstructed based on the SDF value using a marching cube algorithm. The vertex coordinates of each mesh are reconstructed using the semantic category corresponding to the vertex coordinates of each mesh, thereby performing dense 3D reconstruction in real time to obtain a continuous 3D scene perception result. The above method is the same as the prior art, and a detailed description of the above method can be found in the prior art. The present disclosure will not be elaborated on here.

[0092] For example, Figure 3 FIG. 1 is a schematic diagram of a three-dimensional scene proposed according to this embodiment. Figure 3 As shown, continuous three-dimensional scene perception results can be obtained through the above method.

[0093] In some embodiments, Figure 4 This is a schematic diagram comparing the prediction accuracy of different three-dimensional scene perception methods proposed in the embodiment of the present disclosure. Among them, mIoU is a semantic segmentation evaluation index, specifically the ratio of the intersection and union of two sets of calculated true values ​​and predicted values. In addition, if the value of mIoU is larger, it means that the predicted value is closer to the true value, then the corresponding predicted value accuracy is higher, that is, the corresponding three-dimensional scene perception method prediction is more accurate; if the value of mIoU is smaller, it means that the predicted value is farther away from the true value, then the corresponding predicted value accuracy is lower, that is, the corresponding three-dimensional scene perception method prediction is less accurate. Based on this, if Figure 4 As shown, under the same resolution image and backbone network, the mIoU values ​​corresponding to the present invention are higher than those of other methods, thereby confirming that the prediction results of the present invention are more accurate.

[0094] In some embodiments, Figure 5 A comparative schematic diagram of the reconstruction effects of different three-dimensional scene perception methods proposed in the embodiments of the present invention. Among them, the reconstruction effects of different methods are evaluated by the evaluation method of the depth prediction task. In addition, AbsRel (AbsoluteRelative Error) represents the average value of the ratio of the absolute difference between the predicted depth and the true depth to the true depth, and the smaller the AbsRel, the smaller the error and the better the reconstruction effect; RMSE (RootMean Squared Error) represents the square root of the average difference between the predicted depth and the true depth, and the smaller the RMSE, the smaller the error and the better the reconstruction effect; SqRel (Squared Relative error) represents the average value of the ratio of the square of the absolute difference between the predicted depth and the true depth to the true depth, and the smaller the SqRel, the smaller the error and the better the reconstruction effect. Based on this, Figure 5As shown, the values ​​of AbsRel, RMSE and SqRel corresponding to the present invention are lower than those of other methods, thereby confirming that the three-dimensional reconstruction effect obtained by the present invention is better.

[0095] In one or more embodiments of the present disclosure, a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area are obtained, wherein the target area belongs to an area included in the plurality of surrounding images, the plurality of surrounding images and the first query coordinate point are input into a target three-dimensional perception model, an SDF value and a semantic category of the first query coordinate point are obtained, and a three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category. Thus, the present disclosure uses a lidar point cloud image and an occupancy annotation image as supervisory signals, and adopts Eikonal constraints to enable the target three-dimensional perception model to obtain a valid SDF value, so that dense three-dimensional reconstruction can be performed in real time based on the SDF value and the semantic category to obtain a continuous three-dimensional scene perception result. At the same time, the resolution of the three-dimensional scene corresponding to the first query coordinate point in the present disclosure can be arbitrary, that is, a three-dimensional scene of any resolution can be generated, thereby improving the accuracy of three-dimensional scene reconstruction.

[0096] Figure 6 FIG. 1 is a flow chart of a three-dimensional scene perception method proposed in an embodiment of the present disclosure. Figure 6 As shown, the three-dimensional scene perception method may include:

[0097] S601: Acquire a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area.

[0098] For the detailed description of S601, please refer to the above embodiment and will not be repeated here.

[0099] S602, obtaining a plurality of sample surrounding images, lidar point cloud images, occupancy annotation images and second query coordinate points of the sample area of ​​the vehicle's surrounding environment.

[0100] In some embodiments, a lidar point cloud image and an occupancy annotation image correspond to the sample surrounding image. The lidar point cloud image is generated by a lidar sensor, and points on the object surface are determined using laser points in the lidar point cloud image. The occupancy annotation image is an annotated image of the SDF GT corresponding to the sample image, which facilitates the subsequent determination of points inside and outside the object.

[0101] In some embodiments, the sample area belongs to an area included in a plurality of sample ambient images.

[0102] S603: Training the to-be-trained 3D perception model based on the plurality of sample surrounding images, the lidar point cloud images, the occupancy annotation images, and the second query coordinate point to obtain a target perception model.

[0103] In some embodiments, the above-mentioned three-dimensional perception model to be trained may include an encoder to be trained and an implicit prediction model to be trained. In addition, the above-mentioned method of training the three-dimensional perception model to be trained based on multiple sample surrounding images, lidar point cloud images, occupancy annotation images and the second query coordinate point to obtain a target perception model may include the following steps: using the encoder to be trained to extract voxel features of multiple sample surrounding images to obtain three-dimensional voxel features of the sample surrounding images, and then inputting the three-dimensional voxel features and the second query coordinate point into the implicit prediction model to be trained to obtain the predicted SDF value and predicted semantic category of the second query coordinate point, and based on the predicted SDF value, predicted semantic category, lidar point cloud image and occupancy annotation image of the second query coordinate point, obtain the loss value of the three-dimensional perception model to be trained through the loss function, and update the parameters of the training encoder and the implicit prediction model to be trained based on the loss value until the three-dimensional perception model to be trained converges to obtain the target perception model. This part will be described in detail in subsequent embodiments, and the embodiments of the present disclosure will not be repeated here.

[0104] In some embodiments, the SDF value returns the closest distance between a given point in any spatial location and an object in the spatial scene. Therefore, the true SDF value corresponding to the second query point cannot be obtained in the real scene. Based on this, in the aforementioned process of training the 3D perception model based on multiple sample surrounding images, LiDAR point cloud images, occupancy annotated images, and the second query coordinate point, the LiDAR point cloud images and occupancy annotated images are used as supervisory signals, and Eikonal constraints are employed to make the SDF value predicted by the target perception model more accurate.

[0105] S604: Input the plurality of surrounding images and the first query coordinate point into the target three-dimensional perception model to obtain the SDF value and semantic category of the first query coordinate point.

[0106] S605 : Obtain a three-dimensional scene corresponding to the first query coordinate point based on the SDF value and the semantic category.

[0107] For the description of S604-S605, please refer to the above embodiment for details, which will not be repeated here.

[0108] In one or more embodiments of the present disclosure, a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area are obtained, wherein the target area belongs to an area included in the plurality of surrounding images, the plurality of surrounding images and the first query coordinate point are input into a target three-dimensional perception model, an SDF value and a semantic category of the first query coordinate point are obtained, and a three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category. Thus, the present disclosure uses a lidar point cloud image and an occupancy annotation image as supervisory signals, and adopts Eikonal constraints to enable the target three-dimensional perception model to obtain a valid SDF value, so that dense three-dimensional reconstruction can be performed in real time based on the SDF value and the semantic category to obtain a continuous three-dimensional scene perception result. At the same time, the resolution of the three-dimensional scene corresponding to the first query coordinate point in the present disclosure can be arbitrary, that is, a three-dimensional scene of any resolution can be generated, thereby improving the accuracy of three-dimensional scene reconstruction.

[0109] Figure 7 FIG. 1 is a flow chart of a three-dimensional scene perception method proposed in an embodiment of the present disclosure. Figure 7 As shown, the three-dimensional scene perception method may include:

[0110] S701: Acquire multiple surrounding images of the vehicle's surroundings and a first query coordinate point of a target area.

[0111] The description of S701 can be found in the above embodiment and will not be repeated here.

[0112] S702 , obtaining a plurality of sample surrounding images, lidar point cloud images, occupancy annotation images, and second query coordinate points of the sample area of ​​the vehicle's surrounding environment.

[0113] S703 , extracting voxel features of a plurality of sample surrounding images using the encoder to be trained to obtain three-dimensional voxel features of the sample surrounding images.

[0114] In some embodiments, the encoder to be trained may include a 2D backbone network, an LSS network, and a BEV backbone network. Furthermore, after acquiring multiple sample surround-view images of the vehicle's surroundings, the 2D backbone network may be used to extract 2D image features from these multiple sample surround-view images. The resulting 2D image features are then input into the LSS network to obtain BEV (Bird's-Eye View) features corresponding to the 2D image features. The BEV features are then input into the BEV backbone network to obtain 3D voxel features. Each voxel is considered a small 3D spatial unit, and the point cloud points contained within the voxel are aggregated into a voxel feature representation.

[0115] S704 : Input the three-dimensional voxel feature and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category of the second query coordinate point.

[0116] In some embodiments, the implicit prediction model to be trained includes a SDF head network to be trained and a semantic head network to be trained. Furthermore, in some embodiments, the method of inputting the three-dimensional voxel features and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category for the second query coordinate point may include the following steps:

[0117] Step a: Based on the second query coordinate point, use a cubic linear interpolation method to interpolate the three-dimensional voxel feature to the second query coordinate point to obtain a feature vector of the second query coordinate point;

[0118] In some embodiments, the above-mentioned feature vectors correspond to the second query coordinate points in a one-to-one manner, that is, each of the plurality of second query coordinate points has a corresponding feature vector.

[0119] In some embodiments, the above-mentioned linear interpolation method is the same as that in the prior art. For a detailed introduction to the above-mentioned method, reference may be made to the prior art, and the implementation of the present disclosure will not be described in detail here.

[0120] Step b, converting the second query coordinate point into a coordinate vector, and concatenating the coordinate vector and the feature vector to obtain a target vector;

[0121] In some embodiments, the second query coordinate point may be converted into a coordinate vector using a PE coordinate conversion algorithm.

[0122] In addition, in some embodiments, after obtaining the coordinate vector and feature vector corresponding to each second query coordinate point, the feature vector corresponding to each second query coordinate point can be spliced ​​according to the order in which the query coordinate points are input to obtain a first splicing vector, and the coordinate vector corresponding to each second query coordinate point can be spliced ​​to obtain a second splicing vector, and then the first splicing vector and the second splicing vector can be spliced ​​to obtain a target vector.

[0123] Step c: input the target vector into the SDF head network to be trained and the semantic head network to be trained respectively to obtain the predicted SDF value and predicted semantic category of the second query coordinate point.

[0124] In some embodiments, the target vector is input into the SDF head network to be trained to obtain the predicted SDF value corresponding to each second query coordinate point; the target vector is input into the semantic head network to be trained to obtain the probability of each semantic category in the semantic head network corresponding to each second query coordinate point, and the semantic type with the highest probability is determined as the predicted semantic category of the second query coordinate point.

[0125] In some embodiments, the SDF header network to be trained and the semantic header network to be trained may be multi-layer fully connected layers.

[0126] S705 , based on the predicted SDF value of the second query coordinate point, the predicted semantic category, the lidar point cloud image, and the occupancy annotation image, a loss value of the 3D perception model to be trained is obtained through a loss function.

[0127] In some embodiments, for a point in space, Indicates the SDF value at point x.

[0128] Among them, for the predicted SDF value, the loss function based on the Eikonal constraint is as follows:

[0129]

[0130]

[0131] Among them, θ0 is the three-dimensional space of the sample area, θ1 is the set of surface points of the object in the lidar point cloud image, and Ω2 is the set of points with SDF GT annotations in the occupancy annotation image. is the gradient of the SDF at point x, n(x) is the true value of the surface normal vector, is the estimated SDF value at point x, and SDF(x) is the true SDF value at point x.

[0132] In some embodiments, the SDF value is a given position point in any space, and the distance between this point and the spatial scene object is returned. Therefore, it is impossible to obtain the true value of the SDF value corresponding to the second query point. Based on this, supervision is performed through LiDAR point cloud images and occupancy annotation images. Specifically, in some embodiments, LiDAR points are used as supervision for the SDF of the object surface. Due to the characteristics of the LiDAR sensor, the position where the laser point appears on the object surface is constrained to 0 based on this.

[0133] In some embodiments, supervision of points inside and outside an object is achieved based on occupancy annotations in an occupancy annotation image. Specifically, based on the generation rules of occupancy annotations, at least a portion of a voxel grid labeled "occupied" is inside the object. Based on this, the minimum SDF value of each point in each grid labeled "occupied" is constrained to be less than 0 (i.e., SDF < 0) to ensure that the point exists within the grid labeled "occupied"; the entire voxel grid labeled "passable" is outside the object. Based on this, the SDF value of the points in the grid labeled "passable" is constrained to be greater than 0 (SDF > 0). Based on the above description, O2 can be expressed as:

[0134]

[0135] in, represents the i-th grid marked as occupied, is the jth grid marked as passable, and ψ() is a monotonically decreasing function.

[0136] In some embodiments, based on the above description, the loss function may be:

[0137]

[0138]

[0139]

[0140] S706: Update the parameters of the training encoder and the implicit prediction model to be trained based on the loss value until the 3D perception model to be trained converges to obtain the target perception model.

[0141] S707: Input the plurality of surrounding images and the first query coordinate point into the target three-dimensional perception model to obtain the SDF value and semantic category of the first query coordinate point.

[0142] S708 : Obtain a three-dimensional scene corresponding to the first query coordinate point based on the SDF value and the semantic category.

[0143] For the description of S707 - S708 , please refer to the above embodiment for details, which will not be repeated here.

[0144] In one or more embodiments of the present disclosure, a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area are obtained, wherein the target area belongs to an area included in the plurality of surrounding images, the plurality of surrounding images and the first query coordinate point are input into a target three-dimensional perception model, an SDF value and a semantic category of the first query coordinate point are obtained, and a three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category. Thus, the present disclosure uses a lidar point cloud image and an occupancy annotation image as supervisory signals, and adopts Eikonal constraints to enable the target three-dimensional perception model to obtain a valid SDF value, so that dense three-dimensional reconstruction can be performed in real time based on the SDF value and the semantic category to obtain a continuous three-dimensional scene perception result. At the same time, the resolution of the three-dimensional scene corresponding to the first query coordinate point in the present disclosure can be arbitrary, that is, a three-dimensional scene of any resolution can be generated, thereby improving the accuracy of three-dimensional scene reconstruction.

[0145] Figure 8 FIG. 1 is a flow chart of a three-dimensional scene perception method proposed in an embodiment of the present disclosure. Figure 8 As shown, the three-dimensional scene perception method may include:

[0146] S801: Acquire multiple surrounding images of the vehicle's surroundings and a first query coordinate point of a target area.

[0147] S802: Input a plurality of panoramic images and a first query coordinate point into a target three-dimensional perception model to obtain an SDF value and a semantic category of the first query coordinate point.

[0148] S803: Obtain a three-dimensional scene corresponding to the first query coordinate point based on the SDF value and the semantic category.

[0149] For the description of S801-S803, please refer to the above embodiment for details, which will not be repeated here.

[0150] S804: Obtain vehicle trajectory and positioning information.

[0151] In some embodiments, the ego-vehicle trajectory may be a trajectory of the vehicle.

[0152] S805 , based on the vehicle trajectory and positioning information, obtain a three-dimensional reconstruction result of the driving trajectory through the target three-dimensional perception model.

[0153] In some embodiments, the vehicle voxel space is obtained along the vehicle trajectory using positioning information and a target encoder, and then the coordinate point to be queried in the world coordinate system is mapped to the vehicle coordinate system to obtain a third query coordinate point. Based on the third query point and the vehicle voxel space, the corresponding SDF value and semantic category are obtained through the target implicit prediction model.

[0154] In some embodiments, when performing 3D reconstruction of the driving trajectory, a momentum update strategy can be adopted for the overlapping area of ​​the historical area and the current frame area. That is, the reconstruction of the overlapping area can be performed using part of the historical area and part of the current frame area. For example, the reconstruction area result = 0.8 historical reconstruction result + 0.2 current frame reconstruction result. For example, Figure 9a Schematic diagram of a three-dimensional reconstruction result of a driving trajectory proposed according to an embodiment of the present disclosure. Figure 9a As shown, there is a dynamic overlap between the current frame area and the historical area, and the above method is used to reconstruct the dynamic overlap area. In some embodiments, the three-dimensional reconstruction result of the driving trajectory can be obtained by using the marching cube algorithm based on the SDF value.

[0155] For example, Figure 9b This is a three-dimensional reconstruction result of a driving trajectory proposed according to an embodiment of the present disclosure. Figure 9b As shown, the scene reconstruction results under the global perspective and the scene reconstruction results under the local perspective can be obtained.

[0156] In some embodiments, multiple surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area are obtained, wherein the target area belongs to an area included in the multiple surrounding images, the multiple surrounding images and the first query coordinate point are input into a target three-dimensional perception model, the SDF value and semantic category of the first query coordinate point are obtained, and the three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category. Therefore, the present disclosure uses the lidar point cloud image and the occupancy annotation image as supervision signals, and adopts Eikonal constraints to enable the target three-dimensional perception model to obtain a valid SDF value, so that dense three-dimensional reconstruction can be performed in real time based on the SDF value and the semantic category to obtain a continuous three-dimensional scene perception result. At the same time, the resolution of the three-dimensional scene corresponding to the first query coordinate point in the present disclosure can be arbitrary, that is, a three-dimensional scene of any resolution can be generated, thereby improving the accuracy of three-dimensional scene reconstruction.

[0157] Figure 10 FIG. 1 is a flow chart showing a three-dimensional scene perception method according to an exemplary embodiment. Figure 10 As shown in the figure, the panoramic image is input into the encoder to obtain voxel features, and then the voxel features and the query coordinate points are input into the target implicit prediction model to obtain the SDF value and semantic type, and the reconstructed three-dimensional scene is obtained according to the SDF value and semantic type.

[0158] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0159] See Figure 11 , which shows a schematic diagram of the structure of a three-dimensional scene perception device provided by an embodiment of the present disclosure. The three-dimensional scene perception device can be applied to a terminal device. The three-dimensional scene perception device 1100 includes an acquisition module 1101, a processing module 1102, and a reconstruction module 1103, wherein:

[0160] An acquisition module 1101 is configured to acquire a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area, wherein the target area belongs to an area included in the plurality of surrounding images;

[0161] A processing module 1102 is configured to input the plurality of surrounding images and the first query coordinate point into a target three-dimensional perception model to obtain an SDF value and a semantic category of the query coordinate point;

[0162] The reconstruction module 1103 obtains the three-dimensional scene corresponding to the query coordinate point based on the SDF value and the semantic category.

[0163] Optionally, the above device is further used for:

[0164] Acquire a plurality of sample surrounding images, a lidar point cloud image, an occupancy annotation image, and a second query coordinate point of a sample area of ​​the vehicle's surrounding environment, wherein the sample area belongs to an area included in the plurality of sample surrounding images;

[0165] The three-dimensional perception model to be trained is trained based on multiple sample surrounding images, lidar point cloud images, occupancy annotation images and the second query coordinate point to obtain a target perception model.

[0166] Optionally, the 3D perception model to be trained includes an encoder to be trained and an implicit prediction model to be trained; the above apparatus is further used to:

[0167] Extracting voxel features of a plurality of sample surrounding images using the encoder to be trained to obtain three-dimensional voxel features of the sample surrounding images;

[0168] Inputting the three-dimensional voxel features and the second query coordinate point into the implicit prediction model to be trained, obtaining the predicted SDF value and predicted semantic category of the second query coordinate point;

[0169] Based on the predicted SDF value of the second query coordinate point, the predicted semantic category, the lidar point cloud image, and the occupancy annotation image, a loss function is used to obtain the loss value of the 3D perception model to be trained.

[0170] The parameters of the training encoder and the implicit prediction model to be trained are updated based on the loss value until the 3D perception model to be trained converges to obtain the target perception model.

[0171] Optionally, the above device is further used for:

[0172] Based on the second query coordinate point, interpolating the three-dimensional voxel feature onto the second query coordinate point using a cubic linear interpolation method to obtain a feature vector of the second query coordinate point;

[0173] Convert the second query coordinate point into a coordinate vector, and concatenate the coordinate vector and the feature vector to obtain a target vector;

[0174] The target vector is input into the SDF head network to be trained and the semantic head network to be trained respectively to obtain the predicted SDF value and predicted semantic category of the second query coordinate point.

[0175] Optionally, the loss function comprises an Eikonal loss function, wherein supervision is performed based on a lidar point cloud image and an occupancy annotated image;

[0176] Constrain the true SDF value of the point at the laser point coordinate in the lidar point cloud image to 0;

[0177] Constrain the minimum SDF value of each point in each grid marked as occupied in the annotation image to be less than 0;

[0178] The SDF values ​​of points occupying the grid marked as traversable in the labeled image are constrained to be greater than 0.

[0179] Optionally, the reconstruction module 1103 is specifically configured to:

[0180] Based on the SDF value, the reconstructed triangular mesh is output by the marching cube algorithm, wherein the vertex coordinates of each mesh are reconstructed using the semantic category corresponding to the vertex coordinates of each mesh.

[0181] Optionally, the above device is further used for:

[0182] Obtain vehicle trajectory and positioning information;

[0183] Based on the vehicle trajectory and positioning information, the three-dimensional reconstruction result of the driving trajectory is obtained through the target three-dimensional perception model.

[0184] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0185] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0186] Figure 12 1 is a block diagram illustrating a vehicle 1200 according to an exemplary embodiment. For example, vehicle 1200 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or another type of vehicle. Vehicle 1200 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0187] Reference Figure 12 Vehicle 1200 may include various subsystems, such as an infotainment system 1210, a perception system 1220, a decision-making and control system 1230, a drive system 1240, and a computing platform 1250. Vehicle 1200 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of vehicle 1200 may be interconnected via wired or wireless means.

[0188] In some embodiments, the infotainment system 1210 may include a communication system, an entertainment system, a navigation system, and the like.

[0189] The perception system 1220 may include several sensors for sensing information about the environment surrounding the vehicle 1200. For example, the perception system 1220 may include a global positioning system (which may be a GPS system, a BeiDou system, or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter-wave radar, an ultrasonic radar, and a camera.

[0190] The decision control system 1230 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0191] Drive system 1240 may include components that provide power to vehicle 1200. In one embodiment, drive system 1240 may include an engine, a power source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination thereof. The engine is capable of converting energy provided by the power source into mechanical energy.

[0192] Some or all functions of the vehicle 1200 are controlled by a computing platform 1250. The computing platform 1250 may include at least one processor 1251 and a memory 1252. The processor 1251 may execute instructions 1259 stored in the memory 1252.

[0193] The processor 1251 can be any conventional processor, such as a commercially available CPU. The processor can also include a graphics processor (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0194] Memory 1252 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0195] In addition to instructions 1259 , memory 1252 may also store data, such as road maps, route information, and data on the vehicle's location, direction, speed, etc. The data stored in memory 1252 may be used by computing platform 1250 .

[0196] In an embodiment of the present disclosure, the processor 1251 may execute instruction 1259 to complete all or part of the steps of the above-mentioned vehicle perception model pre-training method.

[0197] Furthermore, the word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word exemplary is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies to A or B" is intended to mean any of the natural inclusive permutations. That is, if X applies to A; X applies to B; or X applies to both A and B, then "X applies to A or B" satisfies any of the aforementioned instances. Furthermore, the articles "a" and "an," as used in this application and the appended claims, are generally understood to mean "one or more," unless otherwise specified or clear from the context to indicate a singular form.

[0198] Likewise, although the present disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art after reading and understanding the specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terms used to describe such components are intended to correspond to any component (functionally equivalent) that performs the specific functions of the described components, even if structurally not equivalent to the disclosed structures. In addition, although specific features of the present disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations as may be desired and beneficial for any given or specific application. In addition, with respect to the terms "including," "having," "having," "having," or variations thereof used in the specific embodiments or claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0199] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0200] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0201] In the foregoing detailed description, reference is made to the accompanying drawings, which illustrate, by way of illustration, specific aspects of the present disclosure in which it may be practiced. In this regard, terms indicating directions or expressing positional relationships, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," "clockwise," "counterclockwise," "axial," "radial," "circumferential," etc., may be used with reference to the orientation of the figures being described. Since the components of the described devices may be positioned in a plurality of different orientations, the directional terms may be used for illustrative purposes rather than restrictive. It should be understood that other aspects may be utilized and structural or logical changes may be made without departing from the concepts of the present disclosure. Therefore, the following detailed description should not be taken in a limiting sense.

[0202] It should be understood that, unless otherwise specifically noted, the features of the various embodiments of the present disclosure described herein may be combined with each other. As used herein, the term "and / or" includes any one of the relevant listed items and any combination of any two or more thereof; similarly, "at least one of" includes any one of the relevant listed items and any combination of any two or more thereof.

[0203] It should be understood that, unless otherwise expressly specified or limited, the terms "join," "attach," "install," "connect," "connect," "fix," etc. used in the embodiments of the present disclosure should be understood in a broad sense. For example, they can be fixedly connected, detachably connected, or integrated; they can be mechanically connected, electrically connected, or communicable with each other; they can be directly connected, or indirectly connected through an intermediate medium, and they can be internally connected between two elements or an interactive relationship between two elements, unless otherwise expressly limited. For those skilled in the art, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0204] In addition, the term "over" as used in reference to a component, element, or material layer being formed "over" or located "over" a surface may be used herein to mean that the component, element, or material layer is "indirectly" positioned (e.g., placed, formed, deposited, etc.) on the surface such that one or more additional components, elements, or layers are disposed between the surface and the component, element, or material layer. However, the term "over" as used in reference to a component, element, or material layer being formed "over" or located "over" a surface may alternatively have a specific meaning: the component, element, or material layer is "directly" positioned (e.g., placed, formed, deposited, etc.) on the surface, e.g., in direct contact with the surface.

[0205] Although terms such as "first", "second" and "third" may be used herein to describe various components, parts, regions, layers or sections, these components, parts, regions, layers or sections are not limited to these terms. On the contrary, these terms are only used to distinguish one component, part, region, layer or section from another component, part, region, layer or section. Therefore, without departing from the teachings of each example, the first component, part, region, layer or section mentioned in the examples described herein may also be referred to as the second component, part, region, layer or section. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one such feature. In the description herein, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically defined.

[0206] It should be understood that spatially relative terms such as "above," "upper," "below," and "lower" are used herein to describe the relationship of one element to another element shown in the figures. In addition to the orientation depicted in the figures, such spatially relative terms are intended to encompass different orientations of the device in use or operation. For example, if the device in the figures is flipped, an element described as being "above" or "upper" relative to another element will be "below" or "lower" relative to the other element. Thus, the term "above" encompasses both above and below orientations, depending on the spatial orientation of the device. The device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative terms used herein should be interpreted accordingly.

Claims

1. A three-dimensional scene perception method, characterized in that: The method comprises: Acquire a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area, wherein the target area belongs to an area included in the plurality of surrounding images; Inputting the plurality of surrounding images and the first query coordinate point into a target three-dimensional perception model to obtain a signed distance field (SDF) value and a semantic category of the first query coordinate point; A three-dimensional scene corresponding to the first query coordinate point is obtained based on the SDF value and the semantic category.

2. The method according to claim 1, wherein Before inputting the plurality of surrounding images and the first query coordinate point into the target three-dimensional perception model to obtain the SDF value and semantic category of the first query coordinate point, the method further includes: Acquire a plurality of sample surrounding images, a lidar point cloud image, an occupancy annotation image, and a second query coordinate point of a sample area of ​​the vehicle's surrounding environment, wherein the sample area belongs to an area included in the plurality of sample surrounding images; The three-dimensional perception model to be trained is trained based on the multiple sample surrounding images, the lidar point cloud images, the occupancy annotation images and the second query coordinate points to obtain a target perception model.

3. The method according to claim 2, wherein The three-dimensional perception model to be trained includes an encoder to be trained and an implicit prediction model to be trained; and the three-dimensional perception model to be trained is trained based on the plurality of sample surrounding images, the lidar point cloud images, the occupancy annotation images, and the second query coordinate points to obtain a target perception model, including: Extracting voxel features of the plurality of sample surrounding images using the encoder to be trained to obtain three-dimensional voxel features of the sample surrounding images; Inputting the three-dimensional voxel feature and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category of the second query coordinate point; Based on the predicted SDF value, predicted semantic category, lidar point cloud image, and occupancy annotation image of the second query coordinate point, a loss value of the to-be-trained 3D perception model is obtained through a loss function; The parameters of the training encoder and the implicit prediction model to be trained are updated based on the loss value until the three-dimensional perception model to be trained converges to obtain a target perception model.

4. The method according to claim 3, wherein Inputting the three-dimensional voxel feature and the second query coordinate point into the implicit prediction model to be trained to obtain a predicted SDF value and a predicted semantic category of the second query coordinate point includes: Based on the second query coordinate point, interpolate the three-dimensional voxel feature onto the second query coordinate point using a cubic linear interpolation method to obtain a feature vector of the second query coordinate point; Converting the second query coordinate point into a coordinate vector, and concatenating the coordinate vector and the feature vector to obtain a target vector; The target vector is input into the SDF head network to be trained and the semantic head network to be trained respectively to obtain the predicted SDF value and predicted semantic category of the second query coordinate point.

5. The method according to claim 3, wherein The loss function includes an Eikonal loss function, wherein supervision is performed based on a lidar point cloud image and an occupancy annotated image; Constrain the true SDF value of the point at the laser point coordinate in the lidar point cloud image to 0; Constrain the minimum SDF value of each point in each grid marked as occupied in the annotation image to be less than 0; The SDF values ​​of points occupying the grid marked as traversable in the labeled image are constrained to be greater than 0.

6. The method according to claim 1, wherein The obtaining of the three-dimensional scene corresponding to the query coordinate point based on the SDF value and the semantic category includes: Based on the SDF value, a reconstructed triangular mesh is obtained by a marching cube algorithm, wherein the vertex coordinates of each mesh are reconstructed using a semantic category corresponding to the vertex coordinates of each mesh.

7. The method according to claim 1, wherein The method further comprises: Obtain vehicle trajectory and positioning information; Based on the vehicle trajectory and positioning information, a three-dimensional reconstruction result of the driving trajectory is obtained through the target three-dimensional perception model.

8. A three-dimensional scene perception device, characterized in that: The device comprises: an acquisition module, configured to acquire a plurality of surrounding images of the vehicle's surrounding environment and a first query coordinate point of a target area, wherein the target area belongs to an area included in the plurality of surrounding images; a processing module, configured to input the plurality of surrounding images and the first query coordinate point into a target three-dimensional perception model to obtain an SDF value and a semantic category of the query coordinate point; A reconstruction module obtains a three-dimensional scene corresponding to the query coordinate point based on the SDF value and the semantic category.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory in communicative connection with the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A vehicle, characterized in that: The device comprises the electronic device according to claim 9 or is connected to the electronic device according to claim 9.

11. A computer storage medium, characterized in that The computer storage medium stores computer-executable instructions; after the computer-executable instructions are executed by the processor, the method according to any one of claims 1 to 7 can be implemented.

Images