A vehicle digital twin method suitable for autonomous driving scenarios

Abstract

The present invention discloses a vehicle digital twin method suitable for autonomous driving scenarios, which is an application of new view synthesis for rendering a high-fidelity 3D view of a vehicle in a traffic scene. It mainly includes the following steps: 1. Data preprocessing to obtain input data for model rendering; 2. Converting the camera space to center it on the object; 3. Encoding the image using a neural network encoder; 4. Decoding the shape and appearance of the object using a decoder; 5. Rendering pixel by pixel using volume rendering; 6. Coordinate system conversion for practical application in vehicle-side and road-side scenes. The present invention solves the dilemma of NeRF application in traffic vehicle scenarios and realizes the application of digital twin technology in autonomous driving scenarios.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and computer graphics, and is a specially designed vehicle digital twin method suitable for autonomous driving scenarios. Background Art

[0002] NeRF (Neural Radiance Fields) is a deep learning-based 3D reconstruction method that can generate high-quality 3D reconstruction results, capture subtle details and realistic lighting in the real world, and has good adaptability to complex scenes. It is more detailed and realistic than traditional surface-based methods and can freely change perspectives. The emergence of NeRF has greatly promoted the development of 3D rendering and new view synthesis, and also promoted the application of digital twin technology. However, because the training of the NeRF model requires the use of multiple views of the same object, explicit priors are used during training, or pixel-perfect labels are required. In traffic scenes, objects may be occluded and have large variability in scale, which makes it difficult to truly apply NeRF in traffic vehicle scenes. Summary of the Invention

[0003] To address these issues, the present invention provides a vehicle digital twin technology suitable for autonomous driving scenarios. This technology uses a neural network to learn a standardized, object-centric representation that describes and decouples an object's shape, appearance, and pose. This information is decoded into a new target view using appearance and shape decoders and volume rendering techniques, enabling digital twin applications in traffic scenarios.

[0004] To this end, the technical solution adopted in the present invention is as follows:

[0005] A vehicle digital twin method suitable for autonomous driving scenarios includes the following steps:

[0006] S1: Acquire an RGB image of a road with multiple objects of interest RGB images are processed using 3D object detector and panoptic segmentation respectively. Perform 3D object detection and instance segmentation, extracting the 3D information of each object instance and the instance segmentation mask of the entire image , the three-dimensional information of each object instance includes the center position coordinate P, 3D bounding box β and its own rotation matrix R. The object bounding box information is used to transform each object instance from the RGB image and segmentation mask The instance image I' and instance mask Y' are obtained; the instance mask Y' of each object instance is then labeled as a class, and each pixel Label a class label Y that distinguishes foreground pixels, background pixels, and unknown pixels u , thereby generating a 2D occupancy mask Y; using the class label Y of the 2D occupancy mask Y u Set all pixel values except for the foreground pixels in the instance image I' to 0 to obtain a 2D mask image I. In camera space, for each pixel in the 2D mask image I of each object instance, generate a camera ray ρ for each pixel. Using the object instance's own 3D information, each ray ρ in camera space is mapped to the Normalized Object Coordinate Space (NOCS) to form a ray r centered on the object.

[0007] S2: For each object instance, its 2D mask image I is input into the neural network encoder Φ E In the above example, shape codes φ are generated respectively. S and appearance code φ A ;

[0008] S3. For each object instance, its shape code φ S Input a shape decoder Ψ S Then implicitly output an occupancy network σ and transform its shape code φ S and appearance code φ A Input an appearance decoder Ψ A Then implicitly output an appearance network c; at the same time, each ray r is sampled at discrete points in the normalized object coordinate space (NOCS) to obtain the 3D coordinates x of each sampling point and the direction d of the ray; the 3D coordinates x of each sampling point on each ray r are input into the occupancy network σ to obtain the volume density of each sampling point on the ray r in the normalized object coordinate space; the 3D coordinates x of each sampling point on the ray r and the direction d of the ray r are then input into the appearance network c to obtain the RGB color of each sampling point on the ray r in the normalized object coordinate space; the occupancy network σ and the appearance network c generate the neural radiation field corresponding to each object instance;

[0009] S4. For each object instance, perform volume rendering based on its corresponding neural radiation field to obtain an object image in object space;

[0010] S5. Convert all object images in the road RGB image from the object space to the world coordinate system using the object instance's own three-dimensional information to obtain a final rendered image;

[0011] S6. Transform the original camera space according to the specified camera observation angle to obtain a new camera space, and then re-render to obtain a vehicle-side scene view or a road-side scene view under the specified observation angle.

[0012] Preferably, in the 2D occupancy mask Y, the class label Y u∈{-1,0,1}, where label 1 represents foreground pixels belonging to the object instance mask, label -1 represents background pixels that do not occlude the object of interest, and label 0 represents other unknown pixels except foreground pixels and background pixels.

[0013] Preferably, when mapping the ray ρ in the camera space to the ray r in the coordinate space centered on the object, the transformation matrix used is:

[0014]

[0015] Wherein: R is the self-spin matrix of the object detected by the 3D object detector, and P is the 3D coordinate of the object detected by the 3D object detector.

[0016] Preferably, the neural network encoder Φ E The CNN feature extractor is used as the backbone network to extract the intermediate features of the 2D mask image I, and the extracted intermediate features are fed into two parallel multi-layer perceptron-based output heads to obtain the shape code φ corresponding to the 2D mask image I. S and appearance code φ A .

[0017] Preferably, the shape decoder Ψ S and appearance decoder Ψ A Both are multi-layer perceptrons consisting of 6 ResNet blocks with a hidden dimension of 128, which can decouple the shape and appearance color of objects.

[0018] Preferably, in the volume rendering process, each camera ray r needs to be volume rendered separately, and each volume rendering generates a color value of a pixel point in the object image.

[0019] Preferably, the neural network encoder Φ E , shape decoder Ψ S and appearance decoder Ψ A Joint training is performed in advance, and the training loss function is the weighted sum of the photometric loss term and the occupancy loss term;

[0020] The calculation formula of the luminosity loss term is:

[0021]

[0022] in Represents the foreground pixel set of the 2D mask image I, I u represents the pixel value of pixel u in the 2D mask image I, V(r u |σ,c) represents the pixel value of pixel u in the object image obtained by volume rendering;

[0023] The calculation formula for the occupation loss term is:

[0024]

[0025] in Represents the set of foreground and background pixels of the 2D mask image I, Y u represents the value of pixel u in the 2D occupancy mask Y, α(r u ,σ) represents the cumulative transmittance component on the camera ray corresponding to pixel u.

[0026] Preferably, in S5, for each object image, a transformation matrix is used. Project the object from its own coordinate system to the world coordinate system corresponding to the original camera position to obtain the final rendered image.

[0027] As an advantage, in S6, the transformation matrix is used First, each ray r in the camera space u Map it to the new camera space, then map it to the normalized object coordinate space (NOCS), then project the object from its own coordinate system to the world coordinate system corresponding to the new camera position, and finally re-render it, that is, render the image under the new camera position P′(X′Y′Z′) and the new viewing direction R′.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention is a vehicle digital twin technology suitable for autonomous driving scenarios, which improves the limitations of NeRF in the application of traffic vehicle scenarios. Compared with most existing methods, this method only utilizes a single view of the object during training, and does not utilize other 3D object shape priors (such as CAD models). Our method uses a 3D object detector and a panoramic segmenter to learn a standardized object-centric representation, which is independent of the object's posture and is decomposed into shape and appearance codes. These two codes are decoded into an implicit radiation field representation of the object, which can then be rendered as a new target view. Experiments show that the method of the present invention can generalize well to unseen objects, and the application effect is very good whether in the vehicle-side scenario or the road-side scenario. It can well realize the application of digital twins in autonomous driving scenarios and has good application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic flow chart of the vehicle digital twin method steps of the present invention.

[0031] Figure 2 This is a schematic diagram of the vehicle digital twin model structure of the present invention.

[0032] Figure 3 This is a rendering of a vehicle-side scenario in an embodiment of the present invention.

[0033] Figure 4 These are renderings generated from two different perspectives for another vehicle-side scene in an embodiment of the present invention.

[0034] Figure 5 This is a rendering generated from the first perspective of a road-end scene in an embodiment of the present invention.

[0035] Figure 6 This is a rendering generated from the second perspective of a roadside scene in an embodiment of the present invention.

[0036] Figure 7 This is a rendering generated from the third perspective of a roadside scene in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0038] In the present invention, a vehicle digital twin method suitable for autonomous driving scenarios is provided, which includes the following steps:

[0039] S1: Acquire an RGB image of a road with multiple objects of interest , respectively using 3D object detector and panoramic segmentation for RGB images Perform 3D object detection and instance segmentation, extracting the 3D information of each object instance (including the center position coordinates P, 3D bounding box β and its own rotation matrix R) and the instance segmentation mask of the entire image , using the object bounding box information to separate each object instance from the RGB image and segmentation mask The instance image I' and instance mask Y' are obtained; the instance mask Y' of each object instance is then labeled as a class, and each pixel Label a class label Y that distinguishes foreground pixels, background pixels, and unknown pixels u , thereby generating a 2D occupancy mask Y; using the class label Y of the 2D occupancy mask Y uIn the instance image I', all pixel values except for the foreground pixels are set to 0 to obtain a 2D mask image I. In camera space, for each pixel in the 2D mask image I of each object instance, a camera ray ρ is generated. Each ray ρ in camera space is mapped to the normalized object coordinate space (NOCS) using the object instance's own 3D information to form a ray r centered on the object.

[0040] In an embodiment of the present invention, in the process of obtaining the above instance image I' and instance mask Y', if the object is blocked by more than 1 / 2 or more than 1 / 2 of the object is not in the image, the object instance is discarded.

[0041] In the embodiment of the present invention, the class labeling process of the above 2D occupancy mask Y is as follows: since the mask of interest in the instance mask Y' has the largest proportion, the mask with the largest proportion is marked as the foreground, and other masks such as the road and sky are marked as the background, and other occluding or occluded objects are marked as unknown areas. Class label Y u ∈{-1,0,1}, where label 1 represents foreground pixels belonging to the object instance mask, label -1 represents background pixels that do not occlude the object of interest, and label 0 represents other unknown pixels except foreground pixels and background pixels.

[0042] In the embodiment of the present invention, when mapping the ray ρ in the camera space to the ray r in the normalized object coordinate space, the transformation matrix used is:

[0043]

[0044] Wherein: R is the self-spin matrix of the object detected by the 3D object detector, and P is the 3D coordinate of the object detected by the 3D object detector.

[0045] S2: For each object instance, its 2D mask image I is input into the neural network encoder Φ E In the above example, shape codes φ are generated respectively. S and appearance code φ A .

[0046] In an embodiment of the present invention, the neural network encoder Φ E The CNN feature extractor is used as the backbone network to extract the intermediate features of the 2D mask image I, and the extracted intermediate features are fed into two parallel multi-layer perceptron-based output heads to obtain the shape code φ corresponding to the 2D mask image I. S and appearance code φ A , each encoding provides compact information about the object and generalizes well.

[0047] S3. For each object instance, its shape code φ S Input a shape decoder Ψ S Then implicitly output an occupancy network σ and transform its shape code φ S and appearance code φ A Input an appearance decoder Ψ A Then implicitly output an appearance network c; at the same time, discrete point sampling is performed on each ray r in the normalized object coordinate space (NOCS) to obtain the 3D coordinate x of each sampling point and the direction d of the ray; the 3D coordinate x of each sampling point on each ray r is input into the occupancy network σ to obtain the volume density of each sampling point on the ray r in the normalized object coordinate space; the 3D coordinate x of each sampling point on the ray r and the direction d of the ray r are then input into the appearance network c to obtain the RGB color of each sampling point on the ray r in the normalized object coordinate space; the occupancy network σ and the appearance network c generate the neural radiation field corresponding to each object instance.

[0048] In an embodiment of the present invention, the shape decoder Ψ S and appearance decoder Ψ A Both are multi-layer perceptrons (MLPs) consisting of 6 ResNet blocks with a hidden dimension of 128, which can decouple the shape and appearance color of objects.

[0049] S4. For each object instance, volume rendering is performed based on its corresponding neural radiation field to obtain an object image in the object space.

[0050] In the embodiment of the present invention, during the volume rendering process, each camera ray r needs to be volume rendered separately, and each volume rendering generates a color value of a pixel in the object image.

[0051] In an embodiment of the present invention, the neural network encoder Φ E , shape decoder Ψ S and appearance decoder Ψ A Joint training is performed in advance, and the training loss function is the weighted sum of the photometric loss term and the occupancy loss term;

[0052] The calculation formula of the luminosity loss term is:

[0053]

[0054] in Represents the foreground pixel set of the 2D mask image I, I u represents the pixel value of pixel u in the 2D mask image I, V(r u |σ,c) represents the pixel value of pixel u in the object image obtained by volume rendering;

[0055] The calculation formula for the occupation loss term is:

[0056]

[0057] in Represents the set of foreground and background pixels of the 2D mask image I, Y u represents the value of pixel u in the 2D occupancy mask Y, α(r u ,σ) represents the cumulative transmittance component on the camera ray corresponding to pixel u.

[0058] S5. All object images in the road RGB image are converted from the object space to the world coordinate system using the object instance's own three-dimensional information to obtain the final rendered image.

[0059] In the embodiment of the present invention, during the above coordinate conversion process, for each object image, the conversion matrix Project the object from its own coordinate system to the world coordinate system corresponding to the original camera position to obtain the final rendered image.

[0060] S6. Transform the original camera space according to the specified camera observation angle to obtain a new camera space, and then re-render to obtain a vehicle-side scene view or a road-side scene view under the specified observation angle.

[0061] In the embodiment of the present invention, during the above coordinate conversion process, for each object instance, the transformation matrix First, each ray r in the camera space u Mapped to the new camera space, then mapped to the normalized object coordinate space (NOCS), and finally used the transformation matrix M o2w The object is projected from its own coordinate system to the world coordinate system corresponding to the new camera position, and the image at the new camera position P′(X′Y′Z′) and the new viewing direction R′ can be rendered.

[0062] The following is a specific example to demonstrate the specific implementation form and technical effect of the vehicle digital twin method shown in S1 to S5 above, which is suitable for autonomous driving scenarios.

[0063] Example

[0064] In this embodiment, a vehicle digital twin method suitable for autonomous driving scenarios is provided. The detailed process of the training phase and the inference phase is as follows: Figure 1 As shown, it includes the following steps:

[0065] Step 1: Data preprocessing to obtain data for model training and rendering.

[0066] Step 2: Transform the camera space so that it is centered on the object.

[0067] Step 3: Encode the image using a neural network encoder.

[0068] Step 4: Use the decoder to decode the shape and appearance of the object.

[0069] Step 5: Use volume rendering to perform pixel-by-pixel rendering.

[0070] Step 6: Define the loss function and train the model.

[0071] Step 7: Coordinate system conversion for practical application on the vehicle side and road side.

[0072] The specific implementation of the above steps 1 to 7 is described in detail below.

[0073] In step 1, data is preprocessed to obtain data for model training and rendering. The specific process is as follows:

[0074] Step 1.1: Given a 2D road RGB image with multiple objects of interest , using 3D object detector and panoptic segmentation to extract the 3D information of each object instance (including center position coordinates P, 3D bounding box β and self-rotation matrix R) and the instance segmentation mask of the entire image . Each object instance is separated from the RGB image and the segmentation mask using the object bounding box information. The 3D information itself obtains the position, range, and rotation information of the object in the camera space, and the segmentation mask provides pixel-by-pixel information about all object instances in the scene image.

[0075] For ease of description, define the pixels in image I as RGB color of the pixel

[0076] At the same time, the 3D coordinate space of the target is normalized. Each object instance has an associated 3D bounding box β, which can be translated, rotated and scaled into a unit cube, and the 3D points contained in the 3D bounding box β are mapped to the (centered) unit cube O∶=[-1 / 2,1 / 2] through differential isomorphism. 3 , which is called the Normalized Object Coordinate Space (NOCS). Therefore, points can be subsequently mapped from camera space to NOCS in this way.

[0077] Step 1.2: Label the instance mask Y' of each object instance and generate a 2D occupancy mask Y associated with the object instance image I'. The occupancy mask Y is Provide a class label Y u ∈{-1,0,1}. Foreground pixels, i.e., pixels belonging to the object instance mask, are assigned label 1. Background pixels, i.e., pixels that do not occlude the object of interest, are assigned label -1. Other pixels that cannot be determined whether they occlude the object are assigned label 0. The class label Y using this 2D occupancy mask Y is u Except for the foreground pixels in the instance image I', all other pixel values are set to 0 to obtain a 2D mask image I.

[0078] It should be noted that the specific forms of the above-mentioned 3D object detector and panoptic segmenter are not limited, as long as they can achieve the corresponding functions. In this embodiment, the 3D object detector can adopt CaDNN 1 、BEVDet 2 PolarFormer 3 、BEVFormer 4 、Fast-BEV 5 The panoramic segmenter can be implemented using Mask2Former 6 、Segment Anything 7 、OneFormer 8 These 3D object detectors and panoptic segmentation methods are all existing technologies. For related principles and network structures, please refer to the corresponding existing technology literature:

[0079] [1]Reading C, Harakeh A, Chae J, et al.Categorical depth distribution network for monocular 3d object detection[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:8555-8564.

[0080] [2]Huang J, Huang G, Zhu Z, et al. Bevdet: High-performance multi-camera3d object detection in bird-eye-view[J]. arXiv preprint arXiv:2112.11790,2021.

[0081] [3]Jiang Y,Zhang L,Miao Z,et al.Polarformer:Multi-camera 3d objectdetection with polar transformers[J].arXiv preprint arXiv:2206.15398,2022.

[0082] [4]Li Z,Wang W,Li H,et al.Bevformer:Learning bird’s-eye-viewrepresentation frommulti-camera images via spatiotemporal transformers[C] / / Computer Vision–ECCV 2022:17th European Conference,Tel Aviv,Israel,October23–27,2022,Proceedings,Part IX.Cham:Springer Nature Switzerland,2022:1-18.

[0083] [5]Li Y,Huang B,Chen Z,et al.Fast-BEV:AFast and Strong Bird's-EyeView Perception Baseline[J].arXiv preprint arXiv:2301.12511,2023.

[0084] [6]Cheng B,Choudhuri A,Misra I,et al.Mask2former for video instancesegmentation[J].arXiv preprint arXiv:2112.10764,2021.

[0085] [7]Kirillov A,Mintun E,Ravi N,et al.Segment anything[J].arXivpreprint arXiv:2304.02643,2023.

[0086] [8]Jain J, Li J, Chiu MT, et al. Oneformer: One transformer to rule universal image segmentation[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023:2989-2998.

[0087] In step 2, the process of transforming the camera space so that it is centered on the object is as follows:

[0088] Step 2.1: When training the model, it needs to be in a space (coordinate system) centered on the object. To train a camera centered on the object, you need to know the camera's translation matrix. In the camera coordinate system, the position of the camera can be expressed as a three-dimensional vector, namely the camera's translation vector. In order to obtain the camera's translation vector, the object's 3D coordinates P = [XYZ] can be transformed into the camera coordinate system through the object's own rotation matrix R, and then the camera's position is set to the negative value of the object in the camera coordinate system. Finally, the camera's transformation matrix is obtained: The transformation matrix can be used to transform the camera space into the space centered on the object.

[0089] Step 2.2, each 2D mask image I of a 3D scene has an associated camera, denoted by ρ. The camera ρ converts pixels Mapped into the camera space, by ρ u ∶R + →R3 represents that each ray ρ in camera space can be u Mapped to NOCS, generating a ray r centered on the object u . And r u is a reparameterized mapping ray r is called the ray centered on the object, that is, the 2D mask image I of each object instance can be mapped to the normalized object coordinate space (NOCS) by using the object's own spin matrix R to form the ray r centered on the object.

[0090] Step 2.3. At this stage, the complete input data is obtained: the mask image I of the detected object, the corresponding ray r in the NOCS obtained using the object's 3D bounding box, and the occupancy mask Y obtained using panoptic segmentation. This data can be used as samples for subsequent model training.

[0091] In step 3, the image encoding process is as follows:

[0092] Through the neural network encoder Φ E Encode an input image I describing a given object of interest into a shape code φ S and appearance code φ A , that is (φ S ,φ A )∶=Φ E (I). In this embodiment, the encoder includes a CNN feature extractor that outputs intermediate features; these intermediate features are fed into two parallel heads, responsible for generating shape codes and appearance codes, respectively. In this embodiment, both heads are multi-layer perceptrons (MLPs) with 128 hidden units.

[0093] In step 4, the process of using the decoder to decode the shape and appearance of the object is as follows:

[0094] Step 4.1. Sample a given number of points on each ray r. The 3D coordinate of each sampling point is labeled x, and the direction of the ray r is d. S Input to a shape decoder Ψ S , forming an occupied network σ, that is, σ∶=Ψ S (φ S ). The 3D coordinate x∈O is input to the occupancy network σ(x)∶O→R + , and obtain the density of the point in NOCS.

[0095] Step 4.2: Similar to the shape decoder, the appearance decoder Ψ A Input shape code and appearance code to form an appearance network c, that is, c∶=Ψ A (φ S ,φ A ). The 3D coordinate x∈O and the direction of the ray r Input to the appearance network c(x,d): Get the RGB color in NOCS at this point.

[0096] In this embodiment, the shape decoder and appearance decoder are both multi-layer perceptrons (MLPs) composed of 6 ResNet blocks with a hidden dimension of 128, which can decouple the shape and appearance color of an object.

[0097] In step 5, the volume rendering process is as follows:

[0098] The occupancy network σ and the appearance network c form a radiation field representing the object in NOCS by rendering a ray r centered on the object uTo calculate the color associated with u. Since only the object of interest is modeled, the light centered on the object is limited to the point where it intersects O. This process uses volume rendering technology, and the formula is as follows:

[0099]

[0100] where [a u ,b u ] is r u The interval intersecting with O, is r u The direction of the ray,

[0101]

[0102] On the ray r centered at the object u A fixed number of points (coordinates) are sampled and the color V(r) at the corresponding pixel u position is obtained through volume rendering. u |σ,c). Since one volume rendering produces the color value of one pixel, rendering one image requires multiple volume renderings.

[0103] In step 6, the loss function is defined and the model training process is as follows:

[0104] The training of the training model relies on two loss terms: photometric loss and occupancy loss. Given a training example It includes the image I, the occupancy mask Y, and the camera ray r centered at the object. In addition, we assume that the radiance field (σ,c) of the object is obtained using the encoder Φ E and decoder Ψ S and Ψ A Calculated from I. Finally, let Θ represent all the learnable parameters involved in the model.

[0105] Step 6.1, Luminosity loss term L RGB It is actually the mean squared error loss, which forces the output of the model to fit the given input.

[0106]

[0107] in Contains only foreground pixels.

[0108] Step 6.2: Occupancy loss L mask Panoptic segmentation is used to infer whether a pixel is foreground, background, or unknown. This information is encoded in the occupancy mask Y and used to directly supervise the cumulative transmittance component α of the volume rendering equation V. In fact, Indicates that the object is not aligned with ray r u The probability of intersection, or in other words, u may be a background pixel. Similarly, 1-α(ru ,σ) represents the probability that u is a foreground pixel. Therefore, we can directly calculate the classification loss on the cumulative transmittance:

[0109]

[0110] in Contains only foreground and background pixels.

[0111] Step 6.3. Final loss where λ ≥ 0 is a hyperparameter.

[0112] Based on the above losses The neural network encoder Φ E , shape decoder Ψ S and appearance decoder Ψ A Train together, and when the above neural network encoder Φ is completed E , shape decoder Ψ S and appearance decoder Ψ A After training, it can be used for testing and reasoning in practical applications. The object image generation process performed by these encoders, decoders, 3D object detectors and panoptic segmentation is essentially a vehicle digital twin model with a structure like Figure 2 shown.

[0113] In step 7, the actual application of the vehicle and road is carried out. The specific process is as follows:

[0114] Following steps 1 to 5 above, we obtain all object images in the object space of the road RGB image. The object image generation process is consistent with the sample method in the training phase and will not be repeated here. Each object image is transformed from the object space to the world coordinate system using the object's own rotation matrix R, and then transformed according to the specified observation angle to obtain the vehicle-side scene view or road-side scene view under the specified observation angle. The specific steps are as follows:

[0115] Step 7.1. The rendering in step 5 is performed in a space centered on the object, and only an image of the object can be obtained. To this end, the object space needs to be transformed into the world coordinate system. The transformation matrix is This transformation matrix projects the object from its own coordinate system to the world coordinate system, and can also project the object to any specified location to meet the relevant needs in the autonomous driving scenario.

[0116] Step 7.2: During the training process, it is assumed that the camera position remains unchanged at the origin. This can be understood as the observation position remaining unchanged, which makes training easier. However, in actual applications, we want to obtain images from different observation angles, that is, we need to render the view at the new position P'(X'Y'Z') and the new observation direction R'. To do this, we need to transform the camera position, using the transformation matrix Each ray ρ in camera space u Mapping to the new camera space:

[0117] M′ρ u T →ρ u '

[0118] ρ u ′ is a new camera ray set, and then rendering is performed according to step 7.1 to obtain the vehicle-side scene view or road-side scene view under the specified observation angle.

[0119] like Figure 3 As shown in FIG, the vehicle-side effect diagram when the method of the present invention is applied to a certain scene is shown, wherein the viewing angle is the same as the original diagram. Figure 4 As shown in FIG, the vehicle-side effect diagram when the method of the present invention is applied to another similar scene is shown, wherein the viewing angle is adjusted, but the digital twin of the vehicle side can still be well realized. Figure 5 、 Figure 6 and Figure 7 As shown, the road-end rendering when the method of the present invention is applied to a certain scene is displayed, wherein the observation angle is continuously adjusted, indicating that the present invention can also well realize the construction of digital twin renderings in the road-end scene.

[0120] In addition, the method of the present invention is applied to the nuScenes dataset, and the test results are as follows:

[0121] Table 1: Experimental results of car images in the nuScenes dataset

[0122] method PSNR↑ SSIM↑ LPIPS↓ FID↓ pixelNeRF 18.25 0.459 0.236 160.60 CodeNeRF 18.44 0.462 0.241 146.32 Method of the present invention 18.97 0.495 0.219 144.89

[0123] Experiments show that the method of the present invention can be well generalized to invisible objects, and has very good application effects in both vehicle-side and road-side scenarios. It can well realize the application of digital twins in autonomous driving scenarios and has good application value.

[0124] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A vehicle digital twin method suitable for autonomous driving scenarios, characterized by: The following steps are involved: S1: Acquire an RGB image of a road with multiple objects of interest RGB images are processed using 3D object detector and panoptic segmentation respectively. Perform 3D object detection and instance segmentation, extracting the 3D information of each object instance and the instance segmentation mask of the entire image The three-dimensional information of each object instance includes the center position coordinates P, 3D bounding box β and its own rotation matrix R. The object bounding box information is used to transform each object instance from the RGB image and segmentation mask Cut it out and get the instance image I' and instance mask Y'; Then, the instance mask Y' of each object instance is labeled as a class, and each pixel Label a class label Y that distinguishes foreground pixels, background pixels, and unknown pixels u , thereby generating a 2D occupancy mask Y; The class label Y using this 2D occupancy mask Y u Set all pixel values except for the foreground pixels in the instance image I' to 0 to obtain a 2D mask image I. In camera space, for each pixel in the 2D mask image I of each object instance, generate a camera ray ρ for each pixel. Using the object instance's own 3D information, each ray ρ in camera space is mapped to the Normalized Object Coordinate Space (NOCS) to form a ray r centered on the object. S2: For each object instance, its 2D mask image I is input into the neural network encoder Φ E In the above example, shape codes φ are generated respectively. S and appearance code φ A ; S3. For each object instance, its shape code φ S Input a shape decoder Ψ S Then implicitly output an occupancy network σ and transform its shape code φ S and appearance code φ A Input an appearance decoder ψ A Then implicitly output an appearance network c; at the same time, each ray r is sampled at discrete points in the normalized object coordinate space (NOCS) to obtain the 3D coordinates x of each sampling point and the direction d of the ray; the 3D coordinates x of each sampling point on each ray r are input into the occupancy network σ to obtain the volume density of each sampling point on the ray r in the normalized object coordinate space; the 3D coordinates x of each sampling point on the ray r and the direction d of the ray r are then input into the appearance network c to obtain the RGB color of each sampling point on the ray r in the normalized object coordinate space; the occupancy network v and the appearance network c generate the neural radiation field corresponding to each object instance; S4. For each object instance, perform volume rendering based on its corresponding neural radiation field to obtain an object image in object space; S5. Convert all object images in the road RGB image from the object space to the world coordinate system using the object instance's own three-dimensional information to obtain a final rendered image; S6. Transform the original camera space according to the specified camera observation angle to obtain a new camera space, and then re-render to obtain a vehicle-side scene view or a road-side scene view under the specified observation angle.

2. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: The 2D occupancy mask Y, the class label Y u ∈{-1,0,1}, where label 1 represents foreground pixels belonging to the object instance mask, label -1 represents background pixels that do not occlude the object of interest, and label 0 represents other unknown pixels except foreground pixels and background pixels.

3. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: When mapping the ray ρ in the camera space to the ray r in the coordinate space centered on the object, the transformation matrix used is: Wherein: R is the self-spin matrix of the object detected by the 3D object detector, and P is the 3D coordinate of the object detected by the 3D object detector.

4. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: The neural network encoder Φ E The CNN feature extractor is used as the backbone network to extract the intermediate features of the 2D mask image I, and the extracted intermediate features are fed into two parallel multi-layer perceptron-based output heads to obtain the shape code φ corresponding to the 2D mask image I. s and appearance code φ A .

5. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: The shape decoder Ψ S and appearance decoder Ψ A Both are multi-layer perceptrons consisting of 6 ResNet blocks with a hidden dimension of 128, which can decouple the shape and appearance color of objects.

6. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: During the volume rendering process, each camera ray r needs to be volume rendered separately, and each volume rendering generates a color value of a pixel in the object image.

7. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: The neural network encoder Φ E , shape decoder Ψ S and appearance decoder Ψ A Joint training is performed in advance, and the training loss function is the weighted sum of the photometric loss term and the occupancy loss term; The calculation formula of the luminosity loss term is: in Represents the foreground pixel set of the 2D mask image I, I u represents the pixel value of pixel u in the 2D mask image I, V(r u |σ,c) represents the pixel value of pixel u in the object image obtained by volume rendering; The calculation formula for the occupation loss term is: in Represents the set of foreground and background pixels of the 2D mask image I, Y u represents the value of pixel u in the 2D occupancy mask Y, α(r u ,σ) represents the cumulative transmittance component on the camera ray corresponding to pixel u.

8. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 1, characterized in that: In S5, for each object image, the transformation matrix Project the object from its own coordinate system to the world coordinate system corresponding to the original camera position to obtain the final rendered image.

9. The vehicle digital twin method suitable for autonomous driving scenarios according to claim 3, characterized in that: In S6, the transformation matrix First, each ray r in the camera space u Map it to the new camera space, then map it to the normalized object coordinate space (NOCS), then project the object from its own coordinate system to the world coordinate system corresponding to the new camera position, and finally re-render it, that is, render it at the new camera position P ′ (X ′ Y ′ Z ′ ) and the new viewing direction R ′ The image below.

Images