Semantic segmentation-based occupied network training method and system, and electronic equipment

Through self-supervised training of the occupancy network based on semantic segmentation, using voxel features and instance query features of on-board image data, the dependence on manual annotation information and difficulty in 3D space automation annotation in traditional methods is solved, and efficient occupation is achieved Network training.

CN118982669APending Publication Date: 2024-11-19元橡科技(北京)有限公司
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411029627.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Traditional deep learning models require a large amount of manual labeling information when training and occupying the network, which leads to hindering the development and implementation of algorithms in the field of autonomous driving, especially in 3D space.

Method used

Using a semantic segmentation method, by obtaining voxel features and instance query features of on-board image data, calculating voxel-instance probability matrix, generating an instance attribution index, and using a semantic prediction network for semantic prediction, constructing an instance-level semantic loss function , self-supervised training occupies the network and removes dependence on point cloud data.

Benefits of technology

It realizes the use of pure visual data self-supervised training to occupy the network, reduces the dependence on point cloud data, and improves the reliability and efficiency of automated labeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982669A_ABST
    Figure CN118982669A_ABST
Patent Text Reader

Abstract

The invention discloses an occupancy network training method and system based on semantic segmentation and electronic equipment, and the method comprises the steps: obtaining vehicle-mounted image data, and extracting voxel features in the vehicle-mounted image data; constructing instance query features, and calculating a voxel-instance probability matrix based on the instance query features and the voxel features; based on the voxel-instance probability matrix, selecting a plurality of voxel features, dividing the voxel features to instance query features, and generating an instance affiliation set index; when it is judged that the number of the voxel features belonging to any instance query feature is not zero, calculating a central text semantic feature based on the voxel features divided to the corresponding instance query feature; based on the voxel features, semantic prediction is carried out by using a semantic prediction network, and semantic prediction features are generated; and based on the semantic prediction features and the central text semantic features, calculating an instance-level semantic loss function, and training the occupancy network. Through the technical scheme in the application, training of the occupied network by using the pure vision data is facilitated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving, and in particular to an occupancy network training method based on semantic segmentation, an occupancy network training based on semantic segmentation, and an electronic device. Background Art

[0002] With the rapid development of autonomous driving perception technology, related research on the perception side has shown an evolution from perspective to BEV (Bird's Eye View) perspective. At the same time, in order to better express the state of objects existing in space, a finer-grained occupancy network has emerged on the BEV solution. Among them, the occupancy network based on pure visual perception of image data has become the current mainstream.

[0003] However, based on the training mode of traditional deep learning models, the occupancy network requires a large amount of manual annotation information to supervise the model training process during the training process, which greatly hinders the development and implementation of related algorithms of the occupancy network in the field of autonomous driving.

[0004] In the field of 2D images, thanks to the rise of the current visual-language model, using pre-trained large models to input text, it is possible to obtain more reliable labels on image data, which has greatly promoted the development of automatic labeling technology; however, when the relevant experience in the field of 2D images is introduced into the target labeling in 3D space, the inability to obtain accurate target depth information leads to great difficulties in the automatic labeling process, which makes point cloud data generally used for supervision in the existing model training process. For example, the method proposed in the article "POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images" to predict open vocabulary 3D semantic voxel occupancy maps based on 2D images is a typical example of supervision through point clouds.

[0005] Therefore, it is necessary to propose a model training method suitable for pure visual perception occupancy network. Summary of the invention

[0006] The purpose of this application is to learn how to train an occupancy network using pure visual data.

[0007] The technical solution of the first aspect of the present application is: a method for training an occupancy network based on semantic segmentation is provided, the method comprising: step 1: obtaining vehicle-mounted image data and extracting voxel features in the vehicle-mounted image data; step 2: constructing instance query features, and calculating the voxel-instance probability matrix based on the instance query features and the voxel features; step 3: based on the voxel-instance probability matrix, selecting a number of voxel features to be divided into instance query features, and generating an instance attribution set index; step 4: when it is determined that the number of voxel features belonging to any instance query feature is not 0, calculating the central text semantic features based on the voxel features divided into the corresponding instance query features; step 5: based on the voxel features, using the semantic prediction network, performing semantic prediction and generating semantic prediction features; step 6: calculating the instance-level semantic loss function based on the semantic prediction features and the central text semantic features, and training the occupancy network using the instance-level semantic loss function.

[0008] In any of the above technical solutions, further, semantic prediction features are generated, specifically including: step 51: based on the occupancy attribute of the voxel feature, predicting the occupancy probability of the voxel; step 52: arranging the voxel features in descending order according to the occupancy probability to generate a sequence set; step 53: selecting the first K voxel features in the sequence set, and using the semantic prediction network to generate semantic prediction features.

[0009] In any of the above technical solutions, the method further includes: Step 71: extracting text semantic features from vehicle-mounted image data; Step 72: obtaining the spatial coordinates corresponding to each voxel feature in the sequence set, and projecting the obtained spatial coordinates to the image plane to obtain 2D voxel coordinates; Step 73: taking the 2D voxel coordinates as the sampling center, based on the text semantic features, obtaining semantic supervision features through feature interpolation; Step 74: based on the semantic prediction features and the semantic supervision features, constructing a voxel-level semantic loss function in the form of L2 norm; Step 75: using the sum of the instance-level semantic loss function and the voxel-level semantic loss function to train the occupancy network.

[0010] In any of the above technical solutions, further, in step 1, voxel features in the vehicle-mounted image data are extracted, specifically including: step 11: extracting image fusion features in the vehicle-mounted image data; step 12: determining the homogeneous coordinates of the voxel center points based on the set occupancy network perception range and division granularity; step 13: performing a depth transformation on the homogeneous coordinates of the voxel center points based on the internal and external parameters of the vehicle-mounted camera, and using the depth transformation results and image features to generate intermediate voxel features; step 14: using an iterative method, using the intermediate voxel features, updating the homogeneous coordinates of the voxel center points, and re-executing step 13 based on the updated homogeneous coordinates of the voxel center points until the iteration conditions are met, and the intermediate voxel features calculated in the last iteration are recorded as the voxel features of the vehicle-mounted image data.

[0011] In any of the above technical solutions, further, the calculation formula of the central text semantic feature corresponding to the nth instance query feature is:

[0012]

[0013] Where instance_id n Query the instance attribute set index corresponding to the feature of the nth instance, |instance_id n | indicates the query feature belonging to the nth instance The number of voxels, Indicates the corresponding instance query feature The index-th voxel feature in the nth instance belongs to the set index instance_id n The ordinal number of the voxel feature in .

[0014] In any of the above technical solutions, the method further includes: step 81: performing 3D modeling of the occupied network space, and aligning the Gaussian models corresponding to two adjacent frames of vehicle-mounted image data, rendering, and generating a rendered image; step 82: constructing an image rendering loss function based on the rendered image and the current frame of vehicle-mounted image data, and using the image rendering loss function to train the occupancy network.

[0015] The technical solution of the second aspect of the present application is: providing an occupancy network training system based on semantic segmentation, the system comprising: a voxel feature unit, the voxel feature unit being configured to obtain vehicle-mounted image data and extract voxel features in the vehicle-mounted image data; a probability calculation unit, the probability calculation unit being configured to construct an instance query feature, and based on the instance query feature and the voxel feature, calculating a voxel-instance probability matrix; an index unit, the index unit being configured to select a number of voxel features based on the voxel-instance probability matrix and divide them into the instance query feature, and generate an instance attribution set index; a feature calculation unit, the feature calculation unit being configured to calculate a central text semantic feature based on the voxel features divided into the corresponding instance query feature when the number of voxel features determined to belong to any instance query feature is not 0; a prediction unit, the prediction unit being configured to perform semantic prediction based on the voxel feature and using the semantic prediction network to generate a semantic prediction feature; a training unit, the training unit being configured to calculate an instance-level semantic loss function based on the semantic prediction feature and the central text semantic feature, and train the occupancy network using the instance-level semantic loss function.

[0016] In any of the above technical solutions, further, the system also includes: a semantic feature unit, which is configured to extract text semantic features in the vehicle-mounted image data; a projection unit, which is configured to obtain the spatial coordinates corresponding to each voxel feature in the sequence set, and project the obtained spatial coordinates to the image plane to obtain 2D voxel coordinates; a supervision signal unit, which is configured to use the 2D voxel coordinates as the sampling center, and based on the text semantic features, obtain semantic supervision features through feature interpolation; wherein the training unit is also configured to: construct a voxel-level semantic loss function based on the semantic prediction feature and the semantic supervision feature in the form of L2 norm; and train the occupancy network using the sum of the instance-level semantic loss function and the voxel-level semantic loss function.

[0017] In any of the above technical solutions, further, the training unit is also configured to: perform 3D modeling of the occupied network space, and align the Gaussian models corresponding to two adjacent frames of vehicle-mounted image data, and then render to generate a rendered image; based on the rendered image and the current frame of vehicle-mounted image data, construct an image rendering loss function, and use the image rendering loss function to train the occupancy network.

[0018] The technical solution of the third aspect of the present application is: an electronic device is provided, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the semantic segmentation-based occupancy network training method of any one of the above-mentioned first aspect technical solutions when executing the executable instructions.

[0019] The beneficial effects of this application are:

[0020] The technical solution in the present application calculates the voxel-instance probability matrix by multiplying the voxel features corresponding to the vehicle-mounted image data with the constructed instance query features, and then obtains the instance belonging set index and the corresponding central text semantic features, combines the semantic features with the occupancy network, and then constructs the corresponding loss between the semantic features predicted by the model and the semantic features generated by the visual-language model. Through self-supervision, the occupancy network is trained using pure visual data, eliminating the need for point cloud data in the occupancy network training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0022] Figure 1 is a schematic flow chart of an occupancy network training method based on semantic segmentation according to an embodiment of the present application;

[0023] Figure 2It is a schematic block diagram of an occupancy network training system based on semantic segmentation according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0025] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0026] Embodiment 1:

[0027] like Figure 1 As shown, this embodiment provides an occupancy network training method based on semantic segmentation, the method comprising:

[0028] Step 1: Obtain vehicle image data and extract voxel features from the vehicle image data

[0029]

[0030] Specifically, taking the vehicle-mounted surround view camera as an example, the vehicle-mounted images collected by multiple cameras at the same time t are obtained and recorded as a frame of vehicle-mounted image data Where V is the number of cameras.

[0031] For voxel features, after determining the relevant structural parameters of the voxel network (such as perception range and division granularity), the n-layer voxel encoding network VoxelEncoder based on Transformer can be used to generate the image features. Perform feature interaction to obtain the voxel features of the entire prediction space corresponding to the vehicle image data.

[0032] Step 2: Construct instance query feature Q 2 , and based on the instance query features and voxel features, calculate the voxel-instance probability matrix P voxel→instance ;

[0033] Specifically, the feature vector is initialized according to the preset dimension. The initialization method can refer to the initialization method based on normal distribution to generate N query features, which are recorded as Q 0 .

[0034] Afterwards, based on the self-attention mechanism in deep learning, the self-attention operation is performed on the N query features to update the query features, which is recorded as Q 1. Use a network with prediction function such as FFN network to predict the updated query feature Q 1 The sampling points are projected onto the image plane where the vehicle-mounted image data is located.

[0035] Then, according to the coordinates of the sampling points obtained by projection and based on the features of its neighboring points, binary interpolation is performed to obtain the image sampling features corresponding to each sampling point, and the image sampling features and query features are fused to obtain the instance query features.

[0036] In this embodiment, the instance query feature Q 2 And the extracted voxel features The product operation is performed to obtain the probability that each voxel feature belongs to each instance query feature. Therefore, the calculation formula of the voxel-instance probability matrix is:

[0037]

[0038] In the formula, Query feature Q for the instance 2 inversion.

[0039] Step 3: Based on the voxel-instance probability matrix, select several voxel features and classify them into instance query features to generate the instance belonging set index instance_id;

[0040] Specifically, after obtaining the voxel-instance probability matrix, several voxel features can be selected based on the size of the probability value, and then the argmax function can be used to obtain the instance attribution set index instance_id of each selected voxel feature belonging to the corresponding instance query feature, that is, to divide the voxel feature into the corresponding instance query feature. The corresponding calculation formula is:

[0041]

[0042] Where K is the number of selected voxel features.

[0043] Step 4: When it is determined that the number of voxel features belonging to any instance query feature is not 0, the central text semantic feature is calculated based on the voxel features assigned to the corresponding instance query feature

[0044] The number of voxel features belonging to any instance query feature is determined by the instance belonging set index; the central text semantic feature corresponding to the nth (n=1, 2, ..., N) instance query feature is The calculation formula is:

[0045]

[0046] Where instance_id n Query the instance attribute set index corresponding to the feature of the nth instance, |instance_id n | indicates the query feature belonging to the nth instance The number of voxels, Indicates the corresponding instance query feature The index-th voxel feature in the nth instance belongs to the set index instance_id n The ordinal number of the voxel feature in .

[0047] Step 5: Based on the voxel features, use the semantic prediction network to perform semantic prediction and generate the semantic prediction feature F seg_pred ;

[0048] Specifically, since the above voxel features are obtained based on the vehicle image data, the semantic prediction network can be used to perform semantic prediction and generate semantic prediction features. The corresponding calculation formula is:

[0049]

[0050] In the formula, It is a semantic prediction network.

[0051] Step 6: Predict feature F based on semantics seg_pred , semantic features of the central text Calculate the instance-level semantic loss function and use it to train the occupancy network. The calculation formula of the instance-level semantic loss function is:

[0052]

[0053] Wherein, idx is the instance number to which the i-th voxel belongs, which is obtained through the instance set index instance_id, that is, idx = instance_id[i].

[0054] In any of the above embodiments, further, in order to ensure the stability of model training, supervision is performed from the perspective of image consistency, and the voxels in the occupancy network are rendered using a Gaussian model. Therefore, the method further includes:

[0055] Step 81: Perform 3D modeling on the occupied network space, align the Gaussian models corresponding to two adjacent frames of vehicle-mounted image data, and then render to generate a rendered image;

[0056] Step 82: Based on the rendered image and the vehicle-mounted image data of the current frame, an image rendering loss function is constructed, and the occupancy network is trained using the image rendering loss function.

[0057] Specifically, the occupancy network space can be modeled in 3D based on the 3D Gaussian idea to obtain the motion properties of the Gaussian model, and then the Gaussian model corresponding to the current frame image and the Gaussian model after the previous frame image is aligned with the current frame image are projected and rasterized to obtain a rendered image. Combined with the corresponding current frame image data I t , construct the image rendering loss function, and the corresponding calculation formula is:

[0058]

[0059] Where, L render is the image rendering loss function, λ sim is a preset parameter, which can be set to 0.7, L1() is the image consistency loss, L D-ssi () is the structural consistency loss.

[0060] Then, using the instance-level semantic loss function L instance-se , image rendering loss function L render The occupancy network is trained by averaging and averaging. The specific training process will not be described in detail.

[0061] In any of the above embodiments, further, in generating the semantic prediction feature F seg_pred The process also includes:

[0062] Step 51: Voxel-based features The occupancy attribute of the voxel is predicted and the occupancy probability occ_pred is obtained. The corresponding calculation formula is:

[0063]

[0064] Wherein, nn.Linear() is the occupancy probability prediction function, that is, the voxel feature is input into the linear layer network for binary classification prediction to obtain the occupancy probability of the voxel, and then combined with the set occupancy threshold, such as 0.4. When the predicted occupancy probability is greater than or equal to the occupancy threshold, the occupancy attribute corresponding to the voxel feature is "occupied", otherwise, the occupancy attribute is "unoccupied".

[0065] It should be noted that the dimension of the occupancy probability occ_pred is It can be compressed from the first three dimensions and converted into M×C, where M is the number of voxels obtained by dividing the space and C is the dimension of the voxel feature.

[0066] Step 52: Arrange the voxel features in descending order according to the occupancy probability to generate a sequence set Valid_index;

[0067] Step 53: Select the first K voxel features in the sequence set Valid_index, and use the semantic prediction network to generate semantic prediction features. The value of K can be set according to demand, such as 8000.

[0068] In any of the above embodiments, further, the method also includes:

[0069] Step 71: extracting text semantic features from the vehicle-mounted image data;

[0070] Specifically, a semantic network model (such as MaskCLIP) can be used to extract vehicle image data I t Semantic features of text in As follows:

[0071]

[0072] Step 72: Obtain the spatial coordinates corresponding to each voxel feature in the sequence set Valid_index, and project the obtained spatial coordinates onto the image plane to obtain 2D voxel coordinates.

[0073] Specifically, each voxel feature in the sequence set Valid_index can correspond to a voxel, and each voxel has a corresponding spatial coordinate (x i ,y i ,z i ), therefore, the spatial coordinates (x i ,y i ,z i ) is projected onto the image plane of the onboard surround view camera to obtain the 2D voxel coordinates, denoted as Voxel 2d .

[0074] Step 73: 2D Voxel Coordinates 2d is the sampling center, based on the text semantic features By means of feature interpolation (binary interpolation), semantic supervision features (features) are obtained, denoted as

[0075] Step 74: Predict feature F based on semantics seg_pred and semantic supervision features The voxel-level semantic loss function is constructed using the L2 norm form, and the corresponding calculation formula is:

[0076]

[0077] Where C is the semantic prediction feature F seg_pred and semantic supervision features feature dimension.

[0078] Step 75: Train the occupancy network using the sum of the instance-level semantic loss function and the voxel-level semantic loss function.

[0079] In any of the above embodiments, further, the process of extracting voxel features from the vehicle-mounted image data specifically includes:

[0080] Step 11: Extract image fusion features from vehicle-mounted image data;

[0081] Specifically, after obtaining the vehicle image data, the feature network (such as ResNet50, ResNet101, ViT, etc.) can be used to extract features to obtain multi-scale image features, and then the feature fusion network (such as FPN) can be used to fuse features to obtain multi-scale fused image fusion features. The formula is as follows:

[0082]

[0083] In the formula, is the feature network, is a multi-scale image feature, is the image fusion feature, where image fusion feature The dimension is V×h×w×C, where V is the number of onboard surround view cameras; C is the feature dimension, which can be set manually, such as 256; h and w are the width and height of the image feature, respectively.

[0084] Step 12: Determine the homogeneous coordinates of the voxel center point based on the set occupancy network perception range and division granularity;

[0085] Specifically, the sensing range of the occupied network can be set according to actual needs [-X, -Y, Z bottom ,X,Y,Z top ], and divide the granularity occ_size, for example, it can be set to [-40,-40,-1.0,40,40,5.4] and 0.4, the unit is meter (m). Establish a reference coordinate system with the vehicle as the origin, and the corresponding coordinates of the center point of each voxel are:

[0086]

[0087] Therefore, the homogenized coordinates of each voxel center point are sample_pt l =(x i ,y i ,z i ,1), where i is the spatial arrangement number of the voxel after the perception space is divided, and l is the number of iterative updates of the spatial coordinates corresponding to the voxel feature.

[0088] Step 13: Based on the internal and external parameters of the vehicle-mounted camera, perform depth transformation on the homogeneous coordinates of the voxel center point, and use the depth transformation result and image features to generate intermediate voxel features;

[0089] Specifically, after obtaining the homogenized coordinates sample_pt of the center point of each voxel l =(x i ,y i ,z i ,1), we can use the internal and external parameters of the jth camera in the vehicle-mounted surround view camera to project it onto the image plane of V cameras to obtain the depth of the corresponding projection point. The corresponding calculation formula is:

[0090] d ij (u ij ,v ij ,1)=K j @[R; t] j @sample_pt l

[0091] Where, d ij () is the projection point depth of the i-th voxel point on the image plane corresponding to the j-th camera, (u ij ,v ij ,1) is the homogeneous coordinates of the transformed projection point, @ is the product operation, K j is the internal parameter of the jth camera, [R; t] j is the external parameter of the jth camera.

[0092] Then the sampling function grid_sample can be used to extract the features corresponding to each voxel, which are recorded as intermediate voxel features. The corresponding calculation formula is:

[0093]

[0094] Step 14: Adopt an iterative approach and use the intermediate voxel features to update the homogeneous coordinates of the voxel center points, and re-execute step 13 based on the updated homogeneous coordinates of the voxel center points until the iteration condition is met, and record the intermediate voxel features calculated in the last iteration as the voxel features of the vehicle-mounted image data.

[0095] Specifically, after obtaining the intermediate voxel features, the offset prediction network of the lth layer can be used Predicted coordinate offset value (Δx l ,Δy l ,Δz l ), and then update the homogeneous coordinates of the voxel center point. The corresponding calculation formula is:

[0096]

[0097] sample_pt l =sample_pt l-1 +(Δx l ,y l ,Δ l )

[0098] Where, sample_pt l is the offset prediction network based on the lth layer Predicted coordinate offset value (Δx l ,Δy l ,Δz l ) and the homogeneous coordinates of the voxel center point updated for the lth time, sample_pt l-1 is the homogeneous coordinates of the voxel center point updated for the l-1th time.

[0099] Then, based on the updated homogeneous coordinates of the voxel center point, step 13 is re-executed, and after n layers of feature iteration operations, the final voxel features of the vehicle-mounted image data are obtained.

[0100] Embodiment 2:

[0101] like Figure 2 As shown, this embodiment provides an occupancy network training system 100 based on semantic segmentation, and the system 100 includes: a voxel feature unit 10, the voxel feature unit 10 is configured to obtain vehicle image data and extract voxel features in the vehicle image data; a probability calculation unit 20, the probability calculation unit 20 is configured to construct an instance query feature, and calculate a voxel-instance probability matrix based on the instance query feature and the voxel feature; an index unit 30, the index unit 30 is configured to select a number of voxel features based on the voxel-instance probability matrix, divide them into instance query features, and generate an instance attribution set index. Introduction; a feature calculation unit 40, the feature calculation unit 40 is configured to calculate the central text semantic feature based on the voxel features assigned to the corresponding instance query feature when the number of voxel features determined to belong to any instance query feature is not 0; a prediction unit 50, the prediction unit 50 is configured to perform semantic prediction based on the voxel features and use a semantic prediction network to generate semantic prediction features; a training unit 60, the training unit 60 is configured to calculate an instance-level semantic loss function based on the semantic prediction features and the central text semantic features, and use the instance-level semantic loss function to train the occupancy network.

[0102] In any of the above embodiments, further, the system 100 further includes:

[0103] A semantic feature unit 70, the semantic feature unit 70 is configured to extract text semantic features in the vehicle-mounted image data;

[0104] Specifically, a semantic network model (such as MaskCLIP) can be used to extract vehicle image data I t Semantic features of text in As follows:

[0105]

[0106] A projection unit 80, the projection unit 80 is configured to obtain the spatial coordinates corresponding to each voxel feature in the sequence set, and project the obtained spatial coordinates onto the image plane to obtain 2D voxel coordinates;

[0107] Specifically, each voxel feature in the sequence set Valid_index can correspond to a voxel, and each voxel has a corresponding spatial coordinate (x i ,y i ,z i ), therefore, the spatial coordinates (x i ,y i ,z i ) is projected onto the image plane of the onboard surround view camera to obtain the 2D voxel coordinates, denoted as Voxel 2d .

[0108] The supervisory signal unit 90 is configured to obtain semantic supervisory features by feature interpolation based on text semantic features, taking the 2D voxel coordinates as the sampling center;

[0109] The training unit 60 is further configured to: construct a voxel-level semantic loss function based on the semantic prediction feature and the semantic supervision feature in the form of L2 norm; and train the occupancy network using the sum of the instance-level semantic loss function and the voxel-level semantic loss function. The voxel-level semantic loss function has the following calculation formula:

[0110]

[0111] Where C is the semantic prediction feature F seg_pred and semantic supervision features feature dimension.

[0112] In any of the above embodiments, further, the training unit 60 is also configured to: perform 3D modeling of the occupied network space, and align the Gaussian models corresponding to two adjacent frames of vehicle-mounted image data, and then render to generate a rendered image; based on the rendered image and the current frame of vehicle-mounted image data, construct an image rendering loss function, and use the image rendering loss function to train the occupancy network.

[0113] Specifically, the occupancy network space can be modeled in 3D based on the 3D Gaussian idea to obtain the motion properties of the Gaussian model, and then the Gaussian model corresponding to the current frame image and the Gaussian model after the previous frame image is aligned with the current frame image are projected and rasterized to obtain a rendered image. Combined with the corresponding current frame image data I t , construct the image rendering loss function, and the corresponding calculation formula is:

[0114]

[0115] Where, L render is the image rendering loss function, λ sim is a preset parameter, which can be set to 0.7, L1() is the image consistency loss, L D-s () is the structural consistency loss.

[0116] Then, using the instance-level semantic loss function L instance-semantic , image rendering loss function L render The occupancy network is trained by averaging and averaging. The specific training process will not be described in detail.

[0117] Embodiment 3:

[0118] This embodiment provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the semantic segmentation-based occupancy network training method of any one of the above-mentioned embodiments 1 when executing the executable instructions.

[0119] It should be noted that the number of processors may be one or more. At the same time, the electronic device in the embodiment of the present application may also include an input device and an output device. The processor, memory, input device, and output device may be connected via a bus or in other ways, which are not specifically limited here.

[0120] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0121] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0122] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0123] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A method for training an occupancy network based on semantic segmentation, characterized in that: The method comprises: Step 1: Obtain vehicle image data and extract voxel features in the vehicle image data; Step 2: Construct instance query features, and calculate the voxel-instance probability matrix based on the instance query features and voxel features; Step 3: Based on the voxel-instance probability matrix, select several voxel features and classify them into instance query features to generate instance belonging set index; Step 4: when it is determined that the number of voxel features belonging to any instance query feature is not 0, the central text semantic feature is calculated based on the voxel features assigned to the corresponding instance query feature; Step 5: Based on the voxel features, use the semantic prediction network to perform semantic prediction and generate semantic prediction features; Step 6: Based on the semantic prediction features and the central text semantic features, the instance-level semantic loss function is calculated, and the occupancy network is trained using the instance-level semantic loss function.

2. The occupancy network training method based on semantic segmentation according to claim 1, characterized in that: The generating of semantic prediction features specifically includes: Step 51: based on the occupancy attribute of the voxel feature, predict the occupancy probability of the voxel; Step 52: Arrange the voxel features in descending order according to the occupancy probability to generate a sequence set; Step 53: Select the first K voxel features in the sequence set, and use the semantic prediction network to generate semantic prediction features.

3. The occupancy network training method based on semantic segmentation according to claim 2, characterized in that: The method further comprises: Step 71: extracting text semantic features from the vehicle-mounted image data; Step 72: obtaining the spatial coordinates corresponding to each voxel feature in the sequence set, and projecting the obtained spatial coordinates onto the image plane to obtain 2D voxel coordinates; Step 73: Taking the 2D voxel coordinates as the sampling center, based on the text semantic features, obtain the semantic supervision features by feature interpolation; Step 74: constructing a voxel-level semantic loss function based on the semantic prediction features and the semantic supervision features in the form of L2 norm; Step 75: Train the occupancy network using the sum of the instance-level semantic loss function and the voxel-level semantic loss function.

4. The occupancy network training method based on semantic segmentation according to any one of claims 1 to 3, characterized in that: In step 1, extracting voxel features from the vehicle-mounted image data specifically includes: Step 11: Extract image fusion features from vehicle-mounted image data; Step 12: Determine the homogeneous coordinates of the voxel center point based on the set occupancy network perception range and division granularity; Step 13: Based on the internal and external parameters of the vehicle-mounted camera, perform depth transformation on the homogeneous coordinates of the voxel center point, and use the depth transformation result and image features to generate intermediate voxel features; Step 14: Adopt an iterative approach and use the intermediate voxel features to update the homogeneous coordinates of the voxel center points, and re-execute step 13 based on the updated homogeneous coordinates of the voxel center points until the iteration condition is met, and record the intermediate voxel features calculated in the last iteration as the voxel features of the vehicle-mounted image data.

5. The occupancy network training method based on semantic segmentation according to any one of claims 1 to 3, characterized in that: The calculation formula of the central text semantic feature corresponding to the nth instance query feature is: Where instance_id n Query the instance attribute set index corresponding to the feature of the nth instance, |instance_id n | indicates the query feature belonging to the nth instance The number of voxels, Indicates the corresponding instance query feature The index-th voxel feature in the nth instance belongs to the set index instance_id n The ordinal number of the voxel feature in .

6. The occupancy network training method based on semantic segmentation according to claim 1, characterized in that: The method further comprises: Step 81: Perform 3D modeling on the occupied network space, align the Gaussian models corresponding to two adjacent frames of vehicle-mounted image data, and then render to generate a rendered image; Step 82: construct an image rendering loss function based on the rendered image and the current frame vehicle image data, and use the image rendering loss function to train the occupancy network.

7. An occupancy network training system based on semantic segmentation, characterized in that: The system comprises: A voxel feature unit, wherein the voxel feature unit is configured to acquire vehicle-mounted image data and extract voxel features in the vehicle-mounted image data; A probability calculation unit, wherein the probability calculation unit is configured to construct an instance query feature and calculate a voxel-instance probability matrix based on the instance query feature and the voxel feature; An indexing unit, wherein the indexing unit is configured to select a number of voxel features and classify them into instance query features based on a voxel-instance probability matrix, and generate an instance belonging set index; A feature calculation unit, wherein the feature calculation unit is configured to calculate a central text semantic feature based on the voxel features assigned to the corresponding instance query feature when it is determined that the number of voxel features belonging to any instance query feature is not zero; A prediction unit, wherein the prediction unit is configured to perform semantic prediction based on voxel features and utilize a semantic prediction network to generate semantic prediction features; A training unit is configured to calculate an instance-level semantic loss function based on semantic prediction features and central text semantic features, and train an occupancy network using the instance-level semantic loss function.

8. The occupancy network training system based on semantic segmentation according to claim 7, characterized in that: The system further comprises: A semantic feature unit, wherein the semantic feature unit is configured to extract text semantic features in the vehicle-mounted image data; A projection unit, wherein the projection unit is configured to obtain the spatial coordinates corresponding to each voxel feature in the sequence set, and project the obtained spatial coordinates onto the image plane to obtain 2D voxel coordinates; A supervisory signal unit, wherein the supervisory signal unit is configured to obtain a semantic supervisory feature by feature interpolation based on a 2D voxel coordinate as a sampling center and based on a text semantic feature; The training unit is further configured to: construct a voxel-level semantic loss function in the form of L2 norm based on the semantic prediction feature and the semantic supervision feature; The occupancy network is trained using the sum of the instance-level semantic loss function and the voxel-level semantic loss function.

9. The occupancy network training system based on semantic segmentation according to claim 7, characterized in that: The training unit is further configured to: Perform 3D modeling on the occupied network space, align the Gaussian models corresponding to two adjacent frames of vehicle-mounted image data, and then render to generate a rendered image; Based on the rendered image and the current frame vehicle-mounted image data, an image rendering loss function is constructed, and the occupancy network is trained using the image rendering loss function.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the occupancy network training method based on semantic segmentation as described in any one of claims 1-6 above when executing the executable instructions.

Citation Information

Patent Citations

  • Point cloud instance segmentation method based on comparative language image pre-training technology

    CN116152267A

  • Network information relation extraction method, relation extraction device and medium

    CN116561322A

  • Multi-task perception three-dimensional scene graph element segmentation and relation reasoning method

    CN117409209A

  • Instance-aware monocular semantic scene completion method, medium and equipment

    CN117422629A