Three-dimensional target recognition model training method, three-dimensional target recognition model application method, three-dimensional target recognition model equipment and medium

Abstract

The invention discloses a three-dimensional target recognition model training method, an application method, equipment and a medium, and relates to the field of intelligent vehicle environment and target perception, and the training method comprises the steps: obtaining a training set; the training set comprises a plurality of multi-view images and a target identification tag corresponding to each multi-view image; training the target recognition model by adopting the training set to obtain a trained target recognition model; the target recognition model comprises a multi-scale feature extraction module, a depth supervision branch, a 3d anchor point supervision branch and a segmentation supervision branch. The method improves the accuracy of 3d target recognition.

Description

Technical Field

[0001] This application relates to the field of intelligent vehicle environment and target perception, and in particular to a 3D object detection model training method, application method, device and medium. Background Technology

[0002] 3D object detection is a crucial component of autonomous driving systems. Compared to LiDAR-based perception schemes, purely visual perception schemes based on multi-view cameras offer lower deployment costs and allow for complementarity among cameras when observing the physical world, leading to their widespread adoption in 3D object detection. Existing multi-view 3D object detection methods can be broadly categorized into Bird's Eye View (BEV)-based methods and sparse model-based methods. BEV-based methods transform the multi-view image features into a unified BEV space, using densely meshed BEV features as a scene representation, and outputting the detected targets through a decoder. However, BEV-based methods represent the surrounding vehicle scene as dense BEV features, employing attention mechanisms to achieve spatiotemporal fusion of these dense BEV features, resulting in high computational complexity and poor real-time performance. Sparse model-based methods, on the other hand, predefine a set of anchors in 3D space and sample sparse features for each anchor, refining the anchors using these sampled features. Because they do not use dense feature representations, sparse model-based methods offer higher computational efficiency. However, in the above solutions, autonomous vehicles have low perception accuracy and poor spatial understanding in complex urban scenarios. Summary of the Invention

[0003] The purpose of this application is to provide a training method, application method, device and medium for a 3D target recognition model, which can improve the accuracy of 3D target recognition and solve the problems of low perception accuracy and poor spatial understanding ability of autonomous vehicles in complex urban scenes.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] Firstly, this application provides a method for training a three-dimensional target recognition model, including:

[0006] Obtain the training set; the training set includes several multi-view images and the target recognition label corresponding to each multi-view image;

[0007] The target recognition model is trained using a training set to obtain a trained target recognition model; the target recognition model includes a multi-scale feature extraction module, a depth supervision branch, a 3D anchor point supervision branch, and a segmentation supervision branch;

[0008] Among them, the multi-scale feature extraction module is used to extract features at multiple scales from multi-view images to obtain multi-scale features;

[0009] The deep supervision branch is used to perform pixel-by-pixel prediction of multi-scale features to obtain the prediction depth of the features in 3D space.

[0010] The 3D anchor point supervision branch includes a first anchor point refinement layer and a second anchor point refinement layer. The first anchor point refinement layer is used to refine the anchor points of the current scene multiple times based on multi-scale features and the anchor point recognition results of the previous time step, so as to obtain the cyclic refinement results. The cyclic refinement results are the anchor point prediction results of all targets in the target scene.

[0011] The segmentation supervision branch includes a region of interest (ROI) prediction head, a mask prediction head, and an alignment and fusion module. The ROI prediction head predicts the pixel coordinate range of the target corresponding to each anchor point in the multi-view image based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer. The mask prediction head predicts the shape mask of the target corresponding to each anchor point based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer. The alignment and fusion module predicts the instance segmentation features of the target corresponding to each anchor point based on the outputs of the ROI prediction head and the mask prediction head.

[0012] The second anchor point refinement layer is used to fuse the instance segmentation features of all anchor points into the cyclic refinement result to obtain the target recognition prediction result.

[0013] Secondly, this application provides a method for applying a three-dimensional target recognition model, including:

[0014] Acquire multi-view images of the target;

[0015] The target multi-view image is input into the trained target recognition model to obtain the target recognition prediction result corresponding to the target multi-view image; the trained target recognition model is a model trained using the above-mentioned three-dimensional target recognition model training method.

[0016] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described three-dimensional target recognition model training method or the above-described three-dimensional target recognition model application method.

[0017] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described three-dimensional target recognition model training method or the above-described three-dimensional target recognition model application method.

[0018] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0019] This application provides a training method, application method, device, and medium for a 3D target recognition model. It adds three parts to the traditional sparse detection framework: region of interest (ROI) prediction under multi-view images, instance segmentation prediction, and alignment and fusion of 2D segmentation features and 3D anchor masks. ① The ROI prediction head predicts the position of the target's segmentation mask in the multi-view images in the form of a 2D bounding box (i.e., the pixel coordinate range of the target in the multi-view images corresponding to each anchor point); ② The instance segmentation prediction module (i.e., the mask prediction head) uses instance features sampled from the image to predict the shape mask of the target corresponding to the anchor point in the multi-view images; ③ The feature alignment and fusion module aligns the 2D instance segmentation with the 3D anchor points in the target recognition model, extracts the boundaries of the 2D segmentation, further encodes the segmentation boundaries, and finally fuses the segmentation boundary encoding with the matched anchor mask to obtain the instance segmentation feature of the target corresponding to each anchor point. Through ROI prediction and instance segmentation prediction, the model can introduce auxiliary supervision based on segmentation annotations during training, which helps the model extract target features more accurately. By aligning and fusing features, 2D segmentation features are used to further refine the 3D spatial features of the target. These three parts improve the accuracy of 3D target recognition and solve the problems of low perception accuracy and poor spatial understanding ability of autonomous vehicles in complex urban scenes. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating the basic principle of the existing scheme 1.

[0022] Figure 2 This is a schematic diagram illustrating the basic principle of the existing Scheme 2.

[0023] Figure 3 This is an application environment diagram of a three-dimensional target recognition model training method in Embodiment 1 of this application.

[0024] Figure 4 This is a flowchart illustrating a three-dimensional target recognition model training method provided in Embodiment 1 of this application.

[0025] Figure 5 This is a schematic diagram of the target recognition model provided in Embodiment 1 of this application.

[0026] Figure 6 This is a flowchart illustrating a three-dimensional target recognition model application method provided in Embodiment 2 of this application.

[0027] Figure 7 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Existing Solution 1: Road Environment Perception Method and Perception Model Training Method. Step S201: Using the feature extraction network of the first perception model, image features are obtained from the vehicle's surround view image; wherein, the surround view image is obtained from multiple image acquisition devices at different angles; Step S202: Using the encoding network of the first perception model, a bird's-eye view feature map is determined based on the image features and the bird's-eye view query vector; Step S203: Using multiple head networks of the first perception model, the output results of multiple perception tasks are determined based on the bird's-eye view feature map. Its basic principle is as follows... Figure 1 As shown. ① Scheme 1 uses a bird's-eye view (BEV)-based environmental representation for target recognition in 3D space. However, the dense, gridded BEV features consume a lot of memory and have a high computational cost, resulting in poor real-time performance. ② This scheme only uses bounding box annotations in 3D space for supervised training, which makes it difficult to fully utilize the complex object features of the original sensor data in 2D space, resulting in low perception accuracy.

[0030] Existing Solution 2: A Multi-Task Visual Environment Perception Method for Intelligent Driving. Step 1: Data Collection and Processing. The BDD100K autonomous driving dataset is cleaned and processed, and divided into training, validation, and test sets in a 7:2:1 ratio. Each image in both the training and validation sets is labeled with vehicle category and a lane line mask image is generated. Step 2: Design of the Multi-Task Visual Environment Perception Algorithm MTP-Net. Using CSP-Darknet53 as the shared backbone network, vehicle detection and lane line segmentation branches are set up. Combined with the design of a weighted loss function, vehicle detection and lane line segmentation tasks can be completed simultaneously. Step 3: MTP-Net Model Training. Based on the network structure and weighted loss function design in Step 2, and combined with training hyperparameter tuning, the MTP-Net algorithm from Step 2 is trained to obtain the optimal model weights. Step 4: MTP-Net Algorithm Performance Validation. The MTP-Net algorithm is validated using the optimal model weights obtained in Step 3 and the test set from Step 1 to evaluate the algorithm's inference performance. Step 5: Embedded deployment of the MTP-Net algorithm on edge computing devices. The optimal model weights obtained in Step 3 are simplified and converted in format. Using TensorRT and C++ programming, the algorithm is deployed to an embedded development board to achieve real-time, fast inference for vehicle detection and lane segmentation. Its basic principle is as follows: Figure 2 As shown. ① This method uses CSP-Darknet53 as a shared backbone network, setting up vehicle detection and lane segmentation branches to simultaneously complete vehicle detection and lane segmentation tasks. However, the vehicle detection and lane segmentation tasks are independent of each other, and their features are not fused, making it difficult to fully utilize the complex object features of the original sensor data in 2D space, resulting in low spatial position perception accuracy. ② This method uses dynamic sampling and convolutional networks to achieve vehicle detection, but does not use an attention model to consider the temporal relationship between vehicles and the spatial relationship between frames, resulting in low displacement perception accuracy and difficulty in distinguishing between static and dynamic targets.

[0031] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] The 3D target recognition model training method provided in this application embodiment can be applied to, for example... Figure 3In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send a training set to server 104. After receiving the training set, server 104 uses the training set to train the target recognition model, obtaining a trained target recognition model. Server 104 can then feed back the trained target recognition model to terminal 102. Furthermore, in some embodiments, the 3D target recognition model training method can also be implemented independently by server 104 or terminal 102. For example, terminal 102 can directly train the model using the training set, or server 104 can obtain the training set from the data storage system and train the model using the training set.

[0033] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0034] In one exemplary embodiment, such as Figure 4 As shown, a method for training a 3D target recognition model is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 3 Taking server 104 as an example, the explanation includes the following steps 201 to 202.

[0035] Step 201: Obtain the training set; the training set includes several multi-view images and the target recognition label corresponding to each multi-view image.

[0036] Step 202: Train the target recognition model using the training set to obtain the trained target recognition model; the target recognition model includes a multi-scale feature extraction module, a deep supervision branch, a 3D anchor point supervision branch, and a segmentation supervision branch.

[0037] Among them, the multi-scale feature extraction module is used to extract features at multiple scales from multi-view images to obtain multi-scale features;

[0038] The deep supervision branch is used to perform pixel-by-pixel prediction of multi-scale features to obtain the prediction depth of the features in 3D space.

[0039] The 3D anchor point supervision branch includes a first anchor point refinement layer and a second anchor point refinement layer. The first anchor point refinement layer is used to refine the anchor points of the current scene multiple times based on multi-scale features and the anchor point recognition results of the previous time step, so as to obtain the cyclic refinement results. The cyclic refinement results are the anchor point prediction results of all targets in the target scene.

[0040] The segmentation supervision branch includes a Region of Interest (ROI) prediction head, a Mask prediction head, and an alignment and fusion module. The ROI prediction head predicts the pixel coordinate range of the target corresponding to each anchor point in the multi-view image based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer. The Mask prediction head predicts the shape mask of the target corresponding to each anchor point based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer. The alignment and fusion module predicts the instance segmentation features of the target corresponding to each anchor point based on the outputs of the ROI prediction head and the Mask prediction head.

[0041] The second anchor point refinement layer is used to fuse the instance segmentation features of all anchor points into the cyclic refinement result to obtain the target recognition prediction result.

[0042] In existing models, both BEV-based and anchor-based methods use bounding box annotations in 3D space for supervised training. Some models also use ground truth depth values ​​of LiDAR point clouds in 3D space for auxiliary supervision. However, these methods struggle to fully utilize the complex object features of raw sensor data in 2D space, resulting in low target detection accuracy. Implementing steps 201 to 202 addresses the problems of low perception accuracy and poor spatial understanding in complex urban scenarios for autonomous vehicles. The main objective is to improve the perception capabilities of autonomous vehicles for pedestrians, vehicles, traffic cones, and other targets in 3D space in complex urban environments, involving 3D target detection methods for complex urban scenarios.

[0043] The target recognition model can be called the Sparse Segmentation Enhancement Recognition Model (SparseSED). Based on the source of the supervised data, the SparseSED model can be divided into three parts: a depth supervision branch, a 3D anchor point supervision branch, and a segmentation supervision branch, such as... Figure 5 As shown. The SparseSEd model first processes the multi-view image I at time T. t As input, multi-scale features F are obtained through backbone and neck. t .

[0044] ① Deep supervised branch input multi-scale features F tA convolutional neural network is used to predict the predicted depth D corresponding to features in 3D space pixel by pixel. Using ground truth depth values ​​from radar, an L2 loss is calculated between the predicted depth D and the network, which is then backpropagated and updated. A depth supervision branch runs only during training, its purpose being to provide some prior depth knowledge to the backbone and neck.

[0045] ② During initialization, the 3D anchor supervision branch obtains the initial anchor point a0 by clustering the training data, i.e., by clustering multi-view images in the sample set. The 3D anchor supervision branch inputs multi-scale features F. t Anchor point identification result at the previous moment (a) t-1 After passing through an anchor point refinement layer, the anchor point 'a' of the current scene is obtained. t Then, the anchor point refinement layer is used 5 times in a loop to refine the anchor point a in the current scene. t Continuously refine the predictions, and output anchor point prediction results containing the predicted 3D spatial positions, velocities, shapes, and rotation angles of all targets in the scene. t This refers to the iterative refinement result. Using the true values ​​of the target from the labeled 3D targets, the iterative refinement result a is calculated. t The L2 loss value is then used for backpropagation and network updates.

[0046] It should be noted that the initial anchor point a0 here is not used as a separate input, but rather as the anchor point identification result a at time t=0 when t=1. t-1 Input. Specifically, this process involves: at t=1, the initialized anchor point a0 is used as the anchor point identification result a. t-1 Input, calculate the anchor point identification result a1 at time t=1; at t=2, a1 is used as a t-1 Input, calculate the anchor point identification result a at time t=2. t This process is repeated until the task is completed.

[0047] ③ The segmentation supervision branch consists of three parts: a region of interest prediction head, a mask prediction head, and an alignment and fusion module. For example... Figure 5 As shown, firstly, the multi-scale feature F t and the output a of the penultimate anchor point refinement layer t (The output of the penultimate anchor point refinement layer, i.e., the output of the penultimate iteration of the first anchor point refinement layer) Input the region of interest prediction head, and output the pixel coordinate range of the target corresponding to each anchor point on the multi-view image; then, the multi-scale feature F t and the output a of the penultimate anchor point refinement layer tThe input is a mask prediction head, which outputs the shape mask of the target corresponding to each anchor point. Then, the pixel coordinate range predicted by the region of interest prediction head and the shape mask predicted by the mask prediction head are input to the alignment and fusion module, which outputs the instance segmentation features of the target corresponding to each anchor point. Finally, the instance segmentation features of the target corresponding to each anchor point are fused into the anchor point 'a'. t The final target recognition prediction result is obtained through the last anchor point refinement layer, that is, through the second anchor point refinement layer.

[0048] The multi-scale feature extraction module includes Backbone and Neck; Backbone is a ResNet50 model, and Neck is an FPN model.

[0049] In the image coordinate system, to improve the perception of targets at different scales, a multi-scale feature extraction structure is introduced to achieve precise target recognition and localization.

[0050] 1) Backbone Feature Extraction. ResNet50 can be used as the backbone network structure to extract edge information, texture features, and high-level semantic features from the image layer by layer from bottom to top, forming multi-level feature maps. Residual connections and cross-layer fusion enhance feature representation capabilities while preserving key structural information in the original spatial resolution.

[0051] 2) Neck Multi-Scale Feature Fusion. An FPN (Feature Pyramid Network) is introduced as the neck structure to perform upsampling, downsampling, and lateral fusion operations on multi-scale feature maps from the backbone. This establishes an efficient information transfer path between features of different resolutions, improving the detection framework's adaptability to complex backgrounds and multi-target scenes.

[0052] 3) Output the Neck multi-scale feature fusion result as multi-scale feature F t .

[0053] In another exemplary embodiment of this application, a training set is used to train the target recognition model to obtain a trained target recognition model, specifically including:

[0054] The deep supervision loss function is used to train the deep supervision branch based on the predicted depth and the real depth corresponding to the predicted depth, so as to obtain the trained deep supervision branch.

[0055] The 3D anchor point supervision loss function is adopted, and the 3D anchor point supervision branch is trained based on the cyclic refinement result and the target true value corresponding to the cyclic refinement result, so as to obtain the trained 3D anchor point supervision branch.

[0056] Using the interest loss function, the region of interest prediction head is trained based on the pixel coordinate range of the target in the multi-view image and the actual pixel coordinate range of the target corresponding to each anchor point, thus obtaining a trained region of interest prediction head.

[0057] The segmentation mask loss function is used to train the mask prediction head based on the shape mask of the target corresponding to each anchor point, so as to obtain the trained mask prediction head.

[0058] The alignment and fusion module is trained to obtain a trained alignment and fusion module; the trained deep supervision branch, the trained 3D anchor point supervision branch, the trained region of interest prediction head, the trained mask prediction head, and the trained alignment and fusion module constitute a trained target recognition model.

[0059] The deep supervision loss function and the 3D anchor point supervision loss function are both L2 loss functions.

[0060] (I) Deep Supervision Branch.

[0061] (1) Input multi-scale features F t Using a convolutional neural network, the prediction depth D corresponding to the feature in 3D space is predicted pixel by pixel.

[0062] D = conv(F) t );

[0063] In the formula, conv represents the convolution operation.

[0064] (2) Using the actual depth from the lidar, calculate and predict the L2 loss value of depth D.

[0065]

[0066] Among them, L dep Here is the deep supervision loss function; N Fpix represents the total number of pixels in the multi-scale feature map; dep i Represents the depth value of the i-th pixel predicted by the model; i = 1, 2, ... N Fpix gt_dep i This represents the actual depth value corresponding to the labeled data.

[0067] (3) Further backpropagate and update the network to provide a 3D depth prior for the backbone and neck.

[0068] (ii) 3D anchor point supervision branch.

[0069] (1) 3D anchor point supervised branch input multi-scale features F t The recognition result a from the previous moment t-1After passing through an anchor point refinement layer, the anchor point 'a' of the current scene is obtained. t The anchor point identification result a at the time preceding t=1. t-1 The initial anchor point a0 is obtained by clustering multi-view images.

[0070] (2) Then, the anchor point refinement layer is applied 5 times in a loop to refine the anchor point a in the current scene. t The refinement process is continuously improved, and the iterative refinement results are output. The iterative refinement results are 'a', which includes the predicted 3D spatial position, velocity, shape, and rotation angle of all targets in the scene. t .

[0071] (3) Using the target ground truth values ​​from the labeled 3D targets, calculate the iterative refinement result a. t The L2 loss value.

[0072]

[0073] Among them, L reg For 3D anchor point supervision loss function; anchor i N represents the i-th anchor point predicted by the model. a The number of anchor points; gt_anchor i This represents the actual anchor point in the labeled data, i.e., the target true value; w reg This represents the weight of different dimensions of the anchor point in the loss calculation.

[0074] (4) Further backpropagate and update the network.

[0075] The first and second anchor refinement layers include a temporal attention layer, a spatial attention layer, a deformable feature layer, a feed-forward network (FFN), a normalization layer (Norm), and a prediction layer (Pred Head).

[0076] Taking the first anchor point refinement layer as an example, its processing procedure is explained as follows: The temporal attention layer is used to extract temporal features from the input of the first anchor point refinement layer to obtain temporal attention features; the spatial attention layer is used to extract spatial features from the temporal attention features to obtain spatial attention features; the deformable feature layer is used to extract features from the spatial attention features using a deformable multi-head attention model to obtain deformable features; the deformable multi-head attention model refers to the ability to dynamically adjust the shape and size of the attention model; the normalization layer is used to normalize the deformable features to obtain normalized deformable features; and the prediction layer is used to predict the anchor point prediction result based on the normalized deformable features.

[0077] The processing procedure for the deformable multi-head attention model is as follows:

[0078] 1. Spatial location coding and sampling:

[0079] Let each anchor point have n reference points p. i ∈[0,1] 2 , which represents its normalized position in space.

[0080] For each scale l, the reference point is mapped to a specific location on the feature map and bilinear interpolation is performed to obtain the feature vector at the corresponding location:

[0081]

[0082] In the formula, W is the feature vector at the corresponding position of the i-th reference point after interpolation on the l-th layer feature map; l H l These are the width and height of the feature map at layer l, respectively; BilinearSample() represents the bilinear interpolation function.

[0083] 2. Multi-head attention mechanism:

[0084] For each anchor point, define a linear transformation of the query / key / value: q i =W Q Q i , Among them W Q W K W V These are learnable parameters.

[0085] The attention score and output for each attention head are as follows:

[0086]

[0087] in Here, L represents the attention weights, L represents the number of scales in the multi-scale image features, and H represents the number of attention heads.

[0088] 3. Fuse attention scores from multiple attention heads and output: After concatenating the outputs of all attention heads, perform a linear transformation to obtain the final deformable feature (instance_feature):

[0089] f i =W O ·Concat[Attention1,…,AttentionH];

[0090] In the formula, f i It is a deformable feature; W O represents a linear transformation; Concat represents a concatenation operation; Attention1 and AttentionH are the outputs of the 1st and Hth attention heads, respectively.

[0091] (III) Divide the supervision branches.

[0092] Based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer, the pixel coordinate range of the target corresponding to each anchor point in the multi-view image is predicted, specifically including:

[0093] For each anchor point in the output of the penultimate loop of the first anchor point thinning layer, the anchor point is mapped onto the multi-view image using camera intrinsic and extrinsic parameters to obtain the projection of the anchor point on the kth image; the multi-view image includes several images.

[0094] If the projection of the anchor point on the k-th image is within the k-th image, then the feature sampling point obtained by the anchor point in the deformable feature layer in the previous iteration of refinement is projected onto the k-th image to obtain the 2d coordinates of the anchor point.

[0095] The 2D coordinates of the anchor point are input into the fully connected network to obtain the predicted rectangular region corresponding to the anchor point; the predicted rectangular region corresponding to the anchor point is the range of pixel coordinates of the target corresponding to the anchor point in the multi-view image.

[0096] (1) Region of Interest (ROI) Prediction Head. The purpose of the ROI prediction head is to predict the rectangular region occupied by the anchor point in the multi-view image. Since the shapes of the objects are different and there is occlusion between objects, this rectangular region cannot be directly achieved by "mapping the bounding box corners to the 2D image space". In the proposed model, ROI prediction is achieved by the ROI prediction head.

[0097] Refine result a using loops t The i-th anchor point a i For example:

[0098] ① First, use the camera's intrinsic and extrinsic parameters to set the i-th anchor point a i Mapped onto multi-view images

[0099]

[0100] in, Let i be the projection of the i-th anchor point onto the k-th image; N is the transformation matrix from the 3D world coordinate system to the k-th camera image coordinate system; a For a t The total number of anchor points; N cam The number of cameras for multi-view images.

[0101] ②If The position is in the kth image If the i-th anchor point a is inside, then... i The feature sampling point p obtained in the previous deformable feature layer. ref,i Projected onto the k-th image From above, we obtain its 2D coordinates:

[0102]

[0103] in, For feature sampling point p ref,i In the kth image The projection on the surface.

[0104] ③Then the feature sampling point p ref,i In the kth image Projection on Inputting the fully connected network yields the predicted rectangular region R. rect,j :

[0105]

[0106] Among them, R rect,j It can be represented as in This represents the coordinates of the bottom left corner of the rectangular area. The coordinates of the top right corner of the rectangular region are represented by FCN, which stands for Fully Connected Network.

[0107] ④ Unlike ROI prediction in models such as Mask-RCNN, the segmentation of a target in 3D space is not one-to-one with that in 2D space, because a target may have projections on multiple camera views. Therefore, in this embodiment, an anchor point corresponds to the range of pixel coordinates (ROI) R of the target on the multi-view image. j Defined as:

[0108]

[0109] Where, N roi This represents the total number of ROIs.

[0110] ⑤ The set of all anchor points' ROIs is represented as:

[0111]

[0112] ⑥ Supervision is performed using instance segmentation annotations of the target:

[0113]

[0114] Among them, L roi The loss function of interest; roi i gt_roi represents the range of coordinates of the i-th pixel predicted by the model. i This represents the actual ROI in the labeled data.

[0115] ⑦ Propagate further in the reverse direction and update the network.

[0116] ⑧R is the output of the region of interest prediction head.

[0117] (2) Segmentation mask prediction.

[0118] Based on multi-scale features and the output of the penultimate iteration of the first anchor point refinement layer, the shape mask of the target corresponding to each anchor point is predicted, specifically including:

[0119] A fully connected layer is used to encode the mapping matrix to obtain the camera parameter mask; the mapping matrix consists of camera intrinsic and extrinsic parameters.

[0120] The camera parameter mask is fused with the instance features obtained by sampling from the deformable feature layer to obtain the fused instance feature mask;

[0121] The fused instance feature mask is input into a three-layer stacked deconvolution layer to obtain the shape mask of the target corresponding to each anchor point.

[0122] The purpose of segmentation mask prediction is to predict the mask corresponding to the anchor point, that is, to predict the position and shape of the target in the 2D image. The segmentation mask is implemented through the mask prediction head in this section. The ROI head predicts the region of interest R in the image for the target mask, and the Mask head predicts the segmentation mask, i.e., the shape mask, which corresponds one-to-one with the region of interest.

[0123] For the j-th roi

[0124] ①The input to the mask head includes the mapping matrix P composed of the camera's intrinsic and extrinsic parameters. cam and instance features F obtained by sampling from deformable feature layers ins,i First, a fully connected layer is used to encode the mapping matrix to obtain the k-th camera parameter mask.

[0125]

[0126] ②Then F ins,i and Fusion is achieved through addition:

[0127]

[0128] ③ Finally, the input is processed through three stacked deconvolutional layers to obtain the predicted shape mask (segmentation mask) M. j :

[0129]

[0130] Where Deconv represents a deconvolutional layer, r w r is the number of pixels in the width of the predicted shape mask. h The number of pixels in height for the predicted shape mask.

[0131] ④ The set of predicted shape masks for all ROIs:

[0132]

[0133] In the formula, M0, M1, They are the 1st, 2nd, and Nth, respectively. roi A predicted shape mask.

[0134] ⑤ Supervision is performed using instance segmentation annotations of the target:

[0135]

[0136] Among them, L mask For segmentation mask loss function; L mask,i The median value corresponding to the i-th pixel; mask i N represents the mask value predicted by the model for the i-th pixel; Mpix =r w *r h σ(mask) represents the total number of pixels in the mask; i ) is an intermediate quantity.

[0137] ⑥ Further backpropagate and update the network.

[0138] ⑦M is the output of the segmentation mask prediction module.

[0139] (3) Alignment and Fusion Module. The purpose of instance segmentation and anchor point alignment and fusion is to fuse the predicted instance segments in 2D space with the corresponding 3D anchor points, further enhancing the position and pose of the anchor points in 3D space. Its network structure is as follows: Figure 5 As shown. The input to this alignment and fusion module includes a mapping matrix P composed of camera intrinsic and extrinsic parameters. cam The region of interest (i.e., the range of pixel coordinates) R predicted by the ROI head, the shape mask M predicted by the Mask head, and the anchor point a in the sparse segmentation-enhanced recognition model. t The encoding E anchor .

[0140] A region of interest is represented as Where j represents the j-th region of interest, i represents the i-th anchor point, and k represents the k-th camera viewpoint. Pixel coordinate range R j With shape mask M j There is a one-to-one correspondence between j and (i, k).

[0141] ①For R j With M j First, a fully connected layer is used to encode the mapping matrix to obtain the k-th camera parameter mask.

[0142]

[0143] It should be noted that the camera parameter encoding network here has the same structure as the camera encoding network in (3), but they are two independent networks and do not share network parameters.

[0144] ② Next, a fully connected layer is used to adjust the pixel coordinate range R. j Encode the region of interest (ROI) to obtain the region of interest code E. roi,j E roi,j =FCN(R) j ).

[0145] ③ Then, the segmentation mask is encoded. To reduce the computational cost of encoding and further enhance the segmentation shape during encoding, we first extract the segmentation mask boundaries and then encode them. This paper uses the Laplace operator to extract the segmentation mask boundaries:

[0146]

[0147] in, This represents the result of the Laplace operator operation on the j-th prediction mask. Represents the boundary of the j-th segmentation mask, seqj,x,y This represents the boundary of the segmentation mask at position (x, y) j.

[0148] ④ Then, use the Hadamard Product to encode the segmentation mask boundaries (seg) to obtain the encoded segmentation features:

[0149] F seg,j,x,y =E mask ⊙seg j,x,y ;

[0150] Among them, F seg,j,x,y for seg j Encoded segmentation features, E mask These are the learnable parameters in the model.

[0151] ⑤ Then sum F seg,j,x,y The features of each pixel are aggregated to obtain the features after fusing the segmentation mask and the region of interest:

[0152]

[0153] Where F mask,j F represents the feature obtained by fusing the j-th segmentation mask with the region of interest. mask,j ∈R 256 .

[0154] ⑥ The set of features that are fully integrated:

[0155]

[0156] ⑦ Finally, using the anchor point index corresponding to each roi retained in the pixel coordinate range R, F mask The code E aggregated to the corresponding anchor point anchor,i The updated anchor code is obtained above:

[0157]

[0158] Where, E on the left side of the equal sign anchor,i For the updated anchor code, the E on the right side of the equals sign anchor,i To refine the result a in a loop i The encoding obtained through FCN is computed in the sparse segmentation-enhanced recognition model.

[0159] ⑧ Finally, the merged anchor code is obtained. That is, the instance segmentation features of the target corresponding to the anchor point.

[0160] This application establishes a 3D target recognition model comprising three parts: a depth supervision branch, a 3D anchor point supervision branch, and a segmentation supervision branch, thereby improving target detection accuracy. Region of interest prediction and instance mask prediction are designed, and fine-grained 2D segmentation annotations are fully utilized for supervision in 3D target detection tasks, providing a new paradigm for 3D target detection training. A module for aligning and fusing 2D instance segmentation and 3D anchor points is designed, realizing the encoding and fusion of 2D segmentation and 3D anchor points in multi-view images, further enhancing the understanding of the position and pose of anchor points in 3D space.

[0161] This application also provides an application scenario in which the above-described 3D target recognition model training method is applied. Specifically, the 3D target recognition model training method provided in this embodiment can be applied to 3D target recognition scenarios. A 3D target recognition scenario includes a content production stage, a content processing chain, and a content distribution stage. The multi-view images to be processed enter the content processing chain from the content production stage, obtain corresponding target recognition prediction results through human-machine collaboration, and then enter the downstream content distribution stage. The 3D target recognition model training method provided in this embodiment belongs to the machine labeling stage in the content processing chain. Specifically, in the content processing chain for the multi-view images to be processed, multi-view images can be labeled based on a collaborative method of machine labeling and manual labeling, that is, corresponding target recognition labels can be added to the multi-view images.

[0162] Example 2.

[0163] The three-dimensional target recognition model application method provided in this application embodiment can be applied to, for example... Figure 3 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send multi-view images of a target to server 104. After receiving the multi-view images, server 104 inputs them into a trained target recognition model to obtain the target recognition prediction result corresponding to the multi-view images. Server 104 can then feed back the obtained target recognition prediction result for the multi-view images to terminal 102. Furthermore, in some embodiments, the 3D target recognition model application method can also be implemented independently by server 104 or terminal 102. For example, terminal 102 can directly perform 3D target recognition on the multi-view images, or server 104 can obtain the multi-view images from the data storage system and perform 3D target recognition on them.

[0164] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0165] like Figure 6 As shown, this embodiment provides a method for applying a three-dimensional target recognition model, including the following steps S1 to S2.

[0166] Step S1: Acquire multi-view images of the target.

[0167] Step S2: Input the multi-view image of the target into the trained target recognition model to obtain the target recognition prediction result corresponding to the multi-view image of the target; the trained target recognition model is the model trained by the three-dimensional target recognition model training method described in Example 1.

[0168] Example 3.

[0169] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores target recognition data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a 3D target recognition model training method or a 3D target recognition model application method.

[0170] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a three-dimensional target recognition model training method as described in Embodiment 1 or a three-dimensional target recognition model application method as described in Embodiment 2.

[0172] Example 4.

[0173] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements a three-dimensional target recognition model training method as described in Embodiment 1 or a three-dimensional target recognition model application method as described in Embodiment 2.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0176] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a three-dimensional target recognition model, characterized in that, The training method for the three-dimensional target recognition model includes: Obtain the training set; the training set includes several multi-view images and the target recognition label corresponding to each multi-view image; The target recognition model is trained using a training set to obtain a trained target recognition model; the target recognition model includes a multi-scale feature extraction module, a depth supervision branch, a 3D anchor point supervision branch, and a segmentation supervision branch; Among them, the multi-scale feature extraction module is used to extract features at multiple scales from multi-view images to obtain multi-scale features; The deep supervision branch is used to perform pixel-by-pixel prediction of multi-scale features to obtain the prediction depth of the features in 3D space. The 3D anchor point supervision branch includes a first anchor point refinement layer and a second anchor point refinement layer. The first anchor point refinement layer is used to refine the anchor points of the current scene multiple times based on multi-scale features and the anchor point recognition results of the previous time step, so as to obtain the cyclic refinement results. The cyclic refinement results are the anchor point prediction results of all targets in the target scene. The segmentation supervision branch includes a region of interest (ROI) prediction head, a mask prediction head, and an alignment and fusion module. The ROI prediction head predicts the pixel coordinate range of the target corresponding to each anchor point in the multi-view image based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer. The mask prediction head predicts the shape mask of the target corresponding to each anchor point based on multi-scale features and the output of the penultimate loop refinement of the first anchor point refinement layer. The alignment and fusion module predicts the instance segmentation features of the target corresponding to each anchor point based on the outputs of the ROI prediction head and the mask prediction head. The second anchor point refinement layer is used to fuse the instance segmentation features of all anchor points into the cyclic refinement result to obtain the target recognition prediction result.

2. The three-dimensional target recognition model training method according to claim 1, characterized in that, The multi-scale feature extraction module includes Backbone and Neck; Backbone is a ResNet50 model, and Neck is an FPN model.

3. The three-dimensional target recognition model training method according to claim 1, characterized in that, The target recognition model is trained using a training set to obtain a trained target recognition model, which specifically includes: The deep supervision loss function is used to train the deep supervision branch based on the predicted depth and the real depth corresponding to the predicted depth, so as to obtain the trained deep supervision branch. The 3D anchor point supervision loss function is adopted, and the 3D anchor point supervision branch is trained based on the cyclic refinement result and the target true value corresponding to the cyclic refinement result, so as to obtain the trained 3D anchor point supervision branch. Using the interest loss function, the region of interest prediction head is trained based on the pixel coordinate range of the target in the multi-view image and the actual pixel coordinate range of the target corresponding to each anchor point, thus obtaining a trained region of interest prediction head. The segmentation mask loss function is used to train the mask prediction head based on the shape mask of the target corresponding to each anchor point, so as to obtain the trained mask prediction head. The alignment and fusion module is trained to obtain a trained alignment and fusion module; the trained deep supervision branch, the trained 3D anchor point supervision branch, the trained region of interest prediction head, the trained mask prediction head, and the trained alignment and fusion module constitute a trained target recognition model.

4. The three-dimensional target recognition model training method according to claim 3, characterized in that, The deep supervision loss function and the 3D anchor point supervision loss function are both L2 loss functions.

5. The three-dimensional target recognition model training method according to claim 1, characterized in that, The first and second anchor refinement layers include a temporal attention layer, a spatial attention layer, a deformable feature layer, a feedforward neural network, a normalization layer, and a prediction layer.

6. The three-dimensional target recognition model training method according to claim 5, characterized in that, Based on multi-scale features and the output of the penultimate iteration of the first anchor point refinement layer, the pixel coordinate range of the target corresponding to each anchor point in the multi-view image is predicted, specifically including: For each anchor point in the output of the penultimate loop of the first anchor point thinning layer, the anchor point is mapped onto the multi-view image using camera intrinsic and extrinsic parameters to obtain the projection of the anchor point on the kth image; the multi-view image includes several images. If the projection of the anchor point on the k-th image is within the k-th image, then the feature sampling point obtained by the anchor point in the deformable feature layer in the previous iteration of refinement is projected onto the k-th image to obtain the 2d coordinates of the anchor point. The 2D coordinates of the anchor point are input into the fully connected network to obtain the predicted rectangular region corresponding to the anchor point; the predicted rectangular region corresponding to the anchor point is the range of pixel coordinates of the target corresponding to the anchor point in the multi-view image.

7. The three-dimensional target recognition model training method according to claim 5, characterized in that, Based on multi-scale features and the output of the penultimate iteration of the first anchor point refinement layer, the shape mask of the target corresponding to each anchor point is predicted, specifically including: A fully connected layer is used to encode the mapping matrix to obtain the camera parameter mask; the mapping matrix consists of camera intrinsic and extrinsic parameters. The camera parameter mask is fused with the instance features obtained by sampling from the deformable feature layer to obtain the fused instance feature mask; The fused instance feature mask is input into a three-layer stacked deconvolution layer to obtain the shape mask of the target corresponding to each anchor point.

8. A method for applying a three-dimensional target recognition model, characterized in that, The application method of the three-dimensional target recognition model includes: Acquire multi-view images of the target; The target multi-view image is input into the trained target recognition model to obtain the target recognition prediction result corresponding to the target multi-view image; the trained target recognition model is a model trained using the three-dimensional target recognition model training method according to any one of claims 1-7.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the three-dimensional target recognition model training method according to any one of claims 1-7 or the three-dimensional target recognition model application method according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the three-dimensional target recognition model training method according to any one of claims 1-7 or the three-dimensional target recognition model application method according to claim 8.

Images