Multi-scale feature extraction fusion neural network and monocular view 3D target detection method

Abstract

The invention discloses a multi-scale feature extraction fusion neural network and an end-to-end monocular view 3D target detection method based on the neural network, and the multi-scale feature extraction fusion neural network comprises a feature extraction module which is used for extracting basic feature information of input data; the encoder module is used for encoding the basic feature information to obtain depth encoding information and visual encoding information; the decoder module is used for decoding the depth coding information and the visual coding information to obtain predicted target information; the matching module is used for carrying out binary matching on the predicted target information and the real target information; and the loss function module is used for calculating an error between the predicted target information and the real target information. The method can improve the 3D target detection capability, is suitable for detection of targets with different scales, and does not need a post-processing method.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a multi-scale feature extraction fusion neural network and an end-to-end monocular vision system based on the neural network. Figure 3 D target detection method. Background Art

[0002] 2D object detection technology has entered an unprecedented period of prosperity, with various new methods constantly emerging. However, with the rapid development of technologies such as autonomous driving and robotics, 3D object detection has become increasingly important, and 3D object detection is more challenging than 2D object detection. In application scenarios such as autonomous driving and robotics, 3D object detection technology has become one of the most important detection methods.

[0003] At present, 3D target detection technology is in a period of rapid development. The detection equipment is mainly monocular camera, binocular camera and multi-line laser radar. From the cost point of view, laser radar is the most expensive and monocular camera is the cheapest. However, from the perspective of detection difficulty, compared with the methods based on laser radar and multi-view images, 3D target detection based on monocular view images is the most difficult. The uncertainty of depth estimation in the monocular view image method will seriously affect the final detection result, resulting in poor performance of the method. Previously, most 3D target detection technologies followed the framework of 2D target detection methods and could not achieve end-to-end detection. A series of post-processing methods were required. However, the emergence of a Transformer-based 2D target detection method, DETR, pointed out a feasible path for end-to-end 2D target detection methods. Subsequently, MonoDETR extended the DETR method from an end-to-end 2D target detection method to an end-to-end 3D target detection method, abandoning post-processing methods such as non-maximum suppression.

[0004] In target detection tasks, the size of the detected target is often not fixed. For example, the 3D target detection task related to autonomous driving may need to detect a large truck and a puppy at the same time. Therefore, how to design a more reasonable multi-scale 3D target detection scheme to improve the recognition ability of various targets has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a multi-scale feature extraction fusion neural network and an end-to-end monocular vision system based on the neural network. Figure 3 D target detection method. It can improve the 3D target detection capability, is suitable for the detection of targets of different scales and does not require post-processing methods.

[0006] The technical solution adopted by the present invention is:

[0007] In the first aspect, the present application discloses a multi-scale feature extraction fusion neural network, comprising: a feature extraction module, used to extract basic feature information of input data; an encoder module, used to encode the basic feature information to obtain depth encoding information and visual encoding information; a decoder module, used to decode the depth encoding information and the visual encoding information to obtain predicted target information; a matching module, used to perform binary matching between the predicted target information and the real target information; and a loss function module, used to calculate the error between the predicted target information and the real target information.

[0008] As an optional technical solution, the feature extraction module adopts a ResNet50 residual network, which includes a first layer of convolution blocks, a second layer of convolution blocks, a third layer of convolution blocks, a fourth layer of convolution blocks and a fifth layer of convolution blocks. The output feature map size of the first layer of convolution blocks is one-fourth of the input size, the sizes of the input and output feature maps of the second layer of convolution blocks remain unchanged, and the output feature map sizes of the third layer of convolution blocks, the fourth layer of convolution blocks and the fifth layer of convolution blocks are half of the input size; the output features of the second layer of convolution blocks, the third layer of convolution blocks, the fourth layer of convolution blocks and the fifth layer of convolution blocks constitute multi-scale features, and the basic feature information includes the multi-scale features; the number of channels of the output features of the second layer of convolution blocks, the third layer of convolution blocks, the fourth layer of convolution blocks and the fifth layer of convolution blocks is twice that of the input features.

[0009] As an optional technical solution, the encoder module includes a deep feature encoding module and at least one multi-scale feature extraction and fusion module; the deep feature encoding module is used to encode the basic feature information to obtain depth encoding information, and the multi-scale feature extraction and fusion module is used to encode the basic feature information to obtain visual encoding information.

[0010] As an optional technical solution, the depth feature encoding module includes a dynamic sparse multi-head attention module, a depth information encoding module and an information embedding module. The dynamic sparse multi-head attention module is used to calculate the correlation between feature map regions to obtain visual information. The depth information encoding module encodes the depth information in the feature map to predict the depth information of each pixel. The information embedding module embeds the predicted depth information into the encoded visual information feature map to obtain depth encoding information.

[0011] As an optional technical solution, the multi-scale feature extraction and fusion module includes four single-scale deformable self-attention modules, a multi-feature interaction fusion module and a first feedforward network module. The four single-scale deformable self-attention modules process input feature maps of four different scales in parallel respectively to realize intra-scale feature interaction. The multi-feature interaction fusion module is used to process the output information of the four single-scale deformable self-attention modules to realize inter-scale feature interaction fusion. Finally, the first feedforward network module encodes the multi-scale features after interaction fusion and outputs them as the feature map of this stage.

[0012] As an optional technical solution, the decoding module includes a deep cross-self-attention module, a visual cross-attention module, a multi-head self-attention module and a second feedforward network module. The deep cross-self-attention module decodes the depth encoding information and then inputs the information into the visual cross-attention module together with the visual encoding information to obtain feature information that is highly integrated with the depth information and the visual information. Finally, the multi-head self-attention module and the second feedforward network module are used to perform decoding operations to obtain the predicted target information.

[0013] As an optional technical solution, the matching module uses the Hungarian algorithm to perform binary matching between the target information predicted by the model and the actual target information.

[0014] As an optional technical solution, the loss function module at least includes a 3D target information loss calculation module, which is used to calculate the loss of target depth, target 3D size, and target direction respectively, wherein the target depth loss function is:

[0015] L focal-loss =(1-p t ) γ ·log(p t );

[0016] The target 3D size loss function is:

[0017]

[0018] The target direction loss function is:

[0019]

[0020] Among them, γ is the modulation parameter, p t Represents the predicted probability of positive and negative samples, when p t If it is greater than 0.5, it is a positive sample. t Less than 0.5 is a negative sample, p i With y i Represent the predicted data and the real data respectively, p(x i ) represents the target distribution, q(xi ) is the predicted matching distribution, and n is the number of samples.

[0021] In a second aspect, the present application discloses a monocular vision system based on the multi-scale feature extraction fusion neural network in the first aspect. Figure 3 The method for detecting a 3D target comprises: inputting monocular view image data and target real information to train a network to be trained, adjusting the network structure according to the detection performance of the network model obtained by training, wherein the network to be trained comprises a multi-scale feature extraction fusion neural network as in the first aspect above; selecting a model with the best performance from the network model after adjusting the structure as a final 3D target detection model; and detecting a 3D target using the final 3D target detection model.

[0022] As an optional technical solution, the method of selecting the best performance model from the network model after adjusting the structure as the final 3D object detection model includes: using a single visual Figure 3 D target detection dataset to train the model; using monocular vision Figure 3 The performance of the model is evaluated on the D target detection validation dataset, and the average precision of the bounding box in 3D space mAP3D and the average precision of the bounding box in the bird's-eye view mAPBEV are obtained; Figure 3 The D target detection dataset continues to train models with different network structures, and selects the model with the best performance as the final end-to-end monocular vision based on the average precision of the 3D space bounding box mAP3D and the average precision of the bounding box of the bird's-eye view mAPBEV of different models. Figure 3 D Object Detection Model.

[0023] The beneficial effects of the present invention are:

[0024] The present invention adopts a multi-scale feature extraction fusion neural network based on DETR to construct an end-to-end monocular vision Figure 3 The D target detection method abandons a series of target detection post-processing methods. Thanks to the multi-scale feature extraction and fusion mechanism, it can improve the detection accuracy of targets of different scales. The present invention performs 3D target detection on monocular view images based on the multi-scale feature extraction and fusion neural network of DETR, which can realize end-to-end 3D target detection and has a better 3D target detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a diagram of the architecture of a multi-scale feature extraction fusion neural network in an exemplary embodiment.

[0026] Figure 2 In an exemplary embodiment, a single-view Figure 3 D Schematic diagram of the process of target detection method.

[0027] Figure 3 It is a schematic diagram of the architecture of a deep feature encoding module in an exemplary embodiment.

[0028] Figure 4 It is a schematic diagram of the architecture of a multi-scale feature extraction and fusion module in an exemplary embodiment.

[0029] Figure 5 is a schematic diagram of a decoder module in an exemplary embodiment.

[0030] Figure 6 is a flow chart of the Hungarian algorithm in an exemplary embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components and method steps of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] Example

[0033] like Figure 1 As shown, the present application discloses a multi-scale feature extraction fusion neural network, including: a feature extraction module, used to extract basic feature information of input data; an encoder module, used to encode the basic feature information to obtain depth coding information and visual coding information; a decoder module, used to decode the depth coding information and the visual coding information to obtain predicted target information, including predicted 2D target information and 3D target information; a matching module, used to perform binary matching between the predicted target information and the real target information; a loss function module, used to calculate the error between the predicted target information and the real target information.

[0034] As an optional implementation, the feature extraction module adopts a ResNet50 residual network, which includes a first layer convolution block, a second layer convolution block, a third layer convolution block, a fourth layer convolution block and a fifth layer convolution block. The output feature map size of the first layer convolution block is one-fourth of the input size, the input and output feature map sizes of the second layer convolution block remain unchanged, and the output feature map sizes of the third layer convolution block, the fourth layer convolution block and the fifth layer convolution block are half of the input size; the output features of the second layer convolution block, the third layer convolution block, the fourth layer convolution block and the fifth layer convolution block constitute multi-scale features, and the basic feature information includes the multi-scale features; the number of channels of the output features of the second layer convolution block, the third layer convolution block, the fourth layer convolution block and the fifth layer convolution block is twice that of the input features, and the channels of their output feature maps are 64, 256, 512, 1024, and 2048, respectively.

[0035] As an optional implementation, the encoder module includes a deep feature encoding module and at least one multi-scale feature extraction and fusion module; the deep feature encoding module is used to encode the basic feature information to obtain deep encoding information, and the multi-scale feature extraction and fusion module is used to encode the basic feature information to obtain visual encoding information. Multiple multi-scale feature extraction and fusion modules can be set. When multiple multi-scale feature extraction and fusion modules are set, each multi-scale feature extraction and fusion module is connected in series in sequence. The output of the previous module in the series structure in this embodiment is used as the input of the next module; when the number of multi-scale feature extraction and fusion modules is greater than or equal to 4, it will cause the number of parameters to be too large and reduce the calculation efficiency. When it is 1 or 2, the network model's ability to extract features for small targets is reduced, which will eventually cause the model to be unable to accurately identify small targets. Therefore, the best implementation is to set three multi-scale feature extraction and fusion modules, which has good effects and high calculation efficiency.

[0036] As an optional implementation, the depth feature encoding module includes a dynamic sparse multi-head attention module, a depth information encoding module and an information embedding module. The dynamic sparse multi-head attention module is used to calculate the correlation between feature map regions to obtain visual information. The depth information encoding module encodes the depth information in the feature map to predict the depth information of each pixel. The information embedding module embeds the predicted depth information into the encoded visual information feature map to obtain depth encoding information. Specifically, the dynamic sparse multi-head attention module uses flexible computing allocation and content perception to form m regional attention blocks, allowing the network to adaptively select k regions that should be paid more attention, calculate the correlation between k regions of the feature map to obtain more critical visual information, the depth information encoding module encodes the depth information in the feature map to predict the depth information of each pixel, and finally uses the information embedding module to embed the predicted depth information into the encoded visual information feature map to obtain the depth information feature map, that is, the depth encoding information.

[0037] As an optional implementation, the multi-scale feature extraction and fusion module includes four single-scale deformable self-attention modules, a multi-feature interaction fusion module and a first feedforward network module. The four single-scale deformable self-attention modules process input feature maps of four different scales in parallel respectively to realize intra-scale feature interaction. The multi-feature interaction fusion module is used to process the output information of the four single-scale deformable self-attention modules to realize inter-scale feature interaction fusion. Finally, the first feedforward network module encodes the multi-scale features after interaction fusion and outputs them as the feature map of this stage.

[0038] As an optional implementation, the decoding module includes a deep cross-self-attention module, a visual cross-attention module, a multi-head self-attention module and a second feedforward network module. The deep cross-self-attention module decodes the depth encoding information and then inputs the information into the visual cross-attention module together with the visual encoding information to obtain feature information that is highly integrated with the depth information and the visual information. Finally, the multi-head self-attention module and the second feedforward network module are used to perform decoding operations to obtain the predicted target information.

[0039] As an optional implementation, the matching module uses the Hungarian algorithm to perform a binary match between the target information predicted by the model and the actual target information.

[0040] As an optional implementation, the loss function module at least includes a 3D target information loss calculation module, which is used to perform loss calculations on target depth, target 3D size, and target direction, respectively, wherein the target depth loss function is:

[0041] L focal-loss =(1-p t ) γ ·log(p t );

[0042] The target 3D size loss function is:

[0043]

[0044] The target direction loss function is:

[0045]

[0046] Among them, γ is the modulation parameter, p t Represents the predicted probability of positive and negative samples, when p t If it is greater than 0.5, it is a positive sample. t Less than 0.5 is a negative sample, p i With y i Represent the predicted data and the real data respectively, p(xi ) represents the target distribution, q(x i ) is the predicted matching distribution, and n is the number of samples.

[0047] like Figure 2 As shown, this embodiment also discloses a monocular vision system based on the multi-scale feature extraction fusion neural network disclosed in the above embodiment. Figure 3 The method for detecting a 3D target comprises: S1, inputting monocular view image data and target real information to train a network to be trained, and adjusting the network structure according to the detection performance of the network model obtained by training, wherein the network to be trained comprises a multi-scale feature extraction fusion neural network as disclosed in the above embodiment; S2, selecting a model with the best performance from the network model after adjusting the structure as a final 3D target detection model; and S3, detecting a 3D target using the final 3D target detection model.

[0048] As an optional implementation, the step S2 includes: S201, using a monocular vision Figure 3 D target detection data set to train the model; S202, using monocular vision Figure 3 The performance of the model is evaluated using the D target detection validation dataset, and the average precision of the 3D space bounding box mAP3D and the average precision of the bounding box of the bird's-eye view mAPBEV are obtained; S203, using the monocular vision Figure 3 The D target detection dataset continues to train models with different network structures, and selects the model with the best performance as the final end-to-end monocular vision based on the average precision of the 3D space bounding box mAP3D and the average precision of the bounding box of the bird's-eye view mAPBEV of different models. Figure 3 D Object Detection Model.

[0049] In order to better understand the present application, the present embodiment is further described below in combination with the above method and the multi-scale feature extraction fusion neural network.

[0050] This example provides an end-to-end monocular vision Figure 3 D target detection method, specifically comprising the following steps:

[0051] Step 1: Build Figure 1 The multi-scale feature extraction fusion neural network based on DETR shown in the figure has basic modules including feature extraction module, deep feature encoding module, multi-scale feature extraction fusion module, decoder module, matching module, and loss function module.

[0052] The feature extraction module mainly obtains the basic feature information of the input data. The feature extraction module uses the ResNet50 residual network, which is composed of 5 stages (i.e., the first convolution block, the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block) connected in series. The output feature map size of the first stage (i.e., the first convolution block) is one-fourth of the input size, the input and output feature sizes of the second stage (i.e., the second convolution block) remain unchanged, and the output feature map size of the last three stages (i.e., the third convolution block, the fourth convolution block, and the fifth convolution block) is half of the input size. The output features of the last four stages (i.e., the second convolution block, the third convolution block, the fourth convolution block, and the fifth convolution block) are selected as the input of the subsequent modules. The number of channels of the output features of the last four stages is twice that of the input features, and the channels of the output feature maps of each stage are 64, 256, 512, 1024, and 2048 respectively.

[0053] The depth feature encoding module is used to encode basic feature information while predicting the depth value of each pixel position and mapping each depth value into a single vector, and finally embedding the mapping vector into the encoded feature information. Figure 3 As shown, the deep feature encoding module is composed of a dynamic sparse multi-head attention module, a deep information encoding module and an information embedding module. The dynamic sparse multi-head attention module adopts a dynamic sparse multi-head attention mechanism to achieve more flexible computing allocation and content perception. It divides the entire feature map into different regions and calculates the correlation between each region and other regions except itself, thereby increasing attention to important information to achieve the purpose of improving model performance; the deep information encoding module uses a convolutional network to predict the depth value of each feature point in the feature map, and maps a single depth value to a single vector; the information embedding module embeds the predicted depth information into the encoded feature information, and the embedding operation is addition.

[0054] As a visual information encoder module, the multi-scale feature extraction and fusion module mainly encodes the basic feature information and optimizes the extraction of important information at different scales, replacing the commonly used single-scale feature encoder. Figure 4 As shown, three multi-scale feature extraction and fusion modules are set and connected in series. The multi-scale feature extraction and fusion module includes four single-scale deformable self-attention modules, one multi-feature interactive fusion module and one first feedforward network module. The four single-scale deformable self-attention modules interact with the features within a single scale in parallel. The multi-feature interactive fusion module is a top-down bidirectional fusion structure for interactive fusion between multi-scale features. Finally, the first feedforward network module further encodes and outputs the encoded feature information of this stage.

[0055] The decoder module is used to decode the depth coding information and the visual coding information to predict the 2D and 3D information of the target. Figure 5 As shown, the decoder module includes a deep cross-attention module, a visual cross-attention module, a multi-head self-attention module and a second feedforward network module, and each module is connected in series in the above order to form a decoder module. Each attention module involves position encoding, and the encoding method is as follows:

[0056]

[0057] Among them, pos is the position number, i is the index, and d model Represents the dimension of the output embedding feature, PE(pos,2i) represents the value of the position encoding on the even dimension, and PE(pos,2i+1) represents the value of the position encoding on the odd dimension.

[0058] The matching module can perform binary matching between the target information predicted by the model and the real target information. The matching algorithm adopts the Hungarian algorithm. The specific process is as follows: Figure 6 As shown, first, there is an empty matching, and then a point is extracted from a set in the bipartite graph and a matching point is found in the other set. If there is no conflict, match it first. If a conflict occurs, find an augmenting path and re-match it. Repeat the above ideas until all the points in the first set find matching points in the other set, or no matching points or augmenting paths are found, and finally a set of matching points is obtained.

[0059] The loss function module is used to calculate the error between the model prediction result and the actual target information, including 2D target information loss and 3D target information loss. The 2D target information (including target category, target 2D size, and target center point) is calculated using L focal-loss Function, L CIou-loss Function, mean absolute error L MAE The function calculates the loss of 2D target information; for 3D target information (including target depth, target 3D size, and target direction), L focal-loss Function, mean absolute error L MAE Function, cross entropy loss function L cross-entropy-loss Calculate the loss of 3D target information:

[0060] L focal-loss =(1-p t ) γ ·log(p t )…………(1)

[0061]

[0062]

[0063] Among them, γ is the modulation parameter, p tRepresents the predicted probability of positive and negative samples, when p t If it is greater than 0.5, it is a positive sample. t Less than 0.5 is a negative sample, IOU stands for intersection over union, b and b gt Represent the center points of the predicted box and the real box respectively, ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the minimum closure area that can contain both the predicted box and the real box, α is the weight function, v is used to measure the similarity of the aspect ratio, and p i With y i Represents the predicted data and the real data, p(x i ) represents the target distribution, q(x i ) is the predicted matching distribution, and b represents the number of samples.

[0064] Step 2: Input the monocular view image dataset to train the multi-scale feature extraction fusion neural network constructed in step 1, and adjust the network structure according to the detection performance of the trained model to obtain the detection accuracy results of network models with different structures.

[0065] Adjustments to the network structure include but are not limited to: replacing the loss function, using different network models in the feature extraction module, replacing the feature extraction unit, changing the order of feature extraction module composition, and adjusting the number of feature extraction modules.

[0066] Step 3: Select the best network adjustment method according to step 2, input the monocular view image dataset to retrain the network, and make the final end-to-end monocular view Figure 3 D object detection model achieves the best detection performance.

[0067] Step 4: Use step 3 to get the final end-to-end monocular vision Figure 3 D The target detection model detects the target.

[0068] In a specific monocular Figure 3During the implementation of 3D target detection, the experimental environment is 13th Gen Intel(R)Core(TM)i5-13600KF, RAM size 32GB, RTX 4080GPU, and Pytorch deep learning framework is used for training. The KITTI dataset is used, including 7,481 training images and 7,518 test images. The KITTI dataset is jointly sponsored by Karlsruhe Institute of Technology in Germany and Toyota Technological University Chicago for research in the field of autonomous driving. The dataset can be divided into Road, City, Residential, Campus, and Person. For 3D object detection, the labels can be subdivided into car, van, truck, pedestrian, pedestrian (sitting), cyclist, tram, and misc, etc. 3,769 images are split from the KITTI training dataset as the validation set, and the remaining 3712 images are used as the training set.

[0069] This method is an end-to-end monocular Figure 3 The D target detection method uses monocular view data as input, first obtains the initial feature information (i.e., the basic feature information mentioned above) through the feature extraction module, and then inputs the initial feature information into the deep feature encoding module and the multi-scale feature extraction and fusion module to obtain the deep feature information (i.e., the deep encoding information mentioned above) and the multi-scale visual feature information (i.e., the visual encoding information mentioned above), and then processes each type of feature information through the decoder module, and finally outputs the 3D target information of the target in the image. The training parameter values ​​of the multi-scale feature fusion neural network based on DETR used in this method are shown in Table 1.

[0070] Table 1. Parameters of multi-scale feature fusion neural network based on DETR

[0071]

[0072] In this embodiment, the multi-scale feature extraction fusion neural network can effectively extract feature information of targets of different scales. Combined with the basic network architecture of MonoDETR, a multi-scale feature extraction fusion neural network based on DETR is designed. The monocular view image data is trained using the multi-scale feature extraction fusion neural network model based on DETR. This can establish an end-to-end 3D target detection method with higher recognition capability, effectively avoiding the use of complex post-processing methods and effectively improving the recognition capability of multi-scale targets.

[0073] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a readable medium or on a network, and includes several instructions to enable an electronic device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0074] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. All technical solutions that fall within the scope defined by the claims of the present invention fall within the protection scope of the present invention.

Claims

1. Multi-scale feature extraction fusion neural network, characterized by: include: Feature extraction module, used to extract basic feature information of input data; An encoder module, used for encoding the basic feature information to obtain depth encoding information and visual encoding information; A decoder module, configured to decode the depth coding information and the visual coding information to obtain predicted target information; A matching module, used to perform binary matching between the predicted target information and the real target information; The loss function module is used to calculate the error between the predicted target information and the actual target information.

2. The multi-scale feature extraction fusion neural network according to claim 1, characterized in that: The feature extraction module adopts the ResNet50 residual network, which includes a first layer of convolution blocks, a second layer of convolution blocks, a third layer of convolution blocks, a fourth layer of convolution blocks and a fifth layer of convolution blocks. The output feature map size of the first layer of convolution blocks is one-fourth of the input size, the sizes of the input and output feature maps of the second layer of convolution blocks remain unchanged, and the output feature map sizes of the third layer of convolution blocks, the fourth layer of convolution blocks and the fifth layer of convolution blocks are half of the input size; the output features of the second layer of convolution blocks, the third layer of convolution blocks, the fourth layer of convolution blocks and the fifth layer of convolution blocks constitute multi-scale features, and the basic feature information includes the multi-scale features; the number of channels of the output features of the second layer of convolution blocks, the third layer of convolution blocks, the fourth layer of convolution blocks and the fifth layer of convolution blocks is twice that of the input features.

3. The multi-scale feature extraction fusion neural network according to claim 1, characterized in that: The encoder module includes a deep feature encoding module and at least one multi-scale feature extraction and fusion module; the deep feature encoding module is used to encode the basic feature information to obtain deep encoding information, and the multi-scale feature extraction and fusion module is used to encode the basic feature information to obtain visual encoding information.

4. The multi-scale feature extraction fusion neural network according to claim 3, characterized in that: The depth feature encoding module includes a dynamic sparse multi-head attention module, a depth information encoding module and an information embedding module. The dynamic sparse multi-head attention module is used to calculate the correlation between feature map regions to obtain visual information. The depth information encoding module encodes the depth information in the feature map to predict the depth information of each pixel. The information embedding module embeds the predicted depth information into the encoded visual information feature map to obtain depth encoding information.

5. The multi-scale feature extraction fusion neural network according to claim 3, characterized in that: The multi-scale feature extraction and fusion module includes four single-scale deformable self-attention modules, a multi-feature interactive fusion module and a first feedforward network module. The four single-scale deformable self-attention modules process input feature maps of four different scales in parallel to realize intra-scale feature interaction. The multi-feature interactive fusion module is used to process the output information of the four single-scale deformable self-attention modules to realize inter-scale feature interactive fusion. Finally, the first feedforward network module encodes the multi-scale features after interactive fusion and outputs them as the feature map of this stage.

6. The multi-scale feature extraction fusion neural network according to claim 1, characterized in that: The decoding module includes a deep cross-self-attention module, a visual cross-attention module, a multi-head self-attention module and a second feedforward network module. The deep cross-self-attention module decodes the deep coding information and inputs it into the visual cross-attention module together with the visual coding information to obtain feature information with a high degree of fusion of the deep information and the visual information. Finally, the multi-head self-attention module and the second feedforward network module are used to perform decoding operations to obtain the predicted target information.

7. The multi-scale feature extraction fusion neural network according to claim 1, characterized in that: The matching module uses the Hungarian algorithm to perform a binary match between the target information predicted by the model and the real target information.

8. The multi-scale feature extraction fusion neural network according to claim 1, characterized in that: The loss function module at least includes a 3D target information loss calculation module, which is used to perform loss calculations on target depth, target 3D size, and target direction, respectively, wherein the target depth loss function is: L focal-loss =(1-p t ) γ ·log(p t ); The target 3D size loss function is: The target direction loss function is: Among them, γ is the modulation parameter, p t Represents the predicted probability of positive and negative samples, when p t If it is greater than 0.5, it is a positive sample. t Less than 0.5 is a negative sample, p i With y i Represent the predicted data and the real data respectively, p(x i ) represents the target distribution, q(x i ) is the predicted matching distribution, and n is the number of samples.

9. A monocular view 3D target detection method based on multi-scale feature extraction and fusion neural network, characterized in that: include: Inputting monocular view image data and target real information to train the network to be trained, and adjusting the network structure according to the detection performance of the network model obtained by training, wherein the network to be trained includes the multi-scale feature extraction fusion neural network according to any one of claims 1 to 8; From the network models after adjusting the structure, select the model with the best performance as the final 3D object detection model; The final 3D object detection model is used to detect 3D objects.

10. The monocular view 3D target detection method based on multi-scale feature extraction and fusion neural network according to claim 9, characterized in that: The step of selecting the best performance model from the network model after adjusting the structure as the final 3D object detection model includes: The model is trained using a monocular view 3D object detection dataset; The performance of the model is evaluated using the monocular view 3D object detection validation dataset, and the average precision of the 3D space bounding box mAP3D and the average precision of the bounding box of the bird's-eye view mAPBEV are obtained; The monocular view 3D object detection dataset is used to continue training models with different network structures, and the model with the best performance is selected as the final end-to-end monocular view 3D object detection model based on the 3D space bounding box average precision mAP3D and the bird's-eye view bounding box average precision mAPBEV of different models.

Images