Intelligent driving front-view image sensing algorithm based on heterogeneous convolution

Abstract

The invention discloses an intelligent driving front-view image perception algorithm based on heterogeneous convolution, and the algorithm comprises the steps: S1, carrying out the basic feature extraction of an original image through an initial convolution layer, and enabling the original image to enter a plurality of parallel convolution units which are integrated together; s2, performing feature fusion on the output of the plurality of parallel convolution units in a weighted fusion mode; s3, stacking a plurality of parallel convolution units into a backbone network, and performing multi-scale target detection through an FPN feature pyramid; and S4, after the detection of the targets of multiple scales is finished, completing the training of the intelligent driving front-view image perception algorithm model. According to the intelligent driving front-view image perception algorithm based on heterogeneous convolution, the reasoning efficiency and the feature extraction capability of the model can be remarkably improved, and the balance problem of the lightweight target detection model between the calculation complexity and the feature extraction capability is solved.

Description

Technical Field

[0001] The present invention relates to an intelligent driving forward vision image perception algorithm based on heterogeneous convolution, belonging to the technical field of vehicle intelligent driving image perception algorithms. Background Art

[0002] At present, with the rapid development of intelligent driving technology, forward-view image perception algorithms play a vital role in autonomous driving systems. Forward-view image perception technology aims to capture road environment images through cameras, identify and understand information such as pedestrians, vehicles, traffic signs and road conditions around the vehicle, and provide accurate environmental perception for vehicle decision-making.

[0003] In order to improve the accuracy and efficiency of front-view image perception, researchers have been constantly exploring and optimizing various convolutional neural network (CNN) architectures. In recent years, heterogeneous convolution technology, as an emerging convolution method, has gradually attracted the attention of researchers. Heterogeneous convolution combines multiple convolution operations together, attempting to improve the performance of the model through the synergy of different types of convolutions. However, in terms of computational complexity and model flexibility, especially in practical applications, there are still deficiencies in the adaptability to different hardware platforms and the optimization of real-time performance.

[0004] In the field of intelligent driving, there are extremely high requirements for the real-time and reliability of the system. Existing technologies are difficult to meet both real-time and reliability without sacrificing accuracy, especially on hardware platforms with limited resources. This makes existing technical solutions mainly focus on optimizing a single convolution operation or selecting the most appropriate convolution type, while there is still a lack of effective solutions for how to efficiently combine multiple convolution operations to achieve lightweight and efficient image perception algorithms. Therefore, there is an urgent need for an innovative heterogeneous convolution module that can reduce computational complexity and parameter quantity while ensuring image perception accuracy, and adapt to the deployment requirements of various hardware platforms.

[0005] The existing technology needs to solve the following problems:

[0006] 1. Computing resource adaptation problem: Different convolution operations have different computing efficiency and resource consumption on different hardware platforms. The existing heterogeneous convolution modules are difficult to effectively adapt to multiple hardware platforms in practical applications, resulting in poor deployment effects.

[0007] 2. Real-time performance optimization problem: Existing heterogeneous convolution technology often ignores the optimization of real-time performance while pursuing high precision, resulting in processing delays and performance bottlenecks when the intelligent driving forward-view image perception algorithm processes high-resolution images or large-scale data.

[0008] 3. Algorithm integration and fusion issues: Most of the current heterogeneous convolution modules focus on the optimization of a single convolution operation. How to effectively integrate and fuse multiple convolution operations is still a technical problem that needs to be solved urgently. Summary of the invention

[0009] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art. The present invention proposes an intelligent driving forward view image perception algorithm based on heterogeneous convolution, which can significantly improve the reasoning efficiency and feature extraction capability of the model, and solves the balance problem between computational complexity and feature extraction capability of lightweight target detection models.

[0010] In order to solve the above technical problems, the technical solution of the present invention is:

[0011] A front view image perception algorithm for intelligent driving based on heterogeneous convolution, which includes:

[0012] Step S1: The original image first passes through an initial convolution layer to extract basic features, and then enters multiple parallel convolution units integrated together;

[0013] Step S2: feature fusion of the outputs of multiple parallel convolution units by weighted fusion;

[0014] Step S3: stack multiple parallel convolutional units into a backbone network, and perform target detection at multiple scales through the FPN feature pyramid;

[0015] Step S4: After the target detection of multiple scales is completed, the training of the intelligent driving front view image perception algorithm model is completed;

[0016] Step S5: Enter the inference phase of the intelligent driving front view image perception algorithm model, and convert the parallel structure during training into a single standard convolution structure through reparameterization technology.

[0017] Furthermore, the multiple parallel convolution units in step S1 include a standard convolution unit, a depthwise separable convolution unit and a dilated convolution unit.

[0018] Furthermore, in step S1, the integration method of multiple parallel convolution units specifically includes the following steps:

[0019] A parallel structure is adopted to integrate the standard convolution unit, the depthwise separable convolution unit and the dilated convolution unit;

[0020] The weight of the standard convolution unit is W std , the bias of the standard convolution unit is b std ;

[0021] The weight of the depthwise separable convolutional unit is W dw, W pw , the bias of the depthwise separable convolutional unit is b dw 、b pw ;

[0022] The weight of the dilated convolution unit is W dil , the bias of the dilated convolution unit is b dil .

[0023] Furthermore, in step S2, the outputs of multiple parallel convolution units are subjected to feature fusion by weighted fusion, which specifically includes the following steps:

[0024] The feature maps output by the standard convolution unit, depth-separable convolution unit and dilated convolution unit after passing through the batch normalization layer are:

[0025] y std =BN(Conv(x,W std )+b std );

[0026] y dw_pw =BN(Conv(Conv(x,W dw )+b dw , W pw )+b pw );

[0027] y dil =BN(Conv(x,W dil )+b dil );

[0028] Among them, y std is the feature map output by the standard convolution unit after the batch normalization layer, y dw_pw is the feature map output by the depthwise separable convolutional unit after the batch normalization layer, y dil It is the feature map output by the hole convolution unit after the batch normalization layer;

[0029] The feature map y std ,y dw_pw and dil After merging through the feature fusion layer, the final comprehensive feature map Y is obtained fused .

[0030] Furthermore, in step S3, multiple parallel convolutional units are stacked into a backbone network, and target detection at multiple scales is performed through the FPN feature pyramid, which specifically includes the following steps:

[0031] Multiple parallel heterogeneous units are stacked into a backbone network, and then the FPN feature pyramid network is used to generate feature maps of multiple scales to achieve target detection at multiple scales.

[0032] There are five parallel heterogeneous units, and the feature maps of different scales include a first scale feature map, a second scale feature map, and a third scale feature map.

[0033] Furthermore, each of the parallel heterogeneous units includes an integrated 2D convolution unit, a standard convolution unit, a depthwise separable convolution unit and a dilated convolution unit.

[0034] Furthermore, in step S5, the parallel structure during training is converted into a single standard convolution structure by a reparameterization technique, which specifically includes the following steps:

[0035] First, weight fusion is performed, combining the weight W of the standard convolution unit std , the weight W of the depthwise separable convolutional unit dw , W pw And the weight W of the dilated convolution unit dil , get a standard convolution weight W combined , the convolution weight W combined for:

[0036] W combined =W std +(W dw ×W pw )+W dil ;

[0037] Secondly, bias fusion is performed to convert the bias b of the standard convolution unit std , the bias b of the depthwise separable convolutional unit dw 、b pw , the bias b of the dilated convolutional unit dil Fusion is performed to obtain an equivalent bias b combined , the equivalent bias b combined for:

[0038] b combined =b std +b dw +b pw +b dil ;

[0039] Through the re-parameterization technique, the parallel convolution structure in the training phase is compressed into a single standard convolution structure.

[0040] By adopting the above technical solution, the present invention has the following beneficial effects:

[0041] 1. The present invention adopts a parallel heterogeneous convolution structure in the training stage, including multiple convolution operations such as standard convolution, depthwise separable convolution and dilated convolution. This parallel structure can fully learn diverse features during the training process. In the inference stage, the present invention adopts reparameterization technology to merge the multi-path convolution operations in the training stage into a single efficient path. This method reduces the consumption of computing resources during inference and adapts to different hardware platforms.

[0042] 2. The present invention re-parameterizes the original parallel convolution operations into a single path. This technology reduces the calculation steps and memory access in the reasoning process, thereby greatly improving the reasoning speed. This is crucial for application scenarios that require real-time processing, especially the forward-looking image perception system in intelligent driving.

[0043] 3. The present invention uses a multi-path parallel convolution structure to capture multi-scale and diverse feature information during the training phase. After re-parameterization, a single path is used during the inference phase to ensure the high efficiency of feature fusion. This strategy ensures that while keeping the model lightweight, it can still provide rich feature information, thereby improving the accuracy of image perception tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a forward-view image perception algorithm for intelligent driving based on heterogeneous convolution of the present invention;

[0045] Figure 2 is a structural diagram of multiple parallel convolution units of the present invention;

[0046] Figure 3 This is a structural diagram of the intelligent driving forward vision image perception algorithm model of the present invention after training is completed;

[0047] Figure 4 This is a structural diagram of the intelligent driving forward vision image perception algorithm model of the present invention after using the re-parameterization technology. DETAILED DESCRIPTION

[0048] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, this embodiment provides an intelligent driving front view image perception algorithm based on heterogeneous convolution, which includes:

[0050] Step S1, the original image first passes through an initial convolution layer to extract basic features, and then enters multiple parallel convolution units integrated together, and the multiple parallel convolution units include standard convolution units, depth-separable convolution units and hole convolution units;

[0051] Step S2: feature fusion of the outputs of multiple parallel convolution units by weighted fusion;

[0052] Step S3: stack multiple parallel convolutional units into a backbone network, and perform target detection at multiple scales through the FPN feature pyramid;

[0053] Step S4: After the target detection of multiple scales is completed, the training of the intelligent driving front view image perception algorithm model is completed;

[0054] Step S5: Enter the inference phase of the intelligent driving forward vision image perception algorithm model, and convert the complex parallel structure during training into a single standard convolution structure through reparameterization technology.

[0055] In step S1 of this embodiment, the integration method of multiple parallel convolution units specifically includes the following steps:

[0056] like Figure 2 As shown in the figure, during the training stage, a parallel structure is adopted to integrate the standard convolution unit, the depthwise separable convolution unit and the hole convolution unit; this integration method enables the intelligent driving forward vision image perception algorithm model to learn diverse features from different convolution operations during the training process, thereby improving the richness of feature expression.

[0057] The standard convolution unit has a powerful feature extraction capability. The weight of the standard convolution unit is W std , the bias of the standard convolution unit is b std .

[0058] The depth-wise separable convolutional unit can reduce the amount of computation while retaining the ability to extract features. The weight of the depth-wise separable convolutional unit is W dw , W pw , the bias of the depthwise separable convolutional unit is b dw 、b pw .

[0059] The dilated convolution unit can expand the receptive field and capture more contextual information. The weight of the dilated convolution unit is W dil , the bias of the dilated convolution unit is b dil .

[0060] In step S2 of this embodiment, the outputs of multiple parallel convolution units are subjected to feature fusion by weighted fusion, which specifically includes the following steps:

[0061] The feature maps output by the standard convolution unit, depth-separable convolution unit and dilated convolution unit after passing through the batch normalization layer are:

[0062] y std =BN(Conv(x,Wstd )+b std );

[0063] y dw_pw =BN(Conv(Conv(x,W dw )+b dw , W pw )+b pw );

[0064] y dil =BN(Conv(x,W dil )+b dil );

[0065] Among them, y std is the feature map output by the standard convolution unit after the batch normalization layer, y dw_pw is the feature map output by the depthwise separable convolutional unit after the batch normalization layer, y dil It is the feature map output by the dilated convolution unit after the batch normalization layer.

[0066] The feature map y std ,y dw_pw and dil After merging through the feature fusion layer, the final comprehensive feature map Y is obtained fused The feature fusion layer can adopt weighted summation, concatenation or other fusion strategies to ensure that the feature information of different convolution operations is effectively integrated and enhance the robustness and generalization ability of the model.

[0067] In step S3 of this embodiment, multiple parallel convolutional units are stacked into a backbone network, and target detection at multiple scales is performed through the FPN feature pyramid, which specifically includes the following steps:

[0068] Multiple parallel heterogeneous units are stacked into a backbone network, and then the FPN feature pyramid network is used to generate feature maps of multiple scales to achieve target detection at multiple scales.

[0069] There are 5 parallel heterogeneous units, which are stage 1 to stage 5. Feature maps of different scales include a first scale feature map (64×120), a second scale feature map (32×60), and a third scale feature map (16×30).

[0070] Each parallel heterogeneous unit includes an integrated 2D convolution unit, a standard convolution unit, a depth-separable convolution unit, and a dilated convolution unit.

[0071] like Figure 3As shown, in order to ensure that the model can effectively process targets of different scales, the present invention adopts a multi-scale detection mechanism. Multi-scale detection uses the FPN feature pyramid network to generate multi-scale feature maps. By constructing feature maps of different levels, it is possible to capture information in the image from different levels of abstraction, which is particularly important for processing targets of different sizes. In this embodiment, the FPN feature pyramid network generates feature maps of three main scales, namely 64×120, 32×60 and 16×30. These feature maps correspond to different levels of image abstraction, thereby performing target detection at different scales. Specifically: the first scale feature map, i.e., the large-scale feature map (64×120), is mainly used to detect smaller targets. Due to its high resolution, it can capture smaller details and local features, thereby improving the detection performance of small targets. The second scale feature map, i.e., the medium-scale feature map (32×60), is used to detect medium-sized targets. The feature map of this scale has a good effect in balancing details and contextual information, and is suitable for target detection in general situations. The third-scale feature map, i.e., the small-scale feature map (16×30), is specifically used for detecting large targets because it contains more contextual information and can better identify and locate large-sized objects.

[0072] In step S5 of this embodiment, the complex parallel structure during training is converted into a single standard convolution structure through the reparameterization technology, which specifically includes the following steps:

[0073] This step solves the problem of model conversion from training to inference, ensuring that the intelligent driving forward image perception algorithm model can fully utilize the advantages of multiple convolution operations during training and maintain high efficiency during inference. At the end of the training phase of the intelligent driving forward image perception algorithm model, a specific mathematical transformation is used to merge the multiple parallel convolution operations into an equivalent, single standard convolution operation. Figure 4 This fusion takes into account the weights and biases of each convolution branch, and derives equivalent weights and biases through formulas to ensure that the output of the model in the inference phase is consistent with that in the training phase. Specifically, the following addition method can be used for fusion.

[0074] First, weight fusion is performed, combining the weight W of the standard convolution unit std , the weight W of the depthwise separable convolutional unit dw , W pw And the weight W of the dilated convolution unit dil , get a standard convolution weight W combined , the convolution weight W combined for:

[0075] W combined =W std +(W dw×W pw )+W dil ;

[0076] Secondly, bias fusion is performed to convert the bias b of the standard convolution unit std , the bias b of the depthwise separable convolutional unit dw 、b pw , the bias b of the dilated convolutional unit dil Fusion is performed to obtain an equivalent bias b combined , equivalent bias b combined for:

[0077] b combined =b std +b dw +b pw +b dil ;

[0078] Through this reparameterization technique, the complex parallel convolution structure in the training phase is compressed into a single standard convolution structure, which significantly improves the computational efficiency in the inference phase.

[0079] This approach ensures consistent performance of the model at different stages while optimizing inference speed and usage of computing resources.

[0080] The specific embodiments described above further illustrate the technical problems, technical solutions and beneficial effects solved by the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent driving front view image perception algorithm based on heterogeneous convolution, characterized in that: It includes: Step S1: The original image first passes through an initial convolution layer to extract basic features, and then enters multiple parallel convolution units integrated together; Step S2: feature fusion of the outputs of multiple parallel convolution units by weighted fusion; Step S3: stack multiple parallel convolutional units into a backbone network, and perform target detection at multiple scales through the FPN feature pyramid; Step S4: After the target detection of multiple scales is completed, the training of the intelligent driving front view image perception algorithm model is completed; Step S5: Enter the inference phase of the intelligent driving front view image perception algorithm model, and convert the parallel structure during training into a single standard convolution structure through reparameterization technology.

2. The intelligent driving forward vision image perception algorithm based on heterogeneous convolution according to claim 1 is characterized by: The multiple parallel convolution units in step S1 include a standard convolution unit, a depthwise separable convolution unit and a dilated convolution unit.

3. The intelligent driving forward vision image perception algorithm based on heterogeneous convolution according to claim 2 is characterized by: In step S1, the integration method of multiple parallel convolution units specifically includes the following steps: A parallel structure is adopted to integrate the standard convolution unit, the depthwise separable convolution unit and the dilated convolution unit; The weight of the standard convolution unit is W std , the bias of the standard convolution unit is b std ; The weight of the depthwise separable convolutional unit is W dw , W pw , the bias of the depthwise separable convolutional unit is b dw , b pw ; The weight of the dilated convolution unit is W dil , the bias of the dilated convolution unit is b dil .

4. The intelligent driving forward vision image perception algorithm based on heterogeneous convolution according to claim 3 is characterized in that: In step S2, the outputs of multiple parallel convolution units are subjected to feature fusion by weighted fusion, which specifically includes the following steps: The feature maps output by the standard convolution unit, depth-separable convolution unit and dilated convolution unit after passing through the batch normalization layer are: y std =BN(Conv(x,W std )+b std ); y dw_pw =BN(Conv(Conv(x,W dw )+b dw ,W pw )+b pw ); y dil =BN(Conv(x,W dil )+b dil ); Among them, y std is the feature map output by the standard convolution unit after the batch normalization layer, y dw_pw is the feature map output by the depthwise separable convolutional unit after the batch normalization layer, y dil It is the feature map output by the hole convolution unit after the batch normalization layer; The feature map y std ,y dw-pw and dil After merging through the feature fusion layer, the final comprehensive feature map Y is obtained fused .

5. The intelligent driving forward vision image perception algorithm based on heterogeneous convolution according to claim 4 is characterized in that: In step S3, multiple parallel convolutional units are stacked into a backbone network, and target detection at multiple scales is performed through the FPN feature pyramid, which specifically includes the following steps: Multiple parallel heterogeneous units are stacked into a backbone network, and then the FPN feature pyramid network is used to generate feature maps of multiple scales to achieve target detection at multiple scales. There are five parallel heterogeneous units, and the feature maps of different scales include a first scale feature map, a second scale feature map, and a third scale feature map.

6. The intelligent driving forward vision image perception algorithm based on heterogeneous convolution according to claim 5 is characterized by: Each of the parallel heterogeneous units includes an integrated 2D convolution unit, a standard convolution unit, a depth-separable convolution unit and a hole convolution unit.

7. The intelligent driving forward vision image perception algorithm based on heterogeneous convolution according to claim 1 is characterized in that: In step S5, the parallel structure during training is converted into a single standard convolution structure through a reparameterization technique, which specifically includes the following steps: First, weight fusion is performed, combining the weight W of the standard convolution unit std , the weight W of the depthwise separable convolutional unit dw , W pw And the weight W of the dilated convolution unit dil , get a standard convolution weight W combined , the convolution weight W combined for: IN combined =In std +(W dw ×W pw )+W dil ; Secondly, bias fusion is performed to convert the bias b of the standard convolution unit std , the bias b of the depthwise separable convolutional unit dw , b pw , the bias b of the dilated convolutional unit dil Fusion is performed to obtain an equivalent bias b combined , the equivalent bias b combined for: b combined =b std +b dw +b pw +b dil ; Through the re-parameterization technique, the parallel convolution structure in the training phase is compressed into a single standard convolution structure.

Images