Lightweight neural memory ordinary differential equation cascade method based on YOLO V5

Abstract

The application discloses a light neural memory ordinary differential equation cascade method based on YOLO V5, which comprises the following steps: outputting feature maps with different resolutions by using a Backbone network, repeatedly fusing the feature maps between each NmODE module through the cascade of the NmODE module, and sufficiently integrating the feature maps with different resolutions, so that fine local information is retained and the expression ability of global context is enhanced, the NmODE module not only receives the output of the previous module as the input, but also fuses the feature maps of the Backbone network, so that the feature expression is further enriched in each cascade, and in addition, after each NmODE module, the semantic features output by the NmODE are fused with the current input of the NmODE module in the current cascade through an enhanced aggregation EF module. This double-path fusion strategy not only enhances the detection ability of multi-scale target objects, but also significantly improves the detection accuracy through the interaction and fusion of features.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence neural networks, and in particular to a lightweight neural memory ordinary differential equation cascade method based on YOLO V5. BACKGROUND

[0002] Deep learning-based object detection tasks are widely used in intelligent security, autonomous driving, robot navigation, and medical diagnosis. Convolutional neural network (CNN) based detection networks have gradually developed the object detection framework from two stages (e.g., Faster RCNN, Mask RCNN, and Cascade RCNN) to one stage (e.g., YOLO). Among them, YOLO stands out with its small size, fast computing speed, and strong generalization ability. It converts the object detection problem into a regression problem, thereby achieving fast and efficient object detection, but the detection accuracy still needs to be improved. Subsequently, a large number of YOLO variants emerged, which seek to balance the detection accuracy and efficiency of the model. These variants can be divided into two categories: anchor-based (e.g., YOLOv3, YOLOv4, and YOLOv5) and anchor-free (e.g., YOLOv8 and YOLOX). These variants almost all contain the same parts: backbone, neck, and head.

[0003] Backbone network architecture has been widely studied (e.g., CSPDarknet, Swin Transformer, or ConvNeXt). Replacing the backbone network is a common method to improve YOLO series models. The neck in the YOLO series usually uses a feature pyramid network (FPN) and its variants to fuse multi-level features in different backbones. Then, the learned features are fed into an intuitive head architecture to predict the position information. However, although FPN and its variants alleviate the information fusion problem to some extent, previous models still have information fusion problems that hinder the detection accuracy of the YOLO series models to some extent. Therefore, a large number of studies focus on improving the neck module. Although these methods effectively improve multi-scale information fusion, they still increase the computational complexity of the original structure and ignore the expression ability of features in different datasets.

[0004] Therefore, the present application proposes a lightweight neural memory ordinary differential equation cascade method based on YOLO V5. SUMMARY

[0005] In order to overcome the shortcomings and deficiencies existing in the prior art, the present application provides a lightweight neural memory ordinary differential equation cascade method based on YOLO V5.

[0006] The method comprises a lightweight neural memory ordinary differential equation cascade method based on YOLO V5.

[0007] Step S1: input pictures are subjected to data enhancement and sent to a Backbone network, the Backbone network being composed of a Focus structure and a CSPDarknet53, different scales of YOLOV5 model Backbone networks having different network scales but outputting three feature maps of the same scale, respectively from a shallow layer, a middle layer and a deep layer network, and being named as P3, P4 and P5;

[0008] Step S2: an NmODE module based on a neural memory ordinary differential equation aggregates deep and shallow features by cascading four NmODE modules, taking the output of a single NmODE module as the h(0) value input of the next NmODE module, and the next NmODE module further aggregates the features output by the Backbone;

[0009] Step S3: a semantic feature output by the NmODE is fused with the current input of the NmODE by an enhanced aggregation EF module, the enhanced aggregation EF module adopting a double-path structure, namely a main path and an auxiliary path; the first main path fuses the semantic feature output by the NmODE with the current input of the NmODE; the second auxiliary path fuses the current input of the NmODE again, the main path and the auxiliary path convert the inputs into feature maps containing three dimensions of target categories, positions (bounding boxes) and confidence, and the prediction result of the auxiliary path is added to the prediction result of the main path to realize feature fusion and prediction correction;

[0010] Step S4: three enhanced aggregation EF modules output feature maps containing three dimensions of target categories, positions (bounding boxes) and confidence, and the three feature maps from left to right correspond to category, position (bounding box) and confidence prediction of middle, small and large targets respectively, and loss calculation is performed to realize network parameter updating.

[0011] Further, in the step S2, the neural memory ordinary differential equation is as follows:

[0012]

[0013] wherein, is a derivative function of the output h(k), represents a network hidden state, h(k) is a current state of the network, S is an activation function, f is a convolution operation, o(k) is a current input of the network, and θ K is a current parameter of the network.

[0014] Further, in the step S2, the first NmODE module is expressed as follows:

[0015] y1=NmODE(P4,Resize ↓ (P3))(2),

[0016] Among them, P4 is the current input. ↓ (·) refers to downsampling achieved through a 3*3 convolution operation with a stride of 2. After this downsampling, the resolution of P3 is reduced from 80*80 to 40*40, which is used as the input of h(0).

[0017] Furthermore, in step S2, the second NmODE module expression is as follows:

[0018] y2=NmODE(Resize ↑ (P5),y1)(3),

[0019] Among them, Resize ↑ (·) indicates the nearest neighbor interpolation layer, that is, P5 is upsampled and the resolution is expanded from 20*20 to 40*40 as the current input. y1 is used as the input of h(0). The output y2 of this module combines the features from three levels (P3, P4 and P5) and is then passed to the next NmODE module and the first enhanced aggregation EF module for further enhancement and fusion.

[0020] Furthermore, in step S2, the third NmODE module expression is as follows:

[0021] y3=NmODE(Resize ↑ (P4),Resize ↑ (y2))(4),

[0022] Among them, Resize ↑ (·) represents the nearest neighbor interpolation layer, that is, P4 and y2 are upsampled, the resolution of P4 is expanded from 40*40 to 80*80, as the current input, and the resolution of y2 is expanded from 40*40 to 80*80, as the input of h(0). The output is then passed to the fourth NmODE module and the second enhanced aggregation EF module for further enhancement and fusion.

[0023] Furthermore, in step S2, the fourth NmODE module expression is as follows:

[0024] y4=NmODE(P5,Resize ↓ (y3))(5),

[0025] Among them, P5 is the current input. ↓(·) refers to downsampling achieved through a 3*3 convolution operation with a stride of 4. After this downsampling, the resolution of y3 is reduced from 80*80 to 20*20, which is used as the input of h(0). The output of this module is then passed to the final enhancement aggregation EF module for further enhancement and feature fusion.

[0026] Furthermore, in step S3, the enhanced aggregation EF module fusion method is as follows:

[0027] The enhanced aggregation EF module adopts a dual-path structure, namely the main path and the auxiliary path; the main path strengthens the semantic features of the fusion NmODE output and the current input of NmODE; as shown in formula (6), the two features are extracted through a standard 3*3 convolution module to obtain a coarse feature map containing three dimensions: target category, position (bounding box) and confidence; the two feature maps are concatenated (concat) and the 1*1 convolution module is used to reduce the dimension of the fused feature map to obtain a feature map F containing more advanced semantic information. P ;

[0028] F P =Conv 1×1 {Concat[Conv 3×3 (S),Conv 3×3 (D)]}(6),

[0029] The second auxiliary path, as shown in formula (7), uses a 3*3 convolution module to transform the deep features of the backbone into a feature map F containing three dimensions: target category, location (bounding box) and confidence. a ;

[0030] F a =Conv 3×3 (D)(7),

[0031] The prediction results of the main path are added to the prediction results of the auxiliary path, as shown in formula (8) to achieve feature fusion and prediction correction;

[0032] F final =F P +F a (8).

[0033] Beneficial effects:

[0034] The application provides a lightweight neural memory ordinary differential equation cascade method based on YOLO V5, which realizes layer-by-layer feature fusion and improves detection accuracy by sequentially cascading multiple NmODE modules. In the method, the Backbone network outputs feature maps of different resolutions, which represent semantic information at different levels, covering detailed features at the shallow layer to abstract features at the deep layer. Through the cascade of NmODE modules, the feature maps are repeatedly fused between each NmODE module, and the feature maps of different resolutions are fully integrated, retaining fine local information and enhancing the expression ability of global context. The structure design of the NmODE module is simple, only including a convolution layer, which significantly reduces the computational complexity of the network while maintaining high-efficiency detection capability. Compared with traditional multi-layer convolutional networks, the lightweight design is suitable for embedded devices or other scenarios with limited computing resources, while still ensuring efficient detection of target objects. During the cascade process, the NmODE module not only receives the output of the previous module as input, but also fuses with the feature maps of the Backbone network, further enriching the feature expression at each level of cascade. In addition, after each NmODE module, the semantic features output by the NmODE are fused again with the current input of the current NmODE module through the enhanced aggregation EF module. This double-path fusion strategy not only enhances the detection capability of multi-scale target objects, but also significantly improves the detection accuracy through feature interaction and fusion. The double-path structure of the enhanced aggregation EF module effectively combines semantic features, spatial information and confidence information, ensuring excellent detection performance on different domain datasets. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The overall step flowchart of the application is shown in the figure. DETAILED DESCRIPTION

[0036] It should be noted that the embodiments and features in the embodiments in the present application can be combined with each other without conflict, and the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] As shown in the figure, the lightweight neural memory ordinary differential equation cascade method based on YOLO V5 includes: Figure 1

[0038] Step S1: The input picture is subjected to data enhancement and sent to the Backbone network, which is composed of a Focus structure and a CSPDarknet53. Different scale YOLOV5 models have different network scales of the Backbone, but output three feature maps of the same scale at different resolutions, respectively from the shallow, middle and deep networks, named P3, P4 and P5; ​

[0039] Specifically, the input image can include images of natural scenes, medical images, traffic monitoring images, and so on. Therefore, the input image can contain multiple objects. The input image is resized to 640*640 pixels. This allows the model to process images of varying sizes while maintaining a consistent input size during the feature extraction phase. The YOLO V5 backbone network consists of the Focus architecture and CSPDarknet53. The Focus architecture is a pre-processing module introduced in YOLO V5 specifically for preprocessing the input image. It slices the input image to convert its spatial information into richer channel information. This operation not only reduces the image size but also the computational effort. During the computational process, the Focus architecture slices each 2*2 pixel region of the input image into four channels (effectively compressing the spatial information into the channel dimension), thereby improving computational efficiency. For example, an input image of 640*640*3 pixels becomes 320*320*12 after passing through the Focus module (each 2*2 pixel block is converted into a feature channel, for a total of 12 channels). CSPDarknet53 is the core backbone architecture of YOLO V5. It is an improvement based on Darknet53 and incorporates the design of Cross Stage Partial Network (CSPNet). CSPNet achieves more efficient feature fusion by segmenting feature maps while reducing redundant computation. CSPDarknet53 consists of multiple residual blocks and extracts features at different levels through layer-by-layer depthwise convolution. Within each residual block, the CSP architecture divides the feature map into two parts: one directly connected via residual connections, and the other merged with the former after convolution. This structure effectively alleviates the vanishing gradient problem and enhances feature representation capabilities. YOLO V5's backbone network extracts three feature maps of different resolutions from the input image, representing shallow, mid-level, and deep semantic information, respectively. These feature maps are named P3 (shallow), P4 (mid-level), and P5 (deep). P3 has a resolution of 80*80 and 256 channels. It is mainly responsible for extracting local details in the image, such as edges and textures. This information is very important for identifying small objects and objects with rich details. P4 has a resolution of 40*40 and 512 channels. It is responsible for further extracting higher-level semantic information while maintaining a certain amount of spatial detail. It combines the detailed information of the shallow network with the abstract semantic information of the deep network. P5 has a resolution of 20*20 and 1024 channels. It mainly extracts global semantic information in the image, such as the shape and category of the object. Because the deep network has a larger receptive field, it can capture the global features of the entire object or even the scene, and is suitable for detecting large objects and more abstract targets.

[0040] Step S2: The NmODE module is based on the neural memory ordinary differential equation. By cascading four NmODE modules, the deep and shallow features are aggregated. The output of a single NmODE module is used as the h(0) value input of the next NmODE module. The next NmODE module then aggregates the features output by the backbone. Each module receives the output of the previous module and fuses it with the deep feature map.

[0041] Specifically, the first NmODE module is responsible for fusing shallow and middle-level features. The shallow feature P3 is used as the initial hidden state h(0). It is first down-sampled by a 3*3 convolution operation with a stride of 2. After the downsampling, the resolution of P3 is reduced from 80*80 to 40*40. After the 3*3 convolution operation, BatchNorm normalization and Si lu activation function processing, it is combined with the middle-level feature P4 through the solution process of the neural memory ordinary differential equation to generate the fused feature y1. Y1 is passed to the second NmODE module as its initial state; the second NmODE module is responsible for fusing y1 and the deep feature P5. P5 is up-sampled by the nearest neighbor interpolation method, and the resolution is expanded from 20*20 to 40*40. After the 3*3 convolution operation, BatchNorm normalization and Si lu activation function processing, it is combined with the middle-level feature P4 through the solution process of the neural memory ordinary differential equation to generate the fused feature y1. Y1 is passed to the second NmODE module as its initial state; the second NmODE module is responsible for fusing y1 and the deep feature P5. P5 is up-sampled by the nearest neighbor interpolation method, and the resolution is expanded from 20*20 to 40*40. After lu activation function processing, it is processed with y1 through the process of solving the neural memory ordinary differential equation to generate the fusion feature y2, and y2 is passed to the third NmODE module as its initial state; the third NmODE module is responsible for fusing y2 and the middle-level feature P4. P4 and y2 are up-sampled by the nearest neighbor interpolation method, and the resolution is expanded from 40*40 to 80*80. After 3*3 convolution operation, BatchNorm normalization and Si lu activation function processing, it is processed through the process of solving the neural memory ordinary differential equation to generate the fusion feature y3, and y3 is passed to the fourth NmODE module as its initial state; the fourth NmODE module is responsible for fusing y3 and deep feature P5. y3 first undergoes a 3×3 convolution operation with a stride of 4 to achieve downsampling. After the downsampling, the resolution of y3 is reduced from 80*80 to 20*20, and then undergoes 3*3 convolution operation, BatchNorm normalization and Si After being processed by the lu activation function, it is combined with y3 through the process of solving the neural memory ordinary differential equation to generate the fusion feature y4.

[0042] Step S3: The semantic features of the NmODE output and the current input of NmODE are fused through the enhanced aggregation EF module. The enhanced aggregation EF module adopts a dual-path structure, namely the main path and the auxiliary path. The first main path strengthens the fusion of the semantic features of the NmODE output and the current input of NmODE; the second auxiliary path fuses the current input of NmODE again. The main path and the auxiliary path convert the input into a feature map output containing three dimensions: target category, position (bounding box), and confidence. The prediction results of the auxiliary path are added to the prediction results of the main path to achieve feature fusion and prediction correction.

[0043] Specifically, the number of channels output by the three enhanced aggregation EF modules is the same. For a dataset with i categories, the output channel dimension is 3*(i+4+1), where 3 is the number of anchors, 4 is the relative coordinates of the bounding box, and 1 is the confidence score of the box. The main path of the first enhanced aggregation EF module fuses the output y2 of the second NmODE and its current input P5, and the auxiliary path fuses the current input of the second NmODE again. The main path and the auxiliary path convert y2 and P5 into feature map outputs containing three dimensions: target category, position (bounding box), and confidence. The prediction results of the auxiliary path are added to the prediction results of the main path to achieve feature fusion and prediction correction. The output resolution of this module is 40*40; the main path of the second enhanced aggregation EF module fuses the output y3 of the third NmODE and its current input P4, and the auxiliary path fuses the current input of the third NmODE again. The main path and the auxiliary path convert y3 and P4 The feature map output contains three dimensions: target category, position (bounding box) and confidence. The prediction results of the auxiliary path are added to the prediction results of the main path to achieve feature fusion and prediction correction. The output resolution of this module is 80*80; the main path of the third enhanced aggregation EF module fuses the output y4 of the fourth NmODE and its current input P5, and the auxiliary path fuses the current input of the fourth NmODE again. The main path and auxiliary path convert y4 and P5 into feature map outputs containing three dimensions: target category, position (bounding box) and confidence. The prediction results of the auxiliary path are added to the prediction results of the main path to achieve feature fusion and prediction correction. The output resolution of this module is 20*20.

[0044] Step S4: The three enhanced aggregation EF modules respectively output feature maps containing three dimensions: target category, position (bounding box) and confidence. The three feature maps from left to right correspond to the category, position (bounding box) and confidence prediction of medium, small and large targets, respectively, and perform loss calculation to update the network parameters.

[0045] Specifically, after the feature map is generated, the network calculates the corresponding loss function by comparing the predicted class, location and confidence with the real label, including classification loss, bounding box regression loss and confidence loss. Through back propagation, the loss is used for optimization and update of network parameters, so as to improve the detection performance of the model on different scale targets and ensure the accuracy of the network in class distinction, location prediction and confidence evaluation.

[0046] Further, the step S2, the neural memory ordinary differential equation expression is:

[0047]

[0048] wherein, is the derivative function of the output h(k), represents the network hidden state, h(k) is the current state of the network, S is the activation function, f is the convolution operation, o(k) is the current input of the network, θ K is the current parameter of the network.

[0049] Further, the step S2, the first NmODE module expression is:

[0050] y1=NmODE(P4,Resize ↓ (P3))(2),

[0051] wherein, P4 as the current input, Resize ↓ (·) means down-sampling through a 3*3 convolution operation with a stride of 2, P3 is down-sampled to a resolution of 40*40 from 80*80 as the input of h(0).

[0052] Further, the step S2, the second NmODE module expression is:

[0053] y2=NmODE(Resize ↑ (P5),y1)(3),

[0054] wherein, Resize ↑ (·) represents the nearest neighbor interpolation layer, that is, P5 is up-sampled to a resolution of 40*40 from 20*20 as the current input. y1 is input as h(0). The output y2 of this module combines the features from three levels (P3, P4 and P5), and then is transmitted to the next NmODE module and the first enhanced aggregation EF module for further enhancement and fusion.

[0055] Further, the step S2, the third NmODE module expression is:

[0056] y3=NmODE(Resize ↑(P4),Resize ↑ (y2))(4),

[0057] Among them, Resize ↑ (·) represents the nearest neighbor interpolation layer, that is, P4 and y2 are upsampled, and the resolution of P4 is expanded from 40*40 to 80*80 as the current input. The resolution of y2 is expanded from 40*40 to 80*80 as the input of h(0). The output is then passed to the fourth NmODE module and the second Enhanced Aggregation EF module for further enhancement and fusion.

[0058] Furthermore, in step S2, the fourth NmODE module expression is:

[0059] y4=NmODE(P5,Resize ↓ (y3))(5),

[0060] Among them, P5 is the current input. ↓ (·) refers to downsampling achieved through a 3*3 convolution operation with a stride of 4. After this downsampling, the resolution of y3 is reduced from 80*80 to 20*20, which is used as the input of h(0). The output of this module is then passed to the final enhancement aggregation EF module for further enhancement and feature fusion.

[0061] Furthermore, in step S3, the enhanced aggregation EF module fusion method is as follows:

[0062] The enhanced aggregation EF module adopts a dual-path structure, namely the main path and the auxiliary path; the main path strengthens the semantic features of the fusion NmODE output and the current input of NmODE; as shown in formula (6), the two features are extracted through a standard 3*3 convolution module to obtain a coarse feature map containing three dimensions: target category, position (bounding box) and confidence; the two feature maps are concatenated (concat) and the 1*1 convolution module is used to reduce the dimension of the fused feature map to obtain a feature map F containing more advanced semantic information. P ;

[0063] F P =Conv 1×1 {Concat[Conv 3×3 (S),Conv 3×3 (D)]}(6),

[0064] The second auxiliary path, as shown in formula (7), uses a 3*3 convolution module to transform the deep features of the backbone into a feature map F containing three dimensions: target category, location (bounding box) and confidence. a ;

[0065] F a = Conv 3×3 (D)(7),

[0066] The prediction result of the main path is added to the prediction result of the auxiliary path to realize feature fusion and prediction correction as in equation (8);

[0067] F final = F P + F a (8).

[0068] Although the embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various equivalents, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight neural memory ordinary differential equation cascade method based on YOLOV5, characterized by , the method includes: Step S1: The input image is data-enhanced and sent to the Backbone network. Backbone consists of Focus structure and CSPDarknet53. The network scales of the Backbone of YOLOV5 models of different scales are different, but they output three feature maps of different resolutions of the same scale, which come from the shallow, middle and deep layers of the network, respectively. They are named 、 and ; Step S2: The NmODE module is based on the neural memory ordinary differential equation. By cascading four NmODE modules, the deep and shallow features are aggregated, and the output of the current NmODE module is used as the output of the next NmODE module. The value is input, and the next NmODE module aggregates the features output by the backbone, where each module receives the output of the previous module and fuses it with the deep feature map; The first NmODE module expression is: in, As the current input, Refers to a step length of of The convolution operation implements downsampling. After this downsampling, the resolution is Reduce to , as enter; The second NmODE module expression is: in, represents the nearest neighbor interpolation layer, i.e. is upsampled, the resolution is increased from Expand to , as the current input, As Input, output of this module Combining three levels 、 and The features are then passed to the next NmODE module and the first enhanced aggregation EF module for further enhancement and fusion; The third NmODE module expression is: in, represents the nearest neighbor interpolation layer, i.e. and is upsampled, Resolution from Expand to , as the current input; Resolution from Expand to , as Input; the output is then passed to the fourth NmODE module and the second enhanced aggregation EF module for further enhancement and fusion; The fourth NmODE module expression is: in, As the current input, Refers to a step length of of The convolution operation implements downsampling. After this downsampling, the resolution is Reduce to , as Input; the output of this module is then passed to the last enhanced aggregation EF module for further enhancement and feature fusion; Step S3: The semantic features of the NmODE output and the current input of NmODE are fused through the enhanced aggregation EF module. The enhanced aggregation EF module adopts a dual-path structure, namely a main path and an auxiliary path. The first main path strengthens the fusion of the semantic features of the NmODE output and the current input of NmODE; the second auxiliary path fuses the current input of NmODE again. The main path and the auxiliary path convert the input into a feature map output containing three dimensions: target category, position, and confidence. The prediction results of the auxiliary path are added to the prediction results of the main path to achieve feature fusion and prediction correction. Step S4: The three enhanced aggregation EF modules respectively output feature maps containing three dimensions: target category, location, and confidence. The three feature maps from left to right correspond to the category, location, and confidence prediction of medium, small, and large targets, respectively, and loss calculation is performed to update the network parameters.

2. The lightweight neural memory ordinary differential equation cascade method based on YOLOV5 according to claim 1, characterized in that: In step S2, the neural memory ordinary differential equation expression is: in, Output h The derivative function represents the hidden state of the network, is the current state of the network, is the activation function, is the convolution operation, is the current input of the network, are the current parameters of the network.

3. The lightweight neural memory ordinary differential equation cascade method based on YOLOV5 according to claim 1, characterized in that: In step S3, the enhanced aggregation EF module fusion method is as follows: The enhanced aggregation EF module adopts a dual-path structure, which is a main path and an auxiliary path. The main path strengthens the semantic features of the NmODE output and the current input of the NmODE. Formula (6) specifically converts the two features into a standard The convolution module is used to extract features and obtain a rough feature map containing three dimensions: target category, position and confidence; the two feature maps are spliced ​​and used The convolution module reduces the dimension of the fused feature map to obtain a feature map containing higher-level semantic information. ; The second auxiliary path, Equation (7) uses a The convolution module transforms the deep features of the backbone into feature maps with three dimensions: target category, position, and confidence. ; The prediction results of the main path are added to the prediction results of the auxiliary path, and formula (8) realizes feature fusion and prediction correction;

Images