A convolutional neural network feature extraction method based on a dynamic receptive field mechanism
The convolutional neural network feature extraction method based on the dynamic receptive field mechanism solves the problem that convolutional neural networks cannot effectively capture contextual information in feature extraction, and improves feature expression and target detection performance.
Patent Information
- Application Number
- CN202211424593.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-15
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2042-11-15
AI Technical Summary
Existing convolutional neural networks cannot effectively capture multi-level and comprehensive contextual information in feature extraction, resulting in limited performance in object detection and instance segmentation tasks, and increasing the kernel size will increase computational complexity and memory usage.
A convolutional neural network feature extraction method based on the dynamic receptive field mechanism is adopted. The relative distances between feature points are divided into different levels, weight values are assigned, the weight coefficient matrix is calculated, and a fixed convolution kernel is combined to perform weighted averaging of feature points and local response output, finally obtaining a global guided response output.
It improves the feature extraction capability of neural networks and the performance of computer vision tasks, especially in target detection tasks, enhancing feature expression and detection effects.
Smart Images
Figure CN115631389B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of computer vision and image processing technology based on neural network technology, and particularly relates to a convolutional neural network feature extraction method based on a dynamic receptive field mechanism. BACKGROUND
[0002] A large number of biological experiments have proved that the receptive field of the primary visual cortex of animals is dynamic and can be changed by the environment. A cell can be activated by a simple stimulus (such as a directional line segment or an edge in the visual field area), but the response can also be modulated by a stimulus located outside the receptive field, which means that the perception of the cell to the local stimulus is affected by the context of the presented features. However, in the CNN network, the response of a node only comes from a fixed receptive field area which is determined by the kernel size of the convolution layer or the pool layer. This is a simple simulation of the receptive field of the visual neuron, which is not sufficient to capture multi-level and comprehensive context information. Intuitively, a large receptive field is more conducive to target detection and instance segmentation tasks, because the integrated information from a large neighborhood can be used to learn the relationship between the target and the context. However, a convolution operation or a pooling layer operation only processes a local neighborhood. Directly increasing the kernel size will greatly increase the computational complexity and memory occupation, so long-range dependencies can only be captured by repeated local operations (increasing the network depth), which may introduce optimization difficulties and gradient disappearance and other shortcomings. The literature "C. Szegedy et al., "Going deeper with convolutions," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1-9" proposes a classic Inception network, which tries to expand the receptive field by parallel branches of different kernel sizes and increase the network width to achieve diversified information extraction, but the network structure needs to be designed manually and the computational complexity is huge. SUMMARY
[0003] To solve the above technical problems, the application provides a convolutional neural network feature extraction method based on a dynamic receptive field mechanism.
[0004] The technical scheme adopted by the application is as follows:
[0005] S1, first, the relative distances between the feature points on the input feature map F are divided into N different levels, and a weight value trained by a neural network is allocated to each level to obtain a weight vector;
[0006] S2, for each feature point on each spatial position in the feature map F of step S1, calculate the relative distance between the feature point and other surrounding feature points, map the calculated relative distance into N different levels, and assign a corresponding weight value learned by step S1 to all feature points on each level, thereby obtaining a weight coefficient matrix of each feature point on the entire feature map relative to its surrounding feature points;
[0007] S3, for each feature point on the feature map F, multiply the weight coefficient matrix obtained in step S2 by the weighted average value calculated by the feature map to obtain the excitation output of the feature point, thereby obtaining the global guided response output G of the feature point based on the relative position relationship;
[0008] S4, for each feature point on the feature map F, input the feature point to a convolution layer with a fixed convolution kernel to obtain the local response output L of the feature point, and finally add the global guided response output G obtained in step S3 and the local response output L to obtain the final output result of the feature point.
[0009] Further, in step S1, the following is specifically performed:
[0010] Taking a feature point f(x1, y1) on the feature map F as an example, x1 and y1 represent the spatial position of the feature point f. h(x other ,y other ) represents other feature points on the feature map F, and x other ,y other represents the spatial position of the other feature points. The relative spatial distance between the feature point f and the other feature points h is calculated as follows:
[0011]
[0012] where H and W represent the length and width of the feature map, respectively, and the maximum value of the relative distance calculated according to formula (1) is dist max . The maximum relative distance dist max is divided into N different levels, and a weight value ω learned by the neural network gradient backpropagation algorithm is assigned to each level. The N weight values form a vector R N , which represents a vector with a length of N.
[0013] Further, in step S2, the following is specifically performed:
[0014] For each feature point f on each spatial position in the feature map F, the relative distance between the feature point and other surrounding feature points h is calculated according to formula (1), and the calculated relative distance is mapped into N different levels. The weight value assigned to each level is calculated as follows:
[0015] p(f, h) = ω k if(k·dist max ) / N≤dist(f,h)<(k+1)·(dist max / N),k∈[0,1,...,N] (2)
[0016] where ω k denotes the weight value corresponding to the k-th level of the weight vector , and p(f, h) denotes the weight coefficient between feature points f and h based on relative position.
[0017] According to formula (2), the weight value ω k in the weight vector is assigned to each feature point on each level, and the weight coefficient matrix P of each feature point relative to its surrounding feature points on the entire feature map is obtained. Finally, the weight coefficient matrix is normalized by a softmax function to obtain a weight coefficient matrix S with a value of [0, 1]:
[0018]
[0019] where s(f, h) denotes the normalized weight coefficient between feature points f and h.
[0020] Further, in step S3, the following is specifically performed:
[0021] For an input feature map F ∈ R C*H*W , C denotes the number of channels, and the feature map F is first input into a 1 × 1 convolution layer W v :
[0022]
[0023] For each feature point in the input feature map F, the weight coefficient matrix S obtained based on step S2 is used to calculate the product of the feature map V and the weight coefficient matrix S, so as to obtain the excitation output o = S T V of the feature point after weighted average of the calculated feature map V and the weight coefficient matrix S. Wherein, denotes convolution operation, and T denotes matrix transposition.
[0024] In actual operation, the above steps S1-S3 are performed for each feature point in the feature map, and finally the output O ∈ R C*H*W of the entire feature map is obtained. Finally, O ∈ R C*H*W is input into a 1 × 1 convolution layer W w to obtain the global guide response output
[0025] Further, in the step S4, the convolutional neural network feature extraction method includes a convolutional neural network module based on a biological visual dynamic receptive field mechanism, specifically as follows:
[0026] In order to obtain the global guide response while preserving the local context information, the input feature map F is input into a convolutional layer with a fixed convolution kernel W C*H*W l , to obtain the local response output L.
[0027] The final output result of the feature map after the convolutional neural network module is the global guide response output G obtained in step S3 plus the obtained local response output L.
[0028] The method of the present application first divides the relative distance between the feature points on the feature map into N different levels, respectively allocates a weight value to each level, calculates the relative distance between each feature point and its surrounding feature points, divides it into N different levels, allocates a corresponding weight value to each level, obtains the weight coefficient matrix of all other feature points relative to the feature point, and then multiplies the feature map to obtain the excitation output of the feature point, obtains the global guide response output of each feature point based on the relative position relationship, inputs the feature map into a convolutional layer with a fixed convolution kernel to obtain the local response output of the feature map, and adds the global guide response output to obtain the final output result of the feature point. The method of the present application is applied to feature extraction in computer vision tasks, and improves the feature extraction, expression ability of the neural network and the performance of subsequent computer vision tasks. BRIEF DESCRIPTION OF DRAWINGS
[0029] Figure 1 The flowchart of the convolutional neural network feature extraction method based on the dynamic receptive field mechanism of the present application.
[0030] Figure 2 The visualization result graph of the weight coefficient matrix of a certain feature point in the embodiment of the present application.
[0031] Figure 3 The schematic diagram of the 0th channel of the global guide response output G in the embodiment of the present application.
[0032] Figure 4 The detailed calculation principle diagram of the convolutional neural network module in the embodiment of the present application.
[0033] Figure 5 The final detection effect example graph of the convolutional neural network module applied to the target detection task in the embodiment of the present application. DETAILED DESCRIPTION
[0034] The present invention will be further described below with reference to the accompanying drawings and examples.
[0035] Through a target detection task, a convolutional neural network feature extraction method based on a dynamic receptive field mechanism of the present invention is used to achieve the purpose of target detection. In this embodiment, a classic target detection framework structure is adopted, and the image (airport scene data set) is input into a feature extraction network comprising 4 stages. Each stage outputs a feature map of different resolutions. The input image size is adjusted to (3,320,320), and the feature extraction network Resne50 is used to obtain feature maps of R4 (2048, 10, 10), R3 (1024, 20, 20), R2 (512, 40, 40), and R1 (256, 80, 80) sizes output at different stages. These feature maps are input into an FPN network to obtain multiple detection frames for target detection, and then these detection frames are input into the NMS module, and the overlapping prediction frames are deleted to obtain the final output result of the network.
[0036] In this embodiment, a convolutional neural network feature extraction method based on a dynamic receptive field mechanism includes a convolutional neural network module (abbreviated as a DRF module) based on a biological visual dynamic receptive field mechanism, which is embedded in the feature extraction network Resne50. The embedding position can be different stages of the feature extraction network to enhance the representation ability of the feature extraction network and thereby improve the effect of target detection. In this embodiment, the DRF module is embedded in the third stage of the feature extraction network to process the intermediate feature map of the third stage.
[0037] like Figure 1 As shown in FIG, a flow chart of a convolutional neural network feature extraction method based on a dynamic receptive field mechanism of the present invention, the specific steps are as follows:
[0038] S1. First, the relative distances between feature points on the input feature map F are divided into N different levels, and a weight value obtained by neural network training is assigned to each level to obtain a weight vector;
[0039] S2. For each feature point at a spatial position in the feature map F of step S1, calculate the relative distance between the feature point and other surrounding feature points, map the calculated relative distances into N different levels, and assign corresponding weight values learned in step S1 to all feature points at each level, thereby obtaining a weight coefficient matrix for each feature point on the entire feature map relative to its surrounding feature points;
[0040] S3, for each feature point on the feature map F, multiply the weight coefficient matrix obtained in step S2 by the weighted average value calculated from the feature map to obtain the excitation output of the feature point, thereby obtaining the global guided response output G of the feature point based on the relative position relationship;
[0041] S4, for each feature point on the feature map F, input the feature point into a convolutional layer with a fixed convolutional kernel to obtain the local response output L of the feature point, and finally add the global guided response output G obtained in step S3 and the local response output L to obtain the final output result of the feature point.
[0042] In this embodiment, in step S1, the following is specifically performed:
[0043] The image input feature extraction network takes the feature map F generated in the third stage of the feature extraction network (the size of the feature map is 1024x20x20) as the input of the convolutional neural network module based on the biological visual dynamic receptive field mechanism.
[0044] Taking a feature point f(x1, y1) on the feature map F as an example, x1 and y1 represent the spatial position of the feature point f. h(x other ,y other ) represents other feature points on the feature map F, and x other ,y other represents the spatial position of the other feature points. The relative spatial distance between the feature point f and the feature point h is calculated as follows:
[0045]
[0046] where H and W represent the length and width of the feature map, respectively, and in this embodiment, 20x20. The maximum value of the spatial relative distance between the feature points in the feature map is set to The maximum relative distance is divided into N different levels, and each level is assigned a weight value ω learned by the neural network gradient backpropagation algorithm. The N weight values form a vector R N , which represents the vector length N.
[0047] In this embodiment, in step S2, the following is specifically performed:
[0048] For each feature point f at each spatial position on the feature map F, the relative distance between the feature point and the surrounding other feature points h is calculated according to formula (1), and the calculated relative distance is mapped into N different levels. The weight value assigned to each level is as follows:
[0049] p(f, h) = ω k if(k·dist max) / N≤dist(f,h)<(k+1)·(dist max / N),k∈[0,1,...,N] (2)
[0050] where ω k denotes the weight value corresponding to the kth level of the weight vector .
[0051] According to formula (2), the weight value ω in the weight vector k is assigned to each feature point on each level, so as to obtain the weight coefficient matrix P of each feature point on the entire feature map relative to its surrounding feature points. Finally, the weight coefficient matrix is normalized by a softmax function to obtain a weight coefficient matrix S∈R 20×20 :
[0052]
[0053] where s(f,h) denotes the normalized weight coefficient between feature points f and h.
[0054] Figure 2 is the visualization result of the weight coefficient matrix S generated in step S2 for a certain feature point.
[0055] In the embodiment, in step S3, the following is specifically performed:
[0056] For an input feature map F∈R C*H*W , C denotes the number of channels, and first, the feature map F is input into a 1×1 convolutional layer W v :
[0057]
[0058] For each feature point in the input feature map F, based on the weight coefficient matrix S obtained in step S2, the product of the feature map V and the weight coefficient matrix S is calculated, so as to obtain the excitation output o=S T V of the feature point after the weighted average of the calculated feature map V and the weight coefficient matrix S. Wherein, denotes convolution operation, and T denotes matrix transposition.
[0059] The calculation of steps S1-S3 is for a feature point in the feature map. In actual operation, the above steps S1-S3 are performed for each feature point in the feature map, and finally the output O∈R C*H*W of the entire feature map is obtained. Finally, O∈R C*H*W is input into a 1×1 convolutional layer W w, to obtain the global guided response output of the feature map
[0060] Figure 3 A schematic diagram of the 0thchannel of the global guided response output G obtained in step S3 is shown in FIG. 3.
[0061] In the present embodiment, the convolutional neural network feature extraction method in step S4 includes a convolutional neural network module based on a biological visual dynamic receptive field mechanism, which is specifically as follows:
[0062] To obtain the global guided response while preserving the local context information, the input feature map F∈R C*H*W A convolutional layer W with a fixed convolution kernel is passed in l , to obtain the local response output
[0063] The final output result of the last feature map after being processed by the convolutional neural network module is the global guided response output G obtained in step S3 plus the obtained local response output L.
[0064] Figure 4 Detailed calculation principle of the convolutional neural network module in the complete convolutional neural network feature extraction method based on a dynamic receptive field mechanism Figure .
[0065] The features processed by the convolutional neural network module based on a biological visual dynamic receptive field mechanism are continuously input to the subsequent feature extraction network. The feature maps output at different stages are input to the FPN network to obtain multiple detection boxes for target detection. These detection boxes are input to the NMS module to delete overlapping prediction boxes, and finally the output is obtained. Figure 5 The detection effect of the neural network model embedded with the convolutional neural network module based on a biological visual dynamic receptive field mechanism on the input image is shown.
[0066] Those skilled in the art will realize that the embodiments described herein are for the purpose of helping the reader understand the use of the present application and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations of the present application without departing from the spirit of the present application, and these modifications and combinations are still within the scope of protection of the present application.
Claims
1. A convolutional neural network feature extraction method based on a dynamic receptive field mechanism, the specific steps are as follows: S1. First, the relative distances between feature points on the input feature map F are divided into N different levels, and a weight value obtained by neural network training is assigned to each level to obtain a weight vector; in, The input feature map data is image data; S2. For each feature point at a spatial position in the feature map F of step S1, calculate the relative distance between the feature point and other surrounding feature points, map the calculated relative distances into N different levels, and assign corresponding weight values learned in step S1 to all feature points at each level, thereby obtaining a weight coefficient matrix for each feature point on the entire feature map relative to its surrounding feature points; S3. For each feature point on the feature map F, multiply the weight coefficient matrix obtained in step S2 above by the weighted average value calculated from the feature map as the excitation output of the feature point, thereby obtaining the global guidance response output G of the feature point based on the relative position relationship; S4. For each feature point on the feature map F, the feature point is input into a convolution layer with a fixed convolution kernel to obtain the local response output L of the feature point. Finally, the global guided response output G obtained in step S3 and the local response output L are added to obtain the final output result of the feature point.
2. The convolutional neural network feature extraction method based on the dynamic receptive field mechanism according to claim 1 is characterized in that: In the step S1, the details are as follows: Take a feature point f(x1,y1) on the feature map F as an example, x1,y1 represents the spatial position of the feature point f, h(x other ,y other ) represents other feature points on the feature map F, x other ,y other Represents the spatial position of other feature points; the relative spatial distance between feature points f and h is calculated as follows: Among them, H and W represent the length and width of the feature map respectively. The maximum value of the relative distance calculated according to formula (1) is dist max ; Set the maximum relative distance dist max Divide into N different levels, and assign each level a weight value ω learned by the neural network gradient back propagation algorithm. The N weight values constitute the vector R N Indicates that the vector length is N.
3. The convolutional neural network feature extraction method based on the dynamic receptive field mechanism according to claim 2 is characterized in that: In the step S2, the details are as follows: For each feature point f at a spatial position in the feature map F, the relative distance between the feature point and the surrounding feature points h is calculated according to formula (1), and the calculated relative distance is mapped into N different levels; the weight value assigned to each level is calculated as follows: p(f,h)=ω k if(k·dist max ) / N≤dist(f,h)<(k+1)·(dist max / N),k∈[0,1,...,N](2) Among them, ω k Represents the weight vector The weight value corresponding to the kth level, p(f,h) represents the weight coefficient between feature points f and h based on relative position; According to formula (2), all feature points at each level are assigned corresponding weight vectors The weight value ω in k , get the weight coefficient matrix P of each feature point on the entire feature map relative to its surrounding feature points; finally, normalize the weight coefficient matrix through a softmax function to get a weight coefficient matrix S with a value of [0,1]: Among them, s(f,h) represents the normalized weight coefficient between feature points f and h.
4. The convolutional neural network feature extraction method based on the dynamic receptive field mechanism according to claim 3 is characterized in that: In step S3, the details are as follows: For an input feature map F∈R C*H*W , C represents the number of channels. First, the feature map F is passed into a 1×1 convolution layer W v : For each feature point in the input feature map F, based on the weight coefficient matrix S obtained in step S2, the product of the feature map V and the weight coefficient matrix S is calculated to obtain the excitation output o=S after the weighted average of the feature map V and the weight coefficient matrix S. T V; among them, Represents the convolution operation, T represents the transpose of the matrix; In the actual operation, the above steps S1-S3 are performed on each feature point in the feature map, and finally the output O∈R of the entire feature map is obtained. C*H*W ;Finally, O∈R C*H*W Pass in a 1×1 convolutional layer W w , get the global guided response output of the feature map 5. The convolutional neural network feature extraction method based on the dynamic receptive field mechanism according to claim 4 is characterized in that: In step S4, the convolutional neural network feature extraction method includes a convolutional neural network module based on the biological visual dynamic receptive field mechanism, specifically as follows: In order to obtain the global guidance response while retaining the local context information, the input feature map F∈R C*H*W Pass in a convolution layer W with a fixed convolution kernel l , get the local response output The final output result of the feature map after being processed by the convolutional neural network module is the global guidance response output G obtained in step S3 plus the local response output L obtained.