A neural network structure, a feature extraction method, a device and a medium

Abstract

The application discloses a neural network structure, a feature extraction method, a device and a medium, and the method comprises the following steps: a first convolution layer with a convolution kernel size of (3*N)*N is used to perform convolution on an input feature map, and then a first activation layer is used for processing, so that a first feature map is obtained; a second convolution layer with a convolution kernel size of N*(3*N) is used to perform convolution on the first feature map, and then a first batch normalization layer and a second activation layer are used for processing, so that a second feature map is obtained; a third convolution layer with a convolution kernel size of (3*N)*N is used to perform convolution on the second feature map, and then a third activation layer is used for processing, so that a third feature map is obtained; a fourth convolution layer with a convolution kernel size of N*(3*N) is used to perform convolution on the third feature map, and then a second batch normalization layer and a fourth activation layer are used for processing, so that a fourth feature map is obtained; the fourth feature map is subjected to upsampling processing, and then pixel category prediction is performed to output a semantic segmentation map. The method improves the segmentation precision of a target with an aspect ratio that is not 1:1.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of feature extraction, and particularly relates to a neural network structure, a feature extraction method, a device and a medium. BACKGROUND

[0002] In the field of border security, fences are widely used as boundary dividers, so fence detection and recognition are crucial for security systems.

[0003] Currently, traditional deep learning segmentation algorithms are mostly used for fence detection, but when designing convolution kernels, a convolution kernel with equal length and width is mostly used to extract fence features. However, a fence is composed of multiple slender metal objects, and the aspect ratio of the fence appearance is not 1:1. Therefore, when using a traditional convolutional neural network to extract features for segmentation, the receptive field does not match the actual object, and details of the fence may not be completely extracted. SUMMARY

[0004] To solve the problem of low segmentation accuracy of existing methods for objects with an aspect ratio other than 1:1, the application provides a neural network structure, a feature extraction method, a device and a medium, which improves the segmentation accuracy of objects with an aspect ratio other than 1:1.

[0005] The object of the application is achieved by the following technical solutions:

[0006] The first aspect of the application provides a neural network structure, comprising:

[0007] a convolution module, the convolution module comprising a first convolution layer, a first activation layer, a second convolution layer, a first batch normalization layer, a second activation layer, a third convolution layer, a third activation layer, a fourth convolution layer, a second batch normalization layer and a fourth activation layer connected in sequence, the convolution kernel size of the first convolution layer and the third convolution layer being (3*N)*N, the convolution kernel size of the second convolution layer and the fourth convolution layer being N*(3*N), N being a natural number greater than or equal to 1;

[0008] an upsampling module, the upsampling module performing an upsampling operation on the output of the convolution module;

[0009] a point convolution module, the point convolution module performing pixel category prediction on the output feature map of the upsampling module to output a semantic segmentation map.

[0010] In a possible design, the convolution kernel size of the first convolution layer and the third convolution layer is 3*1, and the convolution kernel size of the second convolution layer and the fourth convolution layer is 1*3.

[0011] In a possible design, the device further comprises a down-sampling module connected before the convolution module.

[0012] In a possible design, the downsampling module comprises:

[0013] a fifth convolutional layer, a convolutional kernel size of the fifth convolutional layer being (3*N)*(3*N);

[0014] a pooling layer;

[0015] a connection layer, the connection layer being configured to perform feature connection on output results of the fifth convolutional layer and the pooling layer;

[0016] a third batch of normalization layers, the output of the connection layer being subjected to normalization processing;

[0017] a fifth activation layer, the fifth activation layer being configured to perform nonlinear transformation on output results of the third batch of normalization layers.

[0018] In a possible design, a convolutional kernel size of the point convolution module is 1*1, and a step size is 1.

[0019] The second aspect of the present application provides a feature extraction method, comprising the following steps:

[0020] a first convolutional layer with a convolutional kernel size of (3*N)*N is used to perform convolution on an input feature map, and then a first activation layer is used to process the input feature map, so as to obtain a first feature map;

[0021] a second convolutional layer with a convolutional kernel size of N*(3*N) is used to perform convolution on the first feature map, and then a first batch of normalization layers and a second activation layer are used to process the first feature map, so as to obtain a second feature map;

[0022] a third convolutional layer with a convolutional kernel size of (3*N)*N is used to perform convolution on the second feature map, and then a third activation layer is used to process the second feature map, so as to obtain a third feature map;

[0023] a fourth convolutional layer with a convolutional kernel size of N*(3*N) is used to perform convolution on the third feature map, and then a second batch of normalization layers and a fourth activation layer are used to process the third feature map, so as to obtain a fourth feature map;

[0024] the fourth feature map is subjected to upsampling processing, and then pixel categories are predicted to output a semantic segmentation map.

[0025] In a possible design, before the first convolutional layer with a convolutional kernel size of (3*N)*N is used to perform convolution on the input feature map, and then the first activation layer is used to process the input feature map, the following steps are further included:

[0026] the input feature map is subjected to downsampling processing.

[0027] In a possible design, the downsampling processing on the input feature map comprises:

[0028] The input feature map is respectively subjected to convolution and pooling processing by a fifth convolution layer and a pooling layer, and the convolution kernel size of the fifth convolution layer is (3*N)*(3*N);

[0029] The features after convolution and pooling are connected to obtain a fifth feature map.

[0030] The fifth feature map is subjected to batch normalization and nonlinear transformation processing.

[0031] The third aspect of the present application provides a feature extraction device, comprising a memory and a controller which are sequentially connected in communication, the memory stores a computer program, and the controller is used for reading the computer program and executing the feature extraction method in the second aspect and any possible design thereof.

[0032] The fourth aspect of the present application provides a computer readable storage medium, which stores instructions, when the instructions are run on a computer, the feature extraction method in the second aspect and any possible design thereof is executed.

[0033] Compared with the prior art, the present application has at least the following advantages and beneficial effects:

[0034] The present application optimizes the convolution layer of the neural network, effectively improves the problem of segmentation error of the detail part in the target segmentation process with a horizontal and vertical ratio of 1:1, and effectively improves the segmentation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0036] Figure 1 The principle block diagram of the down-sampling module of the present application is shown in the figure.

[0037] Figure 2 The principle block diagram of the convolution module of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0039] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0040] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0041] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0042] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship commonly understood by those skilled in the art, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0043] In the description of the present application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "arrangement", "installation", "connection", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0044] The first aspect of the present application provides a neural network structure, comprising a down-sampling module, a convolution module, an up-sampling module and a point convolution module.

[0045] The downsampling module is a front-end module of the entire network structure, responsible for feature extraction of the image while reducing the dimension of the image, so that the processing efficiency of the downstream module is higher, which is an optimization module. Specifically, as shown in Figure 1 The downsampling module includes a fifth convolutional layer 11, a pooling layer 12, a connection layer 13, a third batch normalization layer 14, and a fifth activation layer 15.

[0046] The convolution kernel size of the fifth convolutional layer is (3*N)*(3*N), and N is a natural number greater than or equal to 1. Preferably, N is 1 or 2. The connection layer 13 is used to connect the output results of the fifth convolutional layer 11 and the pooling layer 12; the output of the connection layer is normalized. The fifth activation layer is used to perform nonlinear transformation on the output results of the third batch normalization layer.

[0047] The convolution module is composed of horizontal and vertical convolution of feature maps. As shown in Figure 2 The convolution module includes a first convolutional layer 211, a first activation layer 221, a second convolutional layer 212, a first batch normalization layer 231, a second activation layer 222, a third convolutional layer 213, a third activation layer 223, a fourth convolutional layer 214, a second batch normalization layer 232, and a fourth activation layer 224 connected in sequence, the convolution kernel size of the first convolutional layer 211 and the third convolutional layer 213 is (3*N)*N, and the convolution kernel size of the second convolutional layer 212 and the fourth convolutional layer 214 is N*(3*N). The convolution kernel size of the first convolutional layer 211 and the third convolutional layer 213 is 3*1 or 6*2; the convolution kernel size of the second convolutional layer 212 and the fourth convolutional layer 214 is 1*3 or 2*6.

[0048] The convolution module has an advantage in processing the appearance contour of a long strip, the aspect ratio of the convolution kernel is closer to the actual aspect ratio of the object, and in addition, the module also introduces an inflation coefficient, which expands the receptive field of the convolution kernel of the convolution module. Through the convolution module, the spatial features of the image are further extracted after downsampling, and the receptive field of the entire neural network structure is expanded by combining different convolution kernels.

[0049] The upsampling module performs upsampling operation on the output of the convolution module. For a feature map, there is a big gap between the pixel size and the original image. In this paper, an upsampling network module is designed to sample the encoded feature map, so that the size difference between the feature map and the original image is reduced.

[0050] The point convolution module predicts the pixel category of the output feature map of the upsampling module to output a semantic segmentation map. Preferably, the convolution kernel size of the point convolution module is 1*1, and the step is 1.

[0051] Based on the neural network structure described above, the second aspect of the present application provides a feature extraction method, which is suitable for extracting features of a target with an aspect ratio other than 1:1, such as a fence. Specifically, the method comprises steps S01 to S03.

[0052] Step S01, the input feature map is down-sampled to reduce the dimension of the input feature map and improve the efficiency of subsequent target extraction. After two down-sampling operations, the down-sampling ratio is 8 times, and the processed feature map F down .

[0053] Specifically, the input feature map input is first convolved and pooled using a fifth convolutional layer and a pooling layer, respectively, and the convolution kernel size of the fifth convolutional layer is (3*N)*(3*N). Then, the features after convolution and pooling are connected to obtain a fifth feature map F down_cat ; and finally, the fifth feature map F down_cat is processed by batch normalization and nonlinear transformation of a fifth activation layer to obtain a feature map F down .

[0054] Step S02, neck convolution processing.

[0055] Specifically, this step first uses a first convolutional layer with a convolution kernel size of (3*N)*N to convolve the feature map F down , and then processes it through a first activation layer to obtain a first feature map F conv3x1_1 . In this embodiment, N is taken as 1 for illustration.

[0056] Then, a second convolutional layer with a convolution kernel size of 1*3 is used to convolve the first feature map F conv3x1_1 , and then it is processed through a first batch normalization layer and a second activation layer to obtain a second feature map F conv1x3_1 . Then, a third convolutional layer with a convolution kernel size of 3*1 is used to convolve the second feature map F conv1x3_1 , and then it is processed through a third activation layer to obtain a third feature map F conv3x1_2 . Finally, a fourth convolutional layer with a convolution kernel size of 1*3 is used to convolve the third feature map F conv3x1_2 , and then it is processed through a second batch normalization layer and a fourth activation layer to obtain a fourth feature map F neck .

[0057] Step S03, the fourth feature map is up-sampled and then the pixel category is predicted to output a semantic segmentation map.

[0058] In a third aspect, the present application provides a feature extraction device, comprising a memory and a controller connected in sequence, wherein the memory stores a computer program, and the controller is configured to read the computer program and execute the feature extraction method in the second aspect or any possible design thereof. Specifically, the memory can include, but is not limited to, a random-access memory (RAM), a read-only memory (ROM), a flash memory, a first-in first-out memory (FIFO), a first-in last-out memory (FILO), and the like; and the controller can be a microcontroller of the STM32F105 series. In addition, the computer device can further include, but is not limited to, a power unit, a display screen, and other necessary components.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and the instructions, when executed on a computer, perform the feature extraction method in the second aspect or any possible design thereof.

[0060] The device and the medium provided in the third and fourth aspects operate according to the method in the second aspect, and thus details are not repeated here.

[0061] Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, and any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A neural network structure, characterized in that, include: The convolution module includes a first convolutional layer, a first activation layer, a second convolutional layer, a first batch normalization layer, a second activation layer, a third convolutional layer, a third activation layer, a fourth convolutional layer, a second batch normalization layer, and a fourth activation layer connected in sequence. The kernel size of the first and third convolutional layers is (3*N)*N, and the kernel size of the second and fourth convolutional layers is N*(3*N), where N is a natural number greater than or equal to 1. An upsampling module, which performs an upsampling operation on the output of the convolution module; A point convolution module is used to predict pixel categories in the output feature map of the upsampling module to output a semantic segmentation map.

2. The neural network structure according to claim 1, characterized in that: The kernel size of the first and third convolutional layers is 3*1, and the kernel size of the second and fourth convolutional layers is 1*3.

3. The neural network structure according to claim 1, characterized in that: It also includes a downsampling module connected before the convolution module.

4. A neural network structure according to claim 1, characterized in that, The downsampling module includes: The fifth convolutional layer has a kernel size of (3*N)*(3*N). Pooling layer; A connection layer is used to perform feature connections on the outputs of the fifth convolutional layer and the pooling layer; a third batch normalization layer is used to normalize the output of the connection layer. The fifth activation layer is used to perform a nonlinear transformation on the output of the third batch of normalization layers.

5. A neural network structure according to claim 1, characterized in that: The point convolution module has a kernel size of 1*1 and a stride of 1.

6. A feature extraction method, characterized in that, Includes the following steps: The first feature map is obtained by convolving the input feature map with a first convolutional layer of kernel size (3*N)*N and then processing it through the first activation layer. The first feature map is convolved by a second convolutional layer with a kernel size of N*(3*N), and then processed by the first batch of normalization layers and the second activation layer to obtain the second feature map. The second feature map is convolved by a third convolutional layer with a kernel size of (3*N)*N, and then processed by a third activation layer to obtain the third feature map. The third feature map is convolved by a fourth convolutional layer with a kernel size of N*(3*N), and then processed by a second batch normalization layer and a fourth activation layer to obtain the fourth feature map. The fourth feature map is upsampled and then pixel category is predicted to output a semantic segmentation map.

7. The feature extraction method according to claim 6, characterized in that, The process involves convolving the input feature map using a first convolutional layer with a kernel size of (3*N)*N, followed by processing through a first activation layer. Prior to this, the process also includes: The input feature map is downsampled.

8. The feature extraction method according to claim 7, characterized in that: The downsampling process of the input feature map includes: The input feature map is processed by convolution and pooling layers respectively using the fifth convolutional layer and the pooling layer. The kernel size of the fifth convolutional layer is (3*N)*(3*N). The convolutional and pooling features are concatenated to obtain the fifth feature map; Batch normalization and nonlinear transformation are performed on the fifth feature map.

9. A feature extraction apparatus, comprising a memory and a controller connected in sequence, wherein the memory stores a computer program, characterized in that: The controller is used to read the computer program and execute a feature extraction method according to any one of claims 6-8.

10. A computer-readable storage medium storing instructions thereon, characterized in that: When the instructions are executed on a computer, a feature extraction method according to any one of claims 6-8 is performed.

Images