Neural network detection and recognition method and system for brain-like situation information processing

Abstract

The application provides a kind of neural network detection and identification method and system of brain-like situation information processing, the method comprises: skeleton network extracts the multi-level feature map with different receptive field from input image;FPN feature pyramid network obtains process feature map;Global semantic context information extraction network extracts global semantic context information;Global detail context information extraction network extracts global detail context information;Classification sub-network fuses global semantic context information and target classification feature according to process feature map, to obtain fused classification feature;Positioning sub-network fuses global detail context information and target positioning feature according to process feature map, to obtain fused positioning feature;Target classifier obtains target class label according to fused classification feature;Target locator obtains target coordinate value according to fused positioning feature.The application can solve the problem that the detection and identification method in the prior art cannot simultaneously meet high precision and high efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision image recognition technology, and in particular to a neural network detection and recognition method and system for brain-like contextual information processing. Background Technology

[0002] Deep learning is an important method for image object detection and recognition. Its main function is to accurately classify or locate objects within images or videos, and the final result is presented as a bounding box marking the objects in the image and providing their classification labels. Deep learning-based image object detection and recognition models generally consist of a feature extraction skeleton network, a multi-scale feature fusion network, an object classification subnetwork, and an object localization subnetwork.

[0003] However, existing deep learning models for image or video object detection and recognition suffer from a significant problem: poor context adaptability. Constrained by this issue, the performance of detection and recognition models degrades considerably when the number and types of objects in an image are large. Object detection and recognition performance depends on both the classification performance of the object classification sub-network and the localization performance of the object localization sub-network. However, object classification requires sufficient extraction of semantic information, while object localization requires relatively precise detailed information. Therefore, the inherent requirements for feature information differ between the classification and localization subtasks, and the lack of global contextual information prevents the effective separation of semantic and detailed features.

[0004] Current deep learning-based object detection and recognition models (such as YOLOX and ArXiv 2021) mostly use separate object classification sub-networks and object localization sub-networks to extract semantic and detail information from feature maps. However, this approach prevents the classification sub-network and localization sub-network from sharing weight parameters and network structure, thereby introducing more model parameters, increasing model computational complexity, and reducing model computational efficiency. Summary of the Invention

[0005] This invention provides a neural network detection and recognition method and system for brain-like contextual information processing, which can solve the technical problem that existing detection and recognition methods cannot simultaneously meet the requirements of high accuracy and high efficiency.

[0006] According to one aspect of the present invention, a neural network detection and recognition method for brain-like contextual information processing is provided, the method comprising:

[0007] The skeleton network extracts multi-level feature maps with different receptive fields from the input image and inputs them into the FPN feature pyramid network. At the same time, the small receptive field feature map is input into the global semantic context information extraction network, and the large receptive field feature map is input into the global detail context information extraction network. The small receptive field feature map is the feature map smaller than the preset receptive field, and the large receptive field feature map is the feature map greater than or equal to the preset receptive field.

[0008] The FPN feature pyramid network obtains process feature maps based on multi-level feature maps with different receptive fields, and inputs them into the classification subnetwork and localization subnetwork of the target detection and recognition network, respectively.

[0009] The global semantic context information extraction network extracts global semantic context information based on small receptive field feature maps;

[0010] The Global Detail Context Information Extraction Network extracts global detail context information based on the large receptive field feature map;

[0011] The classification subnetwork extracts the target classification features based on the process feature map, and fuses the global semantic context information and the target classification features to obtain the fused classification features;

[0012] The localization subnetwork extracts target localization features based on the process feature map, and fuses global detailed contextual information and target localization features to obtain fused localization features;

[0013] The target classifier obtains the target category label based on the fused classification features;

[0014] The target locator obtains the target coordinates based on the fused positioning features.

[0015] Preferably, the global semantic context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence.

[0016] Preferably, the global detail context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence.

[0017] Preferably, the fusion method of global semantic context information and target classification features is matrix addition or channel-wise concatenation.

[0018] Preferably, the fusion method of global detailed contextual information and target localization features is matrix addition or channel-wise concatenation.

[0019] According to another aspect of the present invention, a neural network detection and recognition system for brain-like contextual information processing is provided, the system comprising an image acquisition device, a computing device, and a display output terminal;

[0020] The image acquisition device is used to acquire images and input them into the computing device;

[0021] The computing device includes a skeleton network, an FPN feature pyramid network, a global semantic context information extraction network, a global detail context information extraction network, a classification subnetwork and a localization subnetwork of the target detection and recognition network, a target classifier, and a target localizer. The skeleton network extracts multi-level feature maps with different receptive fields from the input image and inputs them to the FPN feature pyramid network. It also inputs small receptive field feature maps to the global semantic context information extraction network and large receptive field feature maps to the global detail context information extraction network. The FPN feature pyramid network obtains process feature maps based on the multi-level feature maps with different receptive fields and inputs them to the classification and localization subnetworks of the target detection and recognition network, respectively. The network comprises the following subnetworks: a global semantic context information extraction network for extracting global semantic context information from a small receptive field feature map; a global detailed context information extraction network for extracting global detailed context information from a large receptive field feature map; a classification subnetwork for extracting target classification features from a process feature map and for fusing global semantic context information and target classification features to obtain fused classification features; a localization subnetwork for extracting target localization features from a process feature map and for fusing global detailed context information and target localization features to obtain fused localization features; a target classifier for obtaining target category labels based on the fused classification features; and a target localizer for obtaining target coordinate values ​​based on the fused localization features.

[0022] The display output terminal is used to receive the target category label and target coordinate value output by the computing device, and to output and display the target category label and target coordinate value.

[0023] Preferably, the computing device is an FPGA, an AI chip, or a Jetson series intelligent computing terminal.

[0024] Preferably, the global semantic context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence.

[0025] Preferably, the global detail context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence.

[0026] According to another aspect of the present invention, a computer device is provided, including a memory, a processor, and a neural network detection and recognition program for brain-like contextual information processing stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the neural network detection and recognition program for brain-like contextual information processing.

[0027] By applying the technical solution of this invention, global semantic context information is extracted through a small receptive field feature map, and global detailed context information is extracted through a large receptive field feature map. Then, by fusing the global semantic context information with target classification features, and fusing the global detailed context information with target localization features, the performance of image or video target detection and recognition is ultimately improved. This invention can also compress and accelerate the model through lightweight neural network technology, making it more suitable for embedded computing devices and enabling high frame rate detection and recognition functions. Attached Figure Description

[0028] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0029] Figure 1 A flowchart of a neural network detection and recognition method for brain-like contextual information processing according to an embodiment of the present invention is shown;

[0030] Figure 2 A detailed block diagram of a neural network detection and recognition method for brain-like contextual information processing according to an embodiment of the present invention is shown.

[0031] Figure 3 A schematic diagram of a context information extraction network provided according to an embodiment of the present invention is shown;

[0032] Figure 4 A schematic diagram of a neural network detection and recognition system for brain-like contextual information processing according to an embodiment of the present invention is shown. Detailed Implementation

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0035] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0036] like Figure 1 and Figure 2 As shown, the present invention provides a neural network detection and recognition method for brain-like contextual information processing, the method comprising:

[0037] S10. The skeleton network extracts multi-level feature maps with different receptive fields from the input image and inputs them into the FPN feature pyramid network. At the same time, the small receptive field feature map is input into the global semantic context information extraction network, and the large receptive field feature map is input into the global detail context information extraction network. The small receptive field feature map is a feature map smaller than the preset receptive field, and the large receptive field feature map is a feature map greater than or equal to the preset receptive field.

[0038] S20 and FPN feature pyramid networks obtain process feature maps based on multi-level feature maps with different receptive fields, and input them into the classification subnetwork and localization subnetwork of the target detection and recognition network, respectively.

[0039] S30, the global semantic context information extraction network extracts global semantic context information based on the small receptive field feature map;

[0040] S40, the Global Detail Context Information Extraction Network extracts global detail context information based on the large receptive field feature map;

[0041] S50. The classification sub-network extracts the target classification features based on the process feature map, and fuses the global semantic context information and the target classification features to obtain the fused classification features.

[0042] S60. The localization sub-network extracts the target localization features based on the process feature map, and fuses the global detailed context information and the target localization features to obtain the fused localization features.

[0043] S70. The target classifier obtains the target category label based on the fused classification features;

[0044] S80, the target locator obtains the target coordinates based on the fused positioning features.

[0045] This invention extracts global semantic contextual information through small receptive field feature maps and global detailed contextual information through large receptive field feature maps. It then fuses the global semantic contextual information with target classification features and the global detailed contextual information with target localization features, thereby ultimately improving the performance of image or video target detection and recognition. Furthermore, this invention utilizes lightweight neural network techniques for model compression and acceleration, making it more suitable for embedded computing devices and enabling high frame rate detection and recognition.

[0046] According to one embodiment of the present invention, in S10 of the present invention, the skeleton network can use a typical deep learning image classification network or object detection and recognition network, such as VGG, ResNet, etc.

[0047] Specifically, such as Figure 2 As shown, the backbone network can be a ResNet50 network, which can output multi-level feature maps (C1, C2, C3, C4, C5) with five different receptive fields. The receptive fields of the feature maps are C1: 1 / 4, C2: 1 / 8, C3: 1 / 16, C4: 1 / 32, and C5: 1 / 64, respectively.

[0048] According to one embodiment of the present invention, such as Figure 3 As shown, in S30 of the present invention, the global semantic context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence. The number of each type of unit is determined based on the actual required number of parameters, computational load, and accuracy.

[0049] Specifically, the global semantic context information extraction network extracts global semantic context information based on C1 and C2 in the ResNet50 network.

[0050] According to one embodiment of the present invention, such as Figure 3As shown, in S40 of the present invention, the global detail context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence. The number of each type of unit is determined based on the actual required number of parameters, computational load, and accuracy.

[0051] Specifically, the global detail context information extraction network extracts global detail context information based on C3, C4, and C5 in the ResNet50 network.

[0052] According to one embodiment of the present invention, in S50, the global semantic context information and target classification features are fused by matrix addition or channel-wise concatenation. By fusing the global semantic context information and target classification features, the reasoning regarding the contextual relationships between different targets in the image is enhanced during the classification process, thereby improving the training and optimization effect of the classification sub-network and ultimately improving the target classification performance.

[0053] According to one embodiment of the present invention, in S60, the fusion method of global detail context information and target localization features is matrix addition or channel-wise concatenation. By fusing global detail context information and target localization features, the reasoning of the contextual relationships of local image details during the localization regression process is enhanced, thereby improving the training and optimization effect of the localization sub-network and improving target localization performance.

[0054] Furthermore, based on the requirements of the classification and localization subnetworks used, and to satisfy the mathematical operation rules of the matrix, the shape and structure of the two contextual information need to be adjusted. The adjustment methods used are conventional operations and units in deep neural network construction, such as using convolutional units or pooling units with appropriate kernel size, padding size, and stride to adjust the shape parameters of the data, such as the number of channels, width, and depth.

[0055] The method in this application does not impose any special limitations on the structure of the object detection and recognition classification subnetwork, the object detection and recognition localization subnetwork, the object classifier, and the object localizer (also known as the object regressor); any structure based on deep neural networks can be used. The design or creation of these structural components is not within the scope of this application.

[0056] like Figure 4 As shown, the present invention also provides a neural network detection and recognition system for brain-like contextual information processing, the system comprising an image acquisition device, a computing device, and a display output terminal;

[0057] The image acquisition device is used to acquire images and input them into the computing device;

[0058] The computing device includes a skeleton network, an FPN feature pyramid network, a global semantic context information extraction network, a global detail context information extraction network, a classification subnetwork and a localization subnetwork of the target detection and recognition network, a target classifier, and a target localizer. The skeleton network extracts multi-level feature maps with different receptive fields from the input image and inputs them to the FPN feature pyramid network. It also inputs small receptive field feature maps to the global semantic context information extraction network and large receptive field feature maps to the global detail context information extraction network. The FPN feature pyramid network obtains process feature maps based on the multi-level feature maps with different receptive fields and inputs them to the classification and localization subnetworks of the target detection and recognition network, respectively. The network comprises the following subnetworks: a global semantic context information extraction network for extracting global semantic context information from a small receptive field feature map; a global detailed context information extraction network for extracting global detailed context information from a large receptive field feature map; a classification subnetwork for extracting target classification features from a process feature map and for fusing global semantic context information and target classification features to obtain fused classification features; a localization subnetwork for extracting target localization features from a process feature map and for fusing global detailed context information and target localization features to obtain fused localization features; a target classifier for obtaining target category labels based on the fused classification features; and a target localizer for obtaining target coordinate values ​​based on the fused localization features.

[0059] The display output terminal is used to receive the target category label and target coordinate value output by the computing device, and to output and display the target category label and target coordinate value.

[0060] This invention extracts global semantic contextual information through small receptive field feature maps and global detailed contextual information through large receptive field feature maps. It then fuses the global semantic contextual information with target classification features and the global detailed contextual information with target localization features, thereby ultimately improving the performance of image or video target detection and recognition. Furthermore, this invention utilizes lightweight neural network techniques for model compression and acceleration, making it more suitable for embedded computing devices and enabling high frame rate detection and recognition.

[0061] According to one embodiment of the present invention, the global semantic context information extraction network includes convolutional neural network units, normalization units, and activation function units connected in sequence. The number of each type of unit is determined based on the actual required number of parameters, computational load, and accuracy.

[0062] According to one embodiment of the present invention, the global detail context information extraction network includes convolutional neural network units, normalization units, and activation function units connected in sequence. The number of each type of unit is determined based on the actual required number of parameters, computational load, and accuracy.

[0063] According to one embodiment of the present invention, neural network model compression operations can be performed on the global semantic context information extraction network and the global detail context information extraction network, such as fixed-point quantization, compression pruning and other common model lightweighting methods. Then, the extraction network can be operated on a low-power computing device, such as an FPGA, an AI chip or a Jetson series intelligent computing terminal.

[0064] The system does not specify or claim the model compression and acceleration methods used, nor does it specify or claim the low-power computing devices used. Any method that can perform lightweight processing and low-power computing on the context-adaptive model in this application, but which contains the context-adaptive model design method described in this method, constitutes a claim of this application.

[0065] The system described in this application can reduce the parameter size and computational complexity of the model through conventional neural network lightweighting techniques, thereby achieving high-efficiency target detection and recognition when combined with suitable embedded devices. The key is that the lightweighted model still has the contextual information processing method described in this application, thus falling within the claims of this application.

[0066] The present invention also provides a computer device, including a memory, a processor, and a neural network detection and recognition program for brain-like contextual information processing stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the neural network detection and recognition program for brain-like contextual information processing.

[0067] In summary, this invention provides a neural network detection and recognition method and system for brain-like contextual information processing. It extracts global semantic contextual information through small receptive field feature maps and global detailed contextual information through large receptive field feature maps. Furthermore, it fuses global semantic contextual information with target classification features and global detailed contextual information with target localization features, thereby ultimately improving the performance of image or video target detection and recognition. This invention can also utilize lightweight neural network techniques for model compression and acceleration, making it more suitable for embedded computing devices and enabling high frame rate detection and recognition functions.

[0068] In specific deep learning model design, it is necessary to adaptively adjust the channel, height, and width through upsampling or adding auxiliary neural network modules during mathematical operations such as feature map summation and concatenation to conform to the mathematical constraints of matrix operations. However, the above solution omits these commonly used network structures and conventional operation methods in deep learning model construction. In actual model design, it may be necessary to add these conventional structures or implement these conventional operations to work together to achieve the normal operation of the solution in this application.

[0069] The parts of this invention not described in detail are techniques known to those skilled in the art.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A neural network detection and recognition method for brain-like contextual information processing, characterized in that, The method includes: The skeleton network extracts multi-level feature maps with different receptive fields from the input image and inputs them into the FPN feature pyramid network. At the same time, the small receptive field feature map is input into the global semantic context information extraction network, and the large receptive field feature map is input into the global detail context information extraction network. The small receptive field feature map is the feature map smaller than the preset receptive field, and the large receptive field feature map is the feature map greater than or equal to the preset receptive field. The FPN feature pyramid network obtains process feature maps based on multi-level feature maps with different receptive fields, and inputs them into the classification subnetwork and localization subnetwork of the target detection and recognition network, respectively. The global semantic context information extraction network extracts global semantic context information based on small receptive field feature maps; The Global Detail Context Information Extraction Network extracts global detail context information based on the large receptive field feature map; The classification subnetwork extracts the target classification features based on the process feature map, and fuses the global semantic context information and the target classification features to obtain the fused classification features; The localization subnetwork extracts target localization features based on the process feature map, and fuses global detailed contextual information and target localization features to obtain fused localization features; The target classifier obtains the target category label based on the fused classification features; The target locator obtains the target coordinates based on the fused positioning features.

2. The method according to claim 1, characterized in that, The global semantic context information extraction network consists of sequentially connected convolutional neural network units, normalization units, and activation function units.

3. The method according to claim 1, characterized in that, The global detail context information extraction network consists of sequentially connected convolutional neural network units, normalization units, and activation function units.

4. The method according to claim 1, characterized in that, The fusion of global semantic context information and target classification features is achieved by matrix addition or channel-wise concatenation.

5. The method according to claim 1, characterized in that, The fusion method for global detailed contextual information and target localization features is matrix addition or channel-wise splicing.

6. A neural network detection and recognition system for brain-like contextual information processing, characterized in that, The system includes an image acquisition device, a computing device, and a display output terminal; The image acquisition device is used to acquire images and input them into the computing device; The computing device includes a skeleton network, an FPN feature pyramid network, a global semantic context information extraction network, a global detail context information extraction network, a classification subnetwork and a localization subnetwork of the target detection and recognition network, a target classifier, and a target localizer. The skeleton network extracts multi-level feature maps with different receptive fields from the input image and inputs them to the FPN feature pyramid network. It also inputs small receptive field feature maps to the global semantic context information extraction network and large receptive field feature maps to the global detail context information extraction network. The FPN feature pyramid network obtains process feature maps based on the multi-level feature maps with different receptive fields and inputs them to the classification and localization subnetworks of the target detection and recognition network, respectively. The network comprises the following subnetworks: a global semantic context information extraction network for extracting global semantic context information from a small receptive field feature map; a global detailed context information extraction network for extracting global detailed context information from a large receptive field feature map; a classification subnetwork for extracting target classification features from a process feature map and for fusing global semantic context information and target classification features to obtain fused classification features; a localization subnetwork for extracting target localization features from a process feature map and for fusing global detailed context information and target localization features to obtain fused localization features; a target classifier for obtaining target category labels based on the fused classification features; and a target localizer for obtaining target coordinate values ​​based on the fused localization features. The display output terminal is used to receive the target category label and target coordinate value output by the computing device, and to output and display the target category label and target coordinate value.

7. The system according to claim 6, characterized in that, The computing device is an FPGA, an AI chip, or a Jetson series intelligent computing terminal.

8. The system according to claim 6 or 7, characterized in that, The global semantic context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence.

9. The system according to claim 8, characterized in that, The global detail context information extraction network includes a convolutional neural network unit, a normalization unit, and an activation function unit connected in sequence.

10. A computer device, characterized in that, The method includes a memory, a processor, and a neural network detection and recognition program for brain-like contextual information processing stored in the memory and executable on the processor. When the processor executes the neural network detection and recognition program for brain-like contextual information processing, it implements the method of any one of claims 1 to 5.

Images