Target detection method, device, computer equipment and storage medium

Abstract

The present application relates to a target detection method, device, computer equipment and storage medium. The method comprises: obtaining an image to be detected; inputting the image to be detected into a trained target detection model, wherein the target detection model comprises a preprocessing unit, a feature extraction unit and a prediction unit; extracting a first feature map corresponding to the image to be detected by the preprocessing unit to obtain a first low-dimensional feature map corresponding to the first feature map; performing feature extraction on the first low-dimensional feature map by the feature extraction unit to obtain target capsule information corresponding to the image to be detected; performing target detection on the target capsule information by the prediction unit to obtain a target detection result corresponding to the image to be detected. The use of this method can reduce the computational complexity and memory complexity when performing target detection on smaller target objects or partially occluded target objects.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target detection method, device, computer equipment and storage medium. Background Art

[0002] Object detection refers to detecting the target objects in an image and predicting the position and category of each target object. As an important branch of computer vision and digital image processing, object detection is widely used in many fields such as robot navigation, intelligent video surveillance, industrial inspection, aerospace, etc. It has important practical significance to reduce the consumption of human resources through computer vision. At the same time, object detection is also a basic algorithm in the field of identity recognition, which plays a vital role in subsequent tasks such as face recognition, gait recognition, crowd counting, and instance segmentation. Due to the widespread use of deep learning, object detection technology has developed rapidly. In the traditional object detection method, object detection is achieved by extracting the feature map corresponding to the target object in the image, such as through the object detector DETR (Detection Transformer, object detection based on set prediction) for object detection.

[0003] However, smaller target objects are detected on high-resolution feature maps. The traditional method of target detection will result in high computational complexity. Therefore, how to reduce the computational complexity of smaller target objects in the target detection process is a technical problem that needs to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide a target detection method, device, computer equipment and storage medium that can reduce the computational complexity of smaller target objects during target detection in order to address the above technical problems.

[0005] A target detection method, the method comprising:

[0006] Acquire the image to be detected;

[0007] Inputting the image to be detected into a trained target detection model, wherein the target detection model includes a preprocessing unit, a feature extraction unit and a prediction unit;

[0008] Extracting a first feature map corresponding to the image to be detected by the preprocessing unit to obtain a first low-dimensional feature map corresponding to the first feature map;

[0009] Performing feature extraction on the first low-dimensional feature map by the feature extraction unit to obtain target capsule information corresponding to the image to be detected;

[0010] The prediction unit performs target detection on the target capsule information to obtain a target detection result corresponding to the image to be detected.

[0011] In one embodiment, extracting the first feature map corresponding to the image to be detected by the preprocessing unit includes:

[0012] The convolutional neural network in the preprocessing unit is used to extract features of the image to be detected, and the feature maps output by the last two convolutional layers of the convolutional neural network are determined as the first feature maps corresponding to the image to be detected.

[0013] In one embodiment, performing attention-based pooling processing on the first feature map to obtain a first low-dimensional feature map corresponding to the first feature map includes:

[0014] Performing a multi-head attention calculation on the first feature map to obtain a multi-head attention value corresponding to the first feature map;

[0015] The multi-head attention values ​​are normalized to obtain a first low-dimensional feature map corresponding to the first feature map.

[0016] In one embodiment, the feature extraction unit includes an encoding unit and a decoding unit; and extracting features from the first low-dimensional feature map by the feature extraction unit to obtain target capsule information corresponding to the image to be detected includes:

[0017] Performing global feature extraction on the first low-dimensional feature map by the encoding unit to obtain global feature information, and performing capsule conversion on the global feature information to obtain initial capsule information;

[0018] The initial capsule information is input into the decoding unit, category feature extraction is performed on the initial capsule information to obtain category feature information, and capsule conversion is performed on the category feature information to obtain target capsule information.

[0019] In one of the embodiments, performing target detection on the target capsule information by the prediction unit to obtain a target detection result corresponding to the image to be detected includes:

[0020] Performing target detection on the target capsule information based on the attention routing by the prediction unit to obtain a first detection result;

[0021] Performing a linear transformation on the target capsule information by the prediction unit to obtain a second detection result;

[0022] The first detection result and the second detection result are fused to obtain a target detection result corresponding to the image to be detected.

[0023] In one embodiment, before acquiring the image to be detected, the method further includes:

[0024] Get a sample image set;

[0025] Inputting the sample image set into the target detection model to be trained, extracting a second feature map corresponding to the sample image set through a preprocessing unit in the target detection model to be trained, and performing attention-based pooling processing on the second feature map to obtain a second low-dimensional feature map corresponding to the second feature map;

[0026] Extracting features from the second low-dimensional feature map by a feature extraction unit in the target detection model to be trained, to obtain target capsule information corresponding to the sample image set;

[0027] Performing target detection on the target capsule information corresponding to the sample image set by a prediction unit in the target detection model to be trained, thereby obtaining a target detection result corresponding to the sample image set;

[0028] The loss value of the target detection model to be trained is calculated according to the target detection results corresponding to the sample image set, and the network parameters of the target detection model to be trained are updated according to the loss value until the preset conditions are met to obtain the trained target detection model.

[0029] In one embodiment, the sample image set is annotated with target label information; and calculating the loss value of the target detection model to be trained according to the target detection result corresponding to the sample image set includes:

[0030] Performing binary matching on the target detection result corresponding to the sample image set and the target label information to obtain a matching result;

[0031] The loss value of the target detection model to be trained is calculated according to the matching result.

[0032] In one of the embodiments, the loss value of the target detection model to be trained includes a target position offset loss value, a classification loss value, and a matching loss value.

[0033] A target detection device, comprising:

[0034] An image acquisition module, used for acquiring an image to be detected;

[0035] A feature extraction module is used to input the image to be detected into a trained target detection model, wherein the target detection model includes a preprocessing unit, a feature extraction unit, and a prediction unit; extract a first feature map corresponding to the image to be detected by the preprocessing unit to obtain a first low-dimensional feature map corresponding to the first feature map; and extract features from the first low-dimensional feature map by the feature extraction unit to obtain target capsule information corresponding to the image to be detected;

[0036] The target detection module is used to perform target detection on the target capsule information through the prediction unit to obtain a target detection result corresponding to the image to be detected.

[0037] A computer device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0038] A computer-readable storage medium stores a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0039] The above-mentioned target detection method, device, computer equipment and storage medium extract the first feature map corresponding to the image to be detected through the preprocessing unit of the trained target detection model, perform attention-based pooling processing on the first feature map, and obtain the first low-dimensional feature map corresponding to the first feature map. Thus, the first low-dimensional feature map is subjected to feature extraction by the feature extraction unit to obtain the target capsule information corresponding to the image to be detected, and then the target capsule information is subjected to target detection by the prediction unit to obtain the target detection result corresponding to the image to be detected. By performing attention-based pooling processing on the first feature map, irrelevant information in the first feature map can be removed, and only information related to target detection can be focused on, thereby reducing the computational complexity. By performing dimensionality reduction processing on the first feature map, the memory complexity is reduced. When performing target detection on smaller target objects or partially occluded target objects, the computational complexity and memory complexity can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 An application environment diagram of a target detection method in an embodiment;

[0041] Figure 2 Schematic diagram of a target detection method in one embodiment;

[0042] Figure 3 A schematic flow chart of a step of performing attention-based pooling processing on a first feature map to obtain a first low-dimensional feature map corresponding to the first feature map in an embodiment;

[0043] Figure 4 It is a flowchart of a step of extracting features from a first low-dimensional feature map by a feature extraction unit to obtain target capsule information corresponding to an image to be detected in an embodiment;

[0044] Figure 5 1. A flowchart of a step of performing target detection on target capsule information by a prediction unit to obtain a target detection result corresponding to an image to be detected in an embodiment;

[0045] Figure 6 is a structural block diagram of a target detection device in one embodiment;

[0046] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The target detection method provided in the present application can be applied to a computer device, which can be a terminal or a server. It is understandable that the target detection method provided in the present application can be applied to a terminal, a server, or a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0049] The target detection method provided by this application can reduce the computational complexity of smaller target objects and is applicable to multiple application scenarios of target detection. For example, in the face recognition scenario, the target detection method can improve the accuracy of face detection and reduce the misjudgment rate; in the vehicle detection scenario, the vehicle in the monitoring image can be more accurately identified.

[0050] The target detection method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. When target detection is required, the server 104 obtains the image to be detected sent by the terminal 102, and inputs the image to be detected into the trained target detection model. The target detection model includes a preprocessing unit, a feature extraction unit and a prediction unit. The first feature map corresponding to the image to be detected is extracted by the preprocessing unit, and the first feature map is subjected to attention-based pooling processing to obtain the first low-dimensional feature map corresponding to the first feature map, thereby extracting features from the first low-dimensional feature map through the feature extraction unit to obtain the target capsule information corresponding to the image to be detected, and then the target capsule information is subjected to target detection through the prediction unit to obtain the target detection result corresponding to the image to be detected. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0051] In one embodiment, Figure 2 As shown, a target detection method is provided, and the method is applied to a server as an example for description, including the following steps:

[0052] Step 202: Acquire an image to be detected.

[0053] Step 204: input the image to be detected into the trained target detection model, where the target detection model includes a preprocessing unit, a feature extraction unit, and a prediction unit.

[0054] The image to be detected refers to the image that needs to be detected. Object detection refers to detecting the target objects in the image and predicting the position and category of each target object, such as faces, vehicles or buildings, depending on the actual application scenario.

[0055] The server can obtain the target detection request sent by the terminal, parse the target detection request, and obtain the image to be detected. The image to be detected pre-stored in the terminal can be an image including the target object captured by the image sensor, and the position, size, and acquisition angle of the target object in the image to be detected can be arbitrary. For example, the target object is smaller than the image to be detected and can only occupy a small part of the image to be detected. In addition, due to the influence of the acquisition angle, the target object may be tilted in the image to be detected, or the size ratio of the target object may be out of balance compared to its actual size ratio, such as the two parallel sides of the rectangular target object are of different lengths in the image to be detected, and the other two sides are not parallel.

[0056] In one embodiment, the target object in the image to be detected may have a frame of a regular shape, and the target object may have a fixed number of vertices, and the vertices are connected to form the frame of the target object. For example, the frame of the target object may be a square, rectangle, etc. with four vertices.

[0057] After acquiring the image to be detected, the server calls the pre-stored trained target detection model, which is trained by a set of sample images labeled with target category labels. The target detection model includes a preprocessing unit, a feature extraction unit, and a prediction unit, which can effectively reduce the computational complexity of smaller target objects.

[0058] Step 206: extract a first feature map corresponding to the image to be detected through a preprocessing unit, perform attention-based pooling processing on the first feature map, and obtain a first low-dimensional feature map corresponding to the first feature map.

[0059] The first feature map can be used to detect smaller target objects or partially blocked target objects in the image. The first low-dimensional feature map refers to the low-dimensional feature representation corresponding to the first feature map.

[0060] The preprocessing unit in the target detection model is used to obtain a first low-dimensional feature map corresponding to the first feature map. Specifically, the preprocessing unit extracts the first feature map corresponding to the image to be detected, and performs attention pooling processing on the first feature to obtain the first low-dimensional feature map corresponding to the first feature map. Among them, the first feature map may include two feature maps with different resolutions, and the feature map with a higher resolution is used to detect smaller target objects or partially occluded target objects in the image. The first feature map includes global image information of the image to be detected.

[0061] Since the target detection model has high computational complexity and memory complexity when processing higher-resolution feature maps, in order to reduce the computational complexity and memory complexity of higher-resolution feature maps, that is, to reduce the computational complexity of smaller target objects or partially occluded target objects during target detection, the server performs attention pooling on the first feature map before extracting features from the first feature map. Attention pooling refers to removing irrelevant points with small response values ​​in the first feature, that is, by sparsely making the first feature map, only information related to target detection needs to be extracted, thereby reducing the amount of calculation and reducing computational complexity and memory complexity.

[0062] Step 208: extract features from the first low-dimensional feature map by a feature extraction unit to obtain target capsule information corresponding to the image to be detected.

[0063] The target capsule information refers to the capsule representation of the target object in the image to be detected, that is, the feature information of the target object is represented by a capsule.

[0064] The feature extraction unit in the target detection model is used to extract the target capsule information of the target object in the image to be detected. The target capsule information may include multiple capsule vectors, each capsule vector is used to represent the feature information of the corresponding target object, and each capsule vector includes multiple dimensions, each dimension is used to represent the posture information of a local feature of the corresponding target object. Therefore, the local feature information of the target object in the image to be detected can be accurately reflected through the target capsule information, so that the target object in the image to be detected can be represented by the local feature information.

[0065] Step 210: classify the target capsule information through the prediction unit to obtain the target detection result corresponding to the image to be detected.

[0066] After the feature extraction unit extracts the target capsule information corresponding to the image to be detected, the target capsule information is used as the input of the prediction unit, and the prediction unit performs target detection based on the target capsule information to predict the position and corresponding category of the target object in the image to be detected. The position of the target object refers to the frame corresponding to the target object.

[0067] Based on the principle of brain cognition, the brain first performs saliency attention driven by external stimuli in the process of cognition. It does not require active intervention. It is a screening process of the bottom layer to the upper layer, that is, the bottom-up inference should be performed first. Therefore, according to the target information of the upper layer, the corresponding lower layer information is screened, that is, the top-down information transmission is realized. Therefore, in this embodiment, the prediction unit can classify and process the target capsule information respectively through the capsule transmission method and the full connection method. The capsule transmission method can realize bottom-up inference, and the full connection method can realize top-down information transmission. The capsule transmission method uses the local feature information of the target object, and predicts the position and category of the target object through the local feature information, while the full connection method uses the overall information of the target object, and predicts the position and category of the target object through the overall information of the target object. By combining the above two methods, the local feature information and overall information of the target object are fully utilized, and the accuracy of target detection is effectively improved.

[0068] The traditional target detector DETR (Detection Transformer, target detection based on set prediction) has the defects of slow convergence and high computational complexity, especially when performing target detection on smaller target objects, it is necessary to perform detection in a high-resolution feature map, resulting in high computational complexity and memory complexity. In this embodiment, the first feature map corresponding to the image to be detected is extracted by the preprocessing unit of the trained target detection model, and the first feature map is subjected to attention-based pooling processing to obtain the first low-dimensional feature map corresponding to the first feature map. Thus, the first low-dimensional feature map is subjected to feature extraction by the feature extraction unit to obtain the target capsule information corresponding to the image to be detected, and then the target capsule information is subjected to target detection by the prediction unit to obtain the target detection result corresponding to the image to be detected. By performing attention-based pooling processing on the first feature map, irrelevant information in the first feature map can be removed, and only information related to target detection can be focused on, thereby reducing computational complexity, and by performing dimensionality reduction processing on the first feature map, memory complexity can be reduced. When performing target detection on smaller target objects or partially occluded target objects, computational complexity and memory complexity can be reduced.

[0069] In one embodiment, extracting the first feature map corresponding to the image to be detected by the preprocessing unit includes: performing feature extraction on the image to be detected by a convolutional neural network in the preprocessing unit, and determining the feature maps output by the last two convolutional layers of the convolutional neural network as the first feature map corresponding to the image to be detected.

[0070] The preprocessing unit of the trained target detection model may include a convolutional neural network (CNN). The convolutional neural network is used to extract the first feature map corresponding to the image to be detected, so that the subsequent feature extraction unit can extract the target capsule information corresponding to the image to be detected. The convolutional neural network may include multiple network layers, such as an input layer, multiple convolutional layers, a pooling layer, a fully connected layer, etc. The feature maps output by the last two convolutional layers can be determined as the first feature maps corresponding to the image to be detected. As the multiple convolutional layers are processed in sequence, the resolution of the output feature maps gradually decreases. Therefore, the resolution of the feature map output by the second to last convolutional layer is higher than the resolution of the feature map output by the last convolutional layer. For ease of description, the second to last convolutional layers can be referred to as the a-1th layer, and the feature map output by them can be referred to as F. a-1 , the size is [bs,d a-1 ,h a-1 ,w a-1 ], where bs represents F a-1 The batch size (the number of samples in a batch), d a-1 Indicates F a-1 The feature dimension, ha-1 Indicates F a-1 High, w a-1 Indicates F a-1 The last convolutional layer is called the ath layer, and its output feature map can be called F a , the size is [bs,d a ,h a ,w a ], where bs represents F a The batch size (the number of samples in a batch), d a Indicates F a The feature dimension, h a Indicates F a High, w a Indicates F a Width. a-1 and F a The size satisfies h a-1 >h a , w a-1 >w a .

[0071] Furthermore, since the structure of the feature extraction unit does not use any recursive structure or convolution structure, in order for the feature extraction unit to utilize the sequential information of the image to be detected, it is necessary to introduce information that can express the absolute or relative position of each element in the image to be detected. For example, a convolutional neural network can be used to position encode the first feature map, and the encoded first feature map can be subjected to attention-based pooling processing. Position encoding refers to encoding the positions of the elements contained in the first feature map.

[0072] In this embodiment, the first feature map includes not only the feature map output by the last convolutional layer, but also the feature map output by the second to last convolutional layer with higher resolution, thereby enabling target detection of smaller target objects.

[0073] In one embodiment, Figure 3 As shown, the step of performing attention-based pooling processing on the first feature map to obtain a first low-dimensional feature map corresponding to the first feature map includes:

[0074] Step 302: perform multi-head attention calculation on the first feature map to obtain the multi-head attention value corresponding to the first feature map.

[0075] Step 304, normalize the multi-head attention values ​​to obtain a first low-dimensional feature map corresponding to the first feature map.

[0076] The preprocessing unit in the trained target detection model may include a convolutional neural network and a pooling unit. The convolutional neural network is used to extract the first feature map corresponding to the image to be detected. The first feature map includes the feature maps output by the last two convolutional layers and can be represented as the feature map F output by the a-1th layer. a-1 And the feature map F output by layer a a The pooling unit is used to pool the feature map F a-1 and feature map F a Attention-based pooling is performed separately. Attention-based pooling means that multi-head attention calculation is first performed on the feature map, and then the multi-head attention value is normalized. The pooling unit can first perform multi-head attention calculation on the feature map F a-1 Perform attention-based pooling processing and then perform feature map F a Perform attention-based pooling.

[0077] Take the feature map F a-1 The attention-based pooling process is taken as an example. Specifically, the pooling unit can use a multi-head attention mechanism to perform attention on the feature map F a-1 Perform multi-head attention calculation to obtain the multi-head attention value corresponding to the first feature map, and then normalize the multi-head attention value to obtain the first low-dimensional feature map corresponding to the first feature map. The process of attention-based pooling (PMA(Z)) can be expressed by the following formula:

[0078] PMA(Z)=LayerNorm(S+Multihead(S,Z,Z)) (1)

[0079]

[0080] Among them, S represents the first low-dimensional feature map, Z represents the key and value vectors, that is, the feature map F a-1 , Multihead(S,Z,Z) represents the multi-head attention value, LayerNorm represents the normalization processing, Represents the scale factor, dim represents the feature map F a-1 Dimension.

[0081] It can be understood that the feature map F a Perform attention-based pooling processing to obtain the feature map F a The corresponding first low-dimensional feature map. Thus, the feature map F is obtained a-1 The corresponding first low-dimensional feature map and feature map F a The corresponding first low-dimensional feature map.

[0082] In this embodiment, by performing multi-head attention calculation and normalization processing on the first feature map, only information related to target detection needs to be extracted, which effectively reduces the amount of calculation and the computational consumption caused by attention calculation in subsequent feature extraction units, thereby effectively reducing the computational complexity and memory complexity.

[0083] In one embodiment, Figure 4 As shown, the step of extracting features from the first low-dimensional feature map by a feature extraction unit to obtain target capsule information corresponding to the image to be detected includes:

[0084] Step 402: Perform global feature extraction on the first low-dimensional feature map by a coding unit to obtain global feature information, and perform capsule conversion on the global feature information to obtain initial capsule information.

[0085] Step 404: input the initial capsule information into a decoding unit, extract category features from the initial capsule information to obtain category feature information, and perform capsule conversion on the category feature information to obtain target capsule information.

[0086] The feature extraction unit may be a transformer network based on capsule representation. The feature extraction unit includes an encoding unit and a decoding unit. Both the encoding unit and the decoding unit include a capsule conversion unit for converting information into a capsule form.

[0087] The encoding unit in the feature extraction unit is used to extract global feature information in the first low-dimensional feature map, such as color features, texture features, and shape features, so that the global feature information is capsule-converted by the capsule conversion unit in the encoding unit to obtain initial capsule information. The initial capsule information refers to the capsule representation corresponding to the global feature information. Capsule conversion of the global feature information refers to grouping the feature information of the same target object in the global feature information together to generate a capsule. The capsule is embodied in the form of a capsule vector, so the capsule vector includes multiple local features of the target object. The modulus of the capsule vector corresponding to each capsule represents the probability of the existence of each local feature in the target object, and the dimension of the capsule vector represents the posture information corresponding to each local feature in the target object. The first low-dimensional feature map includes a feature map F a-1 The corresponding first low-dimensional feature map and feature map F a The corresponding first low-dimensional feature map, therefore, the initial capsule information includes the feature map F a-1 The corresponding initial capsule representation and feature map F a The corresponding initial capsule representation, feature map F a-1 The corresponding initial capsule is denoted as P a-1 , the size is [bs,mum a-1 ,d a-1 / mum a-1 ,sa-1 ], where bs represents F a-1 The corresponding initial capsule represents the batch size (the number of samples in the batch), mum a-1 Indicates F a-1 The number of capsules represented by the corresponding initial capsule, d a-1 / mum a-1 Indicates F a-1 The capsule vector of each capsule in the corresponding initial capsule representation. Feature map F a The corresponding initial capsule is denoted as P a , the size is [bs,mum a ,d a / mum a ,s a ], where bs represents F a The corresponding initial capsule represents the batch size (the number of samples in the batch), mum a Indicates F a The number of capsules represented by the corresponding initial capsule, d a / mum a Indicates F a The capsule vector of each capsule in the corresponding initial capsule representation. By performing capsule transformation on the global feature information, feature information of the same instance type in the global feature information can be classified into one category, such as the eyes, mouth, nose, etc. of the same target object can be classified into one category.

[0088] The decoding unit in the feature extraction unit is used to extract the category feature information and the boundary information of the target object from the initial capsule information, and the capsule conversion unit in the decoding unit performs capsule conversion on the extracted category feature information and the boundary information of the target object to obtain the target capsule information. The target capsule information includes P a-1 The corresponding target capsule represents O a-1 and P a The corresponding target capsule represents O a The target capsule represents O a-1 The size can be [100,bs,mum a-1 ,d a-1 / mum a-1 ], where 100 represents the number of capsules to be detected in the image to be detected. The target capsule represents O a The size can be [100,bs,mum a ,d a / mum a]. By capsule conversion of the extracted category feature information and the boundary information of the target object, the feature information and the corresponding boundary information of the same target object can be clustered together. Therefore, the target capsule information includes the local feature information corresponding to each target object, the feature information of the target object and the corresponding boundary information.

[0089] In this embodiment, by adding the capsule conversion process in the encoding unit and the decoding unit, the different postures of the target object can be accurately identified, and the target object can be represented by local feature information. The local representation of the target object is more accurate, which is conducive to improving the accuracy of target detection.

[0090] In one embodiment, Figure 5 As shown, the target capsule information is detected by the prediction unit to obtain the target detection result corresponding to the image to be detected, including the following steps:

[0091] Step 502: Perform target detection on the target capsule information based on the attention routing through the prediction unit to obtain a first detection result.

[0092] Step 504: Perform a linear transformation on the target capsule information through a prediction unit to obtain a second detection result.

[0093] Step 506: The first detection result and the second detection result are fused to obtain a target detection result corresponding to the image to be detected.

[0094] Since the target capsule information includes P a-1 The corresponding target capsule represents O a-1 and P a The corresponding target capsule represents O a , the prediction unit can represent the target capsule O a-1 and the target capsule represents O a Target detection refers to identifying the target object in the image to be detected based on the target capsule information and predicting the position and category of each target object. The target detection process of the prediction unit includes two target detection methods, one is to realize target detection based on local feature information through capsule transfer, and the other is to realize target detection based on overall information through full connection. The prediction unit can process the target capsule information through two target detection methods at the same time, and fuse the results of the two target detection methods to make the target detection more accurate. The first detection result includes the target capsule representation. a-1 The corresponding first detection result and target capsule are represented by a The corresponding first detection result, the second detection result includes the target capsule representation O a-1 The corresponding second detection result and target capsule representation O aThe corresponding second detection result.

[0095] To represent the target capsule O a-1 When performing target detection by capsule transfer, the prediction unit can represent the target capsule O based on the bottom-up attention routing algorithm. a-1 and the target capsule represents O a The bottom-up attention routing algorithm refers to a routing algorithm based on a multi-head attention mechanism, which needs to obtain the probability value of the lower-layer capsule being assigned to the upper-layer capsule, such as the probability of assigning the eyes, nose, mouth and other lower-layer capsules to the face of the target object, where the lower-layer capsule can be the target capsule representation. a-1 The upper capsule can be the capsule in the first detection result after target detection. The bottom-up attention routing algorithm represents the target capsule as a-1 The corresponding number of capsules is used as the head of the multi-head attention mechanism, along the target capsule representation O a-1 The dimension where the corresponding number of capsules is located is calculated by using a multi-head attention mechanism to calculate the correlation between each upper-layer capsule and the lower-layer capsule after affine transformation, so as to realize the information transfer between capsules and obtain the target capsule representation O a-1 The calculation formula for the corresponding first detection result can be shown as (2). The first detection result can include multiple capsules, as well as the category and position of the target object corresponding to each capsule. The number of capsules and the number of capsules to be detected in the target capsule information. The fully connected method is a top-down information transmission method. When performing target detection through the fully connected method, the prediction unit represents the target capsule. a-1 Perform linear transformation and use the overall information of the target object to determine the category and position of the target object, and obtain the target capsule representation O a-1 The corresponding second detection result.

[0096] Through the above target detection method, the target capsule representation O can be predicted. a-1 and the target capsule represents O a The corresponding first detection result and second detection result. The first detection result is predicted by the local feature information of the target object, and the second detection result is predicted by the overall information of the target object, so that the prediction unit fuses the first detection result with the second detection result, which can make full use of the local feature information and overall information of the target object, and effectively improve the accuracy of the target detection result.

[0097] In one embodiment, before obtaining the image to be detected, the method further includes: obtaining a sample image set; inputting the sample image set into the target detection model to be trained, extracting a second feature map corresponding to the sample image set through a preprocessing unit in the target detection model to be trained, performing attention-based pooling processing on the second feature map to obtain a second low-dimensional feature map corresponding to the second feature map; performing feature extraction on the second low-dimensional feature map through a feature extraction unit in the target detection model to be trained to obtain target capsule information corresponding to the sample image set; performing target detection on the target capsule information corresponding to the sample image set through a prediction unit in the target detection model to be trained to obtain a target detection result corresponding to the sample image set; calculating a loss value of the target detection model to be trained based on the target detection result corresponding to the sample image set, and updating the network parameters of the target detection model to be trained based on the loss value until a preset condition is met to obtain a trained target detection model.

[0098] The sample image set refers to training samples used to train the target detection model. The sample image set may include multiple sample images, where multiple refers to two or more. The sample image set may be selected according to the application scenario. For example, in a vehicle detection scenario, the sample images in the sample image set may include vehicles, pedestrians, etc. In one embodiment, the sample image set may have multiple storage locations, which may be stored in a database or in a terminal, so that when training the target detection model, the corresponding sample image set is obtained from the database or the terminal.

[0099] The target detection method of the target detection model during the training process and the application process is the same. The target detection model to be trained includes a preprocessing unit, a feature extraction unit and a prediction unit. The preprocessing unit in the target detection model to be trained is used to extract the second feature map corresponding to each sample image in the sample image set. The second feature map refers to the feature map output by the last two convolutional layers of the convolutional neural network when the convolutional neural network in the preprocessing unit extracts features from each sample image. The prediction unit performs attention-based pooling processing on the second feature map to obtain a second low-dimensional feature map corresponding to the second feature map. The second low-dimensional feature map refers to the low-dimensional feature representation corresponding to the second feature map. Thus, the second low-dimensional feature map is used as the input of the feature extraction unit to perform feature extraction and obtain the target capsule information corresponding to the sample image set. The target capsule information corresponding to the sample image set includes the local feature information of the target object in the sample image set. Then, the target capsule information corresponding to the sample image set is detected by the prediction unit in the target detection model to be trained. Specifically, the prediction unit can classify and process the target capsule information respectively through the capsule transmission method and the full connection method. The capsule transmission method can realize bottom-up inference and utilize the local feature information of the target object. The full connection method can realize top-down information transmission and utilize the overall information of the target object. By combining the above two methods, the local feature information and overall information of the target object are fully utilized, which effectively improves the accuracy of the target detection results.

[0100] Furthermore, when the prediction unit in the target detection model to be trained performs target detection by transferring capsules, the prediction unit can use a bottom-up attention routing algorithm to perform target detection on the target capsule information corresponding to the sample image set to obtain the corresponding detection result. Specifically, the sample image set may include the number of categories, and the prediction unit can use a bottom-up attention routing algorithm to expand the target capsule information according to the number of categories, thereby ensuring that the dimension of the output capsule corresponds to the number of categories, that is, the detection result includes the target objects of the number of categories. The expansion method can be to add 1 dimension on the basis of the original dimension of the target capsule information, and copy the first 4 dimensions according to the number of categories. Thus, the number of capsules corresponding to the target capsule information is used as the head of the multi-head attention mechanism, and along the dimension where the number of capsules corresponding to the target capsule information is located, the multi-head attention mechanism is used to calculate the correlation between each output capsule after affine transformation and each capsule in the target capsule information, thereby realizing information transfer between capsules and obtaining the corresponding detection result.

[0101] After obtaining the target detection results, the loss value of the target detection model to be trained can be calculated based on the target detection results corresponding to the sample image set. The loss value is a parameter used to evaluate the prediction effect of the model. The smaller the network loss value, the better the prediction effect of the model. Correspondingly, the loss value of the target detection model to be trained is a parameter used to evaluate the target detection effect of the target detection network. The smaller the loss value, the better the target detection effect.

[0102] The network parameters of the target detection model to be trained are updated according to the loss value and the preset network parameter update method to obtain the updated target detection model. After each update, it is determined whether the updated target detection model to be trained meets the preset conditions. If it does, the model training is stopped, and the updated target detection model to be trained is used as the trained target detection model. If it does not meet the requirements, the step of inputting the sample image set into the target detection model to be trained is returned until the preset conditions are met, and the updated target detection model to be trained is determined as the trained target detection model. Among them, the preset network parameter update method can be any one of the error correction algorithms such as the gradient descent method and the back propagation algorithm. For example, the Adam (Adaptive Moment Estimation) algorithm. The preset condition can be that the loss value of the generated network reaches the loss threshold, or that the number of iterations reaches the number of iterations threshold, which is not limited here.

[0103] In this embodiment, the second feature map corresponding to the sample image set is extracted by the preprocessing unit of the target detection model to be trained, and the second feature map is subjected to attention-based pooling processing to obtain a second low-dimensional feature map corresponding to the second feature map. Thus, the second low-dimensional feature map is subjected to feature extraction by the feature extraction unit to obtain target capsule information corresponding to the sample image set, and then the target capsule information is subjected to target detection by the prediction unit to obtain the target detection result corresponding to the sample image set. By performing attention-based pooling processing on the second feature map, it is possible to reduce computational complexity and memory consumption while extracting effective information, greatly reduce training time, and accelerate model convergence speed. This enables the trained target detection model to reduce computational complexity and memory complexity when performing target detection on smaller target objects or partially occluded target objects.

[0104] In one embodiment, the sample image set is annotated with target label information; calculating the loss value of the target detection model to be trained based on the target detection results corresponding to the sample image set includes: binary matching the target detection results corresponding to the sample image set with the target label information to obtain matching results; and calculating the loss value of the target detection model to be trained based on the matching results.

[0105] The sample image set is annotated with target label information, and the target label information includes the category label of the target object in each sample image in the sample image set and the bounding box corresponding to the target object.

[0106] Specifically, the Hungarian algorithm can be used to perform binary matching on the target detection results corresponding to the sample image set and the target label information to obtain the matching results. The Hungarian algorithm can uniquely match the target detection results with the target label information. By combining the prediction unit with the Hungarian algorithm, multiple target objects can be predicted in parallel. Thus, the loss value of the target detection model to be trained is calculated according to the matching result. In one embodiment, the loss value of the target detection model to be trained includes a target position offset loss value, a classification loss value, and a matching loss value. The target position offset loss value refers to the loss of the position fitting between the border of the target object in the target detection result and the border of the corresponding target object in the target label information, which is used to improve the accuracy of the target object border detection. The target position offset loss value can be an IOU (intersection over union) loss. The classification loss value is the category loss value, and the commonly used cross entropy loss can be used to realize the multi-classification process of the target detection model and directly output the category of the target object. The matching loss value is used to realize the unique matching of the border of the target object in the target detection result and the border of the corresponding target object in the target label information. It is obtained by measuring the distance between the matching results, and is used to improve the matching accuracy of the target detection result and the target label information.

[0107] It should be understood that although Figures 2 to 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figures 2 to 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0108] In one embodiment, Figure 6 As shown, a target detection device is provided, including: an image acquisition module 602, a preprocessing module 604, a feature extraction module 606 and a target detection module 608, wherein:

[0109] The image acquisition module 602 is used to acquire the image to be detected.

[0110] The preprocessing module 604 is used to input the image to be detected into the trained target detection model, which includes a preprocessing unit, a feature extraction unit and a prediction unit; the first feature map corresponding to the image to be detected is extracted by the preprocessing unit to obtain a first low-dimensional feature map corresponding to the first feature map.

[0111] The feature extraction module 606 performs feature extraction on the first low-dimensional feature map through a feature extraction unit to obtain target capsule information corresponding to the image to be detected.

[0112] The target detection module 608 is used to perform target detection on the target capsule information through the prediction unit to obtain the target detection result corresponding to the image to be detected.

[0113] In one embodiment, the preprocessing module 604 is further used to extract features of the image to be detected through the convolutional neural network in the preprocessing unit, and determine the feature map output by the last two convolutional layers of the convolutional neural network as the first feature map corresponding to the image to be detected.

[0114] In one embodiment, the preprocessing module 604 is also used to perform multi-head attention calculation on the first feature map to obtain a multi-head attention value corresponding to the first feature map; and normalize the multi-head attention value to obtain a first low-dimensional feature map corresponding to the first feature map.

[0115] In one embodiment, the feature extraction unit includes an encoding unit and a decoding unit. The feature extraction module 606 is further used to perform global feature extraction on the first low-dimensional feature map through the encoding unit to obtain global feature information, perform capsule conversion on the global feature information to obtain initial capsule information; input the initial capsule information into the decoding unit, perform category feature extraction on the initial capsule information to obtain category feature information, perform capsule conversion on the category feature information to obtain target capsule information.

[0116] In one embodiment, the target detection module 608 is also used to perform target detection on the target capsule information based on the attention routing through the prediction unit to obtain a first detection result; perform linear transformation on the target capsule information through the prediction unit to obtain a second detection result; and fuse the first detection result and the second detection result to obtain a target detection result corresponding to the image to be detected.

[0117] In one embodiment, the above device further comprises:

[0118] The sample acquisition module is used to acquire a sample image set.

[0119] The sample preprocessing module is used to input the sample image set into the target detection model to be trained, extract the second feature map corresponding to the sample image set through the preprocessing unit in the target detection model to be trained, perform attention-based pooling processing on the second feature map, and obtain a second low-dimensional feature map corresponding to the second feature map.

[0120] The sample feature extraction module is used to extract features from the second low-dimensional feature map through a feature extraction unit in the target detection model to be trained, so as to obtain target capsule information corresponding to the sample image set.

[0121] The sample target detection module is used to perform target detection on the target capsule information corresponding to the sample image set through the prediction unit in the target detection model to be trained, and obtain the target detection result corresponding to the sample image set.

[0122] The parameter updating module is used to calculate the loss value of the target detection model to be trained according to the target detection results corresponding to the sample image set, and update the network parameters of the target detection model to be trained according to the loss value until the preset conditions are met to obtain the trained target detection model.

[0123] In one embodiment, the sample image set is annotated with target label information; the parameter updating module is also used to perform binary matching on the target detection results corresponding to the sample image set and the target label information to obtain matching results; and the loss value of the target detection model to be trained is calculated based on the matching results.

[0124] In one embodiment, the parameter updating module is also used to calculate the target position offset loss value, the classification loss value and the matching loss value of the target detection model to be trained.

[0125] For the specific definition of the target detection device, please refer to the definition of the target detection method above, which will not be repeated here. Each module in the above target detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a unit interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of a target detection method. The unit interface of the computer device is used to communicate with an external terminal through a unit connection. When the computer program is executed by the processor, a target detection method is implemented.

[0127] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above embodiments when executing the computer program.

[0129] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above embodiments are implemented.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0132] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire the image to be detected; Inputting the image to be detected into a trained target detection model, wherein the target detection model includes a preprocessing unit, a feature extraction unit and a prediction unit; Extracting a first feature map corresponding to the image to be detected by the preprocessing unit, performing attention-based pooling processing on the first feature map, and obtaining a first low-dimensional feature map corresponding to the first feature map; Performing feature extraction on the first low-dimensional feature map by the feature extraction unit to obtain target capsule information corresponding to the image to be detected; the target capsule information is feature information of the target object in the image to be detected represented by capsules; The prediction unit performs target detection on the target capsule information to obtain a target detection result corresponding to the image to be detected.

2. The method according to claim 1, characterized in that The extracting, by the preprocessing unit, a first feature map corresponding to the image to be detected comprises: The convolutional neural network in the preprocessing unit is used to extract features of the image to be detected, and the feature maps output by the last two convolutional layers of the convolutional neural network are determined as the first feature maps corresponding to the image to be detected.

3. The method according to claim 1, characterized in that The performing attention-based pooling processing on the first feature map to obtain a first low-dimensional feature map corresponding to the first feature map includes: Performing a multi-head attention calculation on the first feature map to obtain a multi-head attention value corresponding to the first feature map; The multi-head attention values ​​are normalized to obtain a first low-dimensional feature map corresponding to the first feature map.

4. The method according to claim 1, characterized in that: The feature extraction unit includes an encoding unit and a decoding unit; the feature extraction unit extracts features from the first low-dimensional feature map to obtain target capsule information corresponding to the image to be detected, including: Performing global feature extraction on the first low-dimensional feature map by the encoding unit to obtain global feature information, and performing capsule conversion on the global feature information to obtain initial capsule information; The initial capsule information is input into the decoding unit, category feature extraction is performed on the initial capsule information to obtain category feature information, and capsule conversion is performed on the category feature information to obtain target capsule information.

5. The method according to claim 1, characterized in that The performing target detection on the target capsule information by the prediction unit to obtain a target detection result corresponding to the image to be detected includes: Performing target detection on the target capsule information based on the attention routing by the prediction unit to obtain a first detection result; Performing a linear transformation on the target capsule information by the prediction unit to obtain a second detection result; The first detection result and the second detection result are fused to obtain a target detection result corresponding to the image to be detected.

6. The method according to claim 1, characterized in that Before acquiring the image to be detected, the method further includes: Get a sample image set; Inputting the sample image set into the target detection model to be trained, extracting a second feature map corresponding to the sample image set through a preprocessing unit in the target detection model to be trained, and performing attention-based pooling processing on the second feature map to obtain a second low-dimensional feature map corresponding to the second feature map; Extracting features from the second low-dimensional feature map by a feature extraction unit in the target detection model to be trained, to obtain target capsule information corresponding to the sample image set; Performing target detection on the target capsule information corresponding to the sample image set by a prediction unit in the target detection model to be trained, thereby obtaining a target detection result corresponding to the sample image set; The loss value of the target detection model to be trained is calculated according to the target detection results corresponding to the sample image set, and the network parameters of the target detection model to be trained are updated according to the loss value until the preset conditions are met to obtain the trained target detection model.

7. The method according to claim 6, characterized in that The sample image set is annotated with target label information; and the loss value of the target detection model to be trained is calculated according to the target detection result corresponding to the sample image set, including: Performing binary matching on the target detection result corresponding to the sample image set and the target label information to obtain a matching result; The loss value of the target detection model to be trained is calculated according to the matching result.

8. The method according to claim 7, characterized in that The loss value of the target detection model to be trained includes a target position offset loss value, a classification loss value, and a matching loss value.

9. A target detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be detected; A preprocessing module, used for inputting the image to be detected into a trained target detection model, wherein the target detection model includes a preprocessing unit, a feature extraction unit, and a prediction unit; extracting a first feature map corresponding to the image to be detected by the preprocessing unit, performing attention-based pooling processing on the first feature map, and obtaining a first low-dimensional feature map corresponding to the first feature map; a feature extraction module, configured to extract features from the first low-dimensional feature map through the feature extraction unit to obtain target capsule information corresponding to the image to be detected; the target capsule information is feature information of the target object in the image to be detected represented by capsules; The target detection module is used to perform target detection on the target capsule information through the prediction unit to obtain a target detection result corresponding to the image to be detected.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images