Target detection method, detection device and computer readable storage medium
By using different pixels within the target area for classification and localization detection, the classification and localization tasks are decoupled. The classification and localization branches of the target neural network are used to process features, which solves the problem of insufficient detection accuracy in the existing technology and achieves higher detection accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-12-01
- Publication Date
- 2026-07-10
AI Technical Summary
Existing target detection technologies suffer from feature conflicts in classification and localization tasks, resulting in insufficient detection accuracy.
By using different pixels within the target area for classification and localization detection, the classification and localization tasks are decoupled. The classification and localization branches in the target neural network are used to process the category prediction and distance prediction of the pixels respectively, and a feature discriminator is combined for decoupled feature training.
It improves the accuracy of target detection, alleviates the feature conflict problem in classification and localization tasks, and enhances the detection effect.
Smart Images
Figure CN116188789B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a target detection method, detection device, and computer-readable storage medium. Background Technology
[0002] Object detection technology, as a crucial upstream task in computer vision, has laid the foundation for downstream computer tasks. The main task of object detection is to locate objects of interest in images or videos and correctly classify these objects. Although object detection technology is now widely used in various fields of daily life and industrial production, such as intelligent transportation, smart homes, smart healthcare, and autonomous driving, its detection accuracy still needs further improvement. Summary of the Invention
[0003] This application provides a target detection method, detection device, and computer-readable storage medium, which can improve the accuracy of target detection.
[0004] A first aspect of this application provides a target detection method, the method comprising: acquiring a target feature map for feature extraction of a target image; determining a target category corresponding to the target region based on the predicted category corresponding to at least one first pixel in the target region, wherein the target region is the region selected by the sliding window each time it slides on the target feature map; determining a target detection box corresponding to the target region based on the predicted distance from at least one second pixel in the target region to a target detection box and the position of the at least one second pixel, wherein any first pixel and any second pixel do not overlap; and determining the category and position of a target object in the target image based on the target category corresponding to the target region and the target detection box.
[0005] A second aspect of this application provides a target detection device, which includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit, respectively. The memory stores program data, and the processor executes the program data in the memory to implement the steps in any of the above methods.
[0006] A third aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the steps in the above-described method.
[0007] The beneficial effects are: This application decouples classification detection and localization detection based on different pixels in the target area, thereby alleviating the feature conflict problem between classification and localization tasks and improving the accuracy of target detection. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:
[0009] Figure 1 This is a flowchart illustrating one embodiment of the target detection method of this application;
[0010] Figure 2 This is a schematic diagram of pixels and target detection boxes in existing technology;
[0011] Figure 3 This is a schematic diagram of the pixels and the target detection box in this application;
[0012] Figure 4 This is a schematic diagram of the structure of the target neural network in this application;
[0013] Figure 5 This is a schematic diagram of one embodiment of the target detection device of this application;
[0014] Figure 6 This is a schematic diagram of another embodiment of the target detection device of this application;
[0015] Figure 7 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation
[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0017] It should be noted that the terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0018] See Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the target detection method of this application, which includes:
[0019] S110: Obtain the target feature map for feature extraction of the target image.
[0020] Specifically, feature extraction is performed on the target image in advance to obtain a target feature map. In the process of feature extraction on the target image, multiple layers of feature extraction can be performed on the target image, and each layer yields a target feature map.
[0021] In other words, the target feature map obtained by feature extraction from the target image can be one or multiple maps, but regardless of whether there is one or multiple target feature maps, the same processing is performed on each target feature map. This application illustrates the processing of a single target feature map.
[0022] S120: Determine the target category corresponding to the target region based on the predicted category corresponding to at least one first pixel in the target region, wherein the target region is the region selected by the sliding window each time it slides on the target feature map.
[0023] Specifically, a sliding window of a preset size is used to traverse the target feature map. The area selected by the sliding window on the target feature map each time is defined as the target region. The size of the sliding window can be 2×2, 3×3, or 4×4, etc., without limitation, and the step size of each slide can be 1 pixel, 2 pixels, or 3 pixels, etc. It can be understood that when the sliding window's step size is 1 pixel, the center point of the sliding window coincides with each pixel on the target feature map in turn, thus obtaining a one-to-one correspondence between the target region and the pixels on the target feature map.
[0024] For each target area selected by the sliding window, classification detection and localization detection are performed. That is, steps S120 and S130 are executed for each target area. The following explanation focuses on processing a single target area. The purpose of classification detection is to determine the target category corresponding to the target area, and the purpose of localization detection is to determine the target detection box corresponding to the target area.
[0025] After obtaining the target category and target detection box corresponding to each target region, the category of each target object in the target image is determined based on the target category corresponding to all target regions in all target feature maps. The detection box of each target object in the target image, that is, the position of each target object, is determined based on the target detection box corresponding to all target regions in all target feature maps.
[0026] Meanwhile, there is at least one first pixel in the target region. When classifying and detecting the target region, the category of at least one first pixel in the target region is predicted to obtain the predicted category of each first pixel. Then, based on the predicted category of each first pixel, the target category corresponding to the target region is obtained.
[0027] When there is only one first pixel, its predicted category can be directly determined as the target category of the target region. When there are multiple first pixels, the predicted categories of the multiple first pixels can be combined to obtain the target category of the target region. For example, if there are three first pixels, two of which are predicted to be category A and one is predicted to be category B, then the target category of the target region is determined to be category A.
[0028] In this application, when performing category prediction, multi-category prediction can be performed. For example, if there are three categories: animal, human, and vehicle, when predicting the category of the first pixel, probability values 1, 2, and 3 are determined. Probability values 1, 2, and 3 represent the probabilities that the category corresponding to the first pixel is animal, human, or vehicle, respectively. Therefore, the predicted category corresponding to the first pixel can be a vector, which includes the probability values of each category corresponding to the first pixel. Correspondingly, the target category of the target region can also be a vector, which includes the probability values of each category corresponding to the target region.
[0029] S130: Determine the target detection box corresponding to the target region based on the predicted distance from each of the at least one second pixel point in the target region to the target detection box and the position of the at least one second pixel point, wherein any first pixel point and any second pixel point do not overlap.
[0030] Specifically, the target region may include one or more second pixels. When there is only one second pixel in the target region, the distance from the second pixel to each border of the target detection box is predicted. Then, based on the predicted distance and the position of the second pixel, the positions of each border of the target detection box are determined, and finally, the target detection box corresponding to the target region is determined. When there are multiple second pixels, the distances from multiple second pixels to the corresponding borders of the target detection box are predicted. Then, based on the predicted distance and the position of each second pixel, the positions of each border of the target detection box are determined, and finally, the target detection box corresponding to the target region is determined. For example, if there are two second pixels, the distance 1 from one second pixel to the top border and the distance 2 to the left border of the target detection box are predicted. Based on distances 1 and 2 and the position of the second pixel, the positions of the top and left borders of the target detection box can be determined. At the same time, the distance 3 from the other second pixel to the right border and the distance 4 to the bottom border of the target detection box are predicted. Based on distances 3 and 4 and the position of the second pixel, the positions of the right and bottom borders of the target detection box can be determined. Thus, based on the positions of the top, left, right, and bottom borders, the target detection box can be determined.
[0031] In short, the number of first pixels can be one or more, and can be set according to actual needs. There are no restrictions here.
[0032] In this embodiment, in the target area, no first pixel point overlaps with any second pixel point. That is to say, the pixels on which the classification detection and localization detection for the target area are based are different.
[0033] like Figure 2 As shown, in the existing technology, the prediction of the target category and the target detection box of the target region are based on the center pixel of the target region. For the center pixel, the corresponding category is predicted to obtain the target category of the target region, and the distance between the center pixel and the four corresponding target detection boxes is predicted. Then, the position of the target detection box is obtained based on these four distances and the position of the center pixel. The existing technology ignores the spatial conflict between the feature response points required for category detection and localization detection, and does not consider that localization features and category features are usually distributed on different pixels. Therefore, the accuracy of target detection in the existing technology needs to be further improved.
[0034] However, as can be seen from the above content of this application, this application uses different pixels for classification detection and localization detection of the target region. By decoupling classification detection and localization detection, the feature conflict problem between classification and localization tasks can be alleviated. Therefore, the method of this application can improve the accuracy of target detection.
[0035] S140: Determine the category and location of the target object in the target image based on the target category corresponding to the target region and the target detection box.
[0036] Specifically, the category of each target object in the target image is determined based on the target category corresponding to all target regions in all target feature maps. The detection box of each target object in the target image, i.e., the location of each target object, is determined based on the target detection box corresponding to all target regions in all target feature maps. The specific process of step S140 is prior art and will not be described in detail here.
[0037] In this embodiment, at least one first pixel includes the center pixel of the target region; at least one second pixel includes a plurality of pixels distributed around the center pixel.
[0038] Specifically, in combination Figure 3 Considering that the information required for category detection is generally semantic information that is distributed at the center of the object and has discriminative properties, and the central pixel of the target region contains rich semantic information, we perform category prediction on the central pixel to obtain the predicted category of the central pixel, and then obtain the target category of the target region based on the predicted category of the central pixel. For example, we determine the predicted category of the central pixel as the target category of the target region.
[0039] The information needed for localization detection is generally detailed information distributed along the object's edges, reflecting the object's boundaries. Therefore, at least one corresponding second pixel is set, including multiple pixels distributed around the central pixel. These multiple pixels are distributed around the central pixel, and the distances between these multiple pixels and the central pixel can be equal or unequal. Furthermore, the spacing between any two adjacent pixels can be equal or unequal; no restrictions are placed here. (Continue reading...) Figure 3 In this embodiment, at least one second pixel includes a first target pixel, a second target pixel, a third target pixel, and a fourth target pixel located at the upper left, upper right, lower left, and upper right corners of the center pixel, respectively. All four target pixels are adjacent to the center pixel. In this case, step S130 specifically includes:
[0040] S131: Determine the left border of the target detection box based on the first predicted distance L from the first target pixel to the left border of the target detection box and the position of the first target pixel.
[0041] Specifically, the first target pixel is located at the upper left corner of the center pixel. Considering that the pixel located at the upper left corner of the center pixel can better reflect the left boundary of the target, the first predicted distance L from the first target pixel to the left border of the target detection box is predicted. Then, based on the first predicted distance L and the position of the first target pixel, the left border of the target detection box is obtained.
[0042] S132: Determine the upper border of the target detection box based on the second predicted distance T from the second target pixel to the upper border of the target detection box and the position of the second target pixel.
[0043] Specifically, the second target pixel is located at the upper right corner of the center pixel. Considering that the pixel located at the upper right corner of the center pixel is better able to reflect the upper boundary of the target, the second predicted distance T from the second target pixel to the upper border of the target detection box is predicted. Then, based on the second predicted distance T and the position of the second target pixel, the upper border of the target detection box is determined.
[0044] S133: Determine the bottom border of the target detection box based on the third predicted distance D from the third target pixel to the bottom border of the target detection box and the position of the third target pixel.
[0045] Specifically, the third target pixel is located at the lower left corner of the center pixel. Considering that the pixel located at the lower left corner of the center pixel can better reflect the lower boundary of the target, the third predicted distance D from the third target pixel to the lower bound of the target detection box is predicted. Then, based on the third predicted distance D and the position of the third target pixel, the lower bound of the target detection box is obtained.
[0046] S134: Determine the right bounding box of the target detection box based on the fourth predicted distance R from the fourth target pixel to the right bounding box of the target detection box and the position of the fourth target pixel.
[0047] Specifically, the fourth target pixel is located at the lower right corner of the center pixel. Considering that the pixel located at the lower right corner of the center pixel can better reflect the right boundary of the target, the fourth predicted distance R from the fourth target pixel to the right bounding box of the target detection box is predicted. Then, based on the fourth predicted distance R and the position of the fourth target pixel, the right bounding box of the target detection box is obtained.
[0048] It should be noted that this application does not limit the second pixel. For example, in other embodiments, at least one second pixel may only include the first target pixel and the fourth target pixel. Then, based on the predicted distance from the first target pixel to the left border of the target detection box, the predicted distance to the top border of the target detection box, and the position of the first target pixel, the left border and the top border of the target detection box are determined. Based on the predicted distance from the fourth target pixel to the bottom border of the target detection box, the predicted distance to the right border of the target detection box, and the position of the fourth target pixel, the bottom border and the right border of the target detection box are determined.
[0049] In other implementations, the second pixel can be any pixel other than the first target pixel, the second target pixel, the third target pixel, and the fourth target pixel. The first pixel can also be any pixel other than the center pixel, as long as the first pixel and the second pixel are located in the target area and the second pixel does not overlap with the first pixel.
[0050] In this embodiment, when at least one second pixel includes multiple pixels distributed around the central pixel, if the central pixel of the target region is located on the boundary of the target feature map, then the multiple pixels distributed around the central pixel may not simultaneously exist on the target feature map. In this case, performing localization detection based on multiple second pixels would be meaningless. Therefore, in this embodiment, before step S130, the following step is also included:
[0051] S150: Determine whether the center pixel of the target region is on the boundary of the target feature map;
[0052] If the center pixel is on the boundary of the target feature map, then step S160 is executed; however, if the center pixel is not on the boundary of the target feature map, then step S130 is executed.
[0053] S160: Determine the target detection box based on the predicted distances from the center pixel to each border of the target detection box and the position of the center pixel.
[0054] Specifically, if the center pixel of the target region is on the boundary of the target feature map, the predicted distance from the center pixel to the four bounding boxes of the target detection box is directly predicted. Then, the target detection box is determined based on the four predicted distances and the position of the center pixel. However, if the center pixel is not on the boundary of the target feature map, step S130 is executed directly.
[0055] Combination Figure 4To improve the accuracy and efficiency of processing the target feature map, a target neural network 100 is used to process the target feature map. Specifically, in step S120, the classification branch 110 in the target neural network 100 predicts the predicted category of at least one first pixel in the target region, and simultaneously determines the target category of the target region based on the predicted category of at least one first pixel. In step S130, the localization branch 120 in the target neural network 100 predicts the predicted distance from at least one second pixel in the target region to the target detection box. After the localization branch 120 outputs the predicted value, the target detection box is determined based on the predicted distance from at least one second pixel to the target detection box and the position of at least one second pixel.
[0056] Specifically, after the target image is fed into the skeleton network, the skeleton network performs multi-layer feature extraction on the target image, obtaining a target feature map at each layer. Then, the target feature maps extracted from each layer are input into the target neural network 100 for processing. The target neural network 100 performs the same processing on each target feature map. The following explanation uses the processing of a single target feature map as an example:
[0057] After receiving the target feature map, the target neural network 100 uses the classification branch 110 to predict the target feature map and obtain the target category corresponding to each target region. The target category corresponding to each target region can be a vector, which includes the probability value of each category corresponding to the target region.
[0058] On the other hand, the localization branch 120 is used to predict the target feature map to obtain the distance from each second pixel in each target region to the target detection box corresponding to its target region. For example, when at least one second pixel in the target region includes a first target pixel, a second target pixel, a third target pixel, and a fourth target pixel, for each target region, the localization branch 120 can obtain the first predicted distance corresponding to the first target pixel, the second predicted distance corresponding to the second target pixel, the third predicted distance corresponding to the third target pixel, and the fourth predicted distance corresponding to the fourth target pixel.
[0059] The prediction process for each target region is the same for classification branch 110 and localization branch 120. For ease of explanation, the prediction of a target region by classification branch 110 and localization branch 120 is illustrated.
[0060] After the localization branch 120 outputs the predicted distance corresponding to each second pixel in the target area, the target detection box corresponding to each target area is determined based on the predicted distance and the position of each second pixel.
[0061] The training process of the target neural network 100 is described below:
[0062] S210: Obtain the sample feature map for feature extraction of the sample image.
[0063] S220: Use classification branch 110 to predict the predicted category of at least one third pixel in the sample region, and use classification branch 110 to determine the sample category of the sample region based on the predicted category of at least one third pixel. The sample region is the area selected by the sliding window each time it slides on the sample feature map.
[0064] Specifically, the process of classifying branch 110 processing the sample feature map is the same as the process of classifying branch 110 processing the target feature map.
[0065] S230: Utilize the localization branch 120 to predict the predicted distance from at least one fourth pixel in the sample region to the sample detection box, wherein the sample detection box corresponds to the sample region, and any third pixel does not overlap with any fourth pixel.
[0066] Specifically, the process of processing the sample feature map by the positioning branch 120 is the same as the process of processing the target feature map by the positioning branch 120 described above.
[0067] The distribution of at least one third pixel in the sample region is the same as the distribution of at least one first pixel in the target region, the distribution of at least one fourth pixel in the sample region is the same as the distribution of at least one second pixel in the sample region, and the position of at least one fourth pixel relative to at least one third pixel is the same as the positional relationship of at least one second pixel relative to at least one first pixel.
[0068] For example, if the first pixel in the target region is the center pixel of the target region and there are 4 second pixels, with the 4 second pixels located at the top left, top right, bottom left, and bottom right corners of the first pixel, then the third pixel in the sample region is also the center pixel of the sample region and there are also 4 fourth pixels, with the 4 fourth pixels located at the top left, top right, bottom left, and bottom right corners of the third pixel.
[0069] S240: Obtain the first loss value based on the sample category and label category corresponding to the sample region.
[0070] Specifically, during the training process, the category corresponding to the sample region is known, that is, the label category of the sample region. Therefore, based on the sample category and label category corresponding to the sample region, the first loss value can be obtained. It can be understood that the first loss value is the localization loss value.
[0071] Specifically, step S240 may involve obtaining a first loss value based on the sample category and label category corresponding to each sample region.
[0072] The first loss value can be calculated using any loss function, and this application does not impose any restrictions.
[0073] S250: The second loss value is obtained based on the predicted distance from each fourth pixel to the sample detection box and the first label distance from each fourth pixel to the sample detection box.
[0074] Specifically, during the training process, the distance to the sample detection box corresponding to the sample region is known, so the distance from each fourth pixel to the sample detection box is known, which is the distance from the fourth pixel to the first label of the sample detection box.
[0075] Therefore, based on the predicted distance corresponding to each fourth pixel and the corresponding first label distance, the second loss value can be obtained. It can be understood that the second loss value is the localization loss value.
[0076] Specifically, step S250 can be: calculating the second loss value of the entire target feature map based on the predicted distance of each fourth pixel in all target regions and the first label distance of each fourth pixel.
[0077] The second loss value can be calculated using any loss function, such as the smoothL1 loss function or the cross-union loss function, without any specific restrictions in this application.
[0078] Here, the sample detection box corresponding to the sample region refers to the detection box of the target object to which the center pixel of the sample region belongs. To illustrate this with an example: if the center pixel of the sample region is a pixel on target A (e.g., a person or a car), then the detection box corresponding to the sample region is the true detection box of target A.
[0079] If the center pixel of the sample region is not a pixel on any target, then the localization loss for the sample region is not calculated. In other words, if the center pixel of the sample region is not a pixel on any target, the localization loss value for the entire target feature map is calculated without considering the sample region.
[0080] In this case, considering that the distance from the center pixel of the sample region to the second label of the sample detection box is usually known, the first label distance corresponding to each fourth pixel in the sample region can be determined based on the distance from the center pixel to the second label of the sample detection box and the target distance between the center pixel and the fourth pixel.
[0081] Specifically, to facilitate understanding, we will illustrate this with examples:
[0082] In this example, the first pixel in the target region is the center pixel of the target region, and there are four second pixels, including the first, second, third, and fourth target pixels. Correspondingly, the third pixel in the sample region is also the center pixel of the sample region, and there are four fourth pixels located at the top left, top right, bottom left, and bottom right corners of the third pixel, respectively. These four fourth pixels are also adjacent to the third pixel. Therefore, during training, the center pixel of each sample region is first determined. The distances from the point to the left, top, bottom, and right borders of the sample detection box are L1, T1, D1, and R1, respectively. Therefore, for the pixel located at the top left corner of the center pixel, its corresponding first label distance is equal to (L1-1), for the pixel located at the top right corner of the center pixel, its corresponding first label distance is equal to (T1-1), for the pixel located at the bottom left corner of the center pixel, its corresponding first label distance is equal to (D1-1), and for the pixel located at the bottom right corner of the center pixel, its corresponding first label distance is equal to (R1-1).
[0083] S260: Generate the total loss value based on the first loss value and the second loss value.
[0084] Specifically, the first loss value and the second loss value can be subjected to various operations, such as direct summation, weighted summation, or averaging, to obtain the total loss value.
[0085] S270: Train the target neural network 100 based on the total loss value.
[0086] Specifically, by following the steps above, the total loss value corresponding to each target feature map can be obtained. Finally, by calculating the total loss values corresponding to all target feature maps, the target loss value can be obtained. Then, the target neural network 100 is trained based on the target loss value.
[0087] Continue reading Figure 4 In this embodiment, the classification branch 110 includes a classification feature extractor 111 and a classifier 112, and the localization branch 120 includes a localization feature extractor 121 and a regressor 122.
[0088] The process of classifying the target feature map by the classification branch 110 includes: extracting classification features from the sample feature map using the classification feature extractor 111 to obtain the classification feature map; performing classification prediction on the classification feature map using the classifier 112 to obtain the predicted category of at least one third pixel in the sample region; and determining the sample category of the sample region based on the predicted category of at least one third pixel.
[0089] Specifically, the classification feature extractor 111 may include multiple cascaded convolutional layers to process the target feature map and obtain a classification feature map. In this embodiment, the target feature map has a dimension of H×W, and the classification feature map has a dimension of H×W×256 (meaning the classification feature map includes H×W pixels, and each pixel has 256 channels of pixel values). This application does not limit the specific processing procedure of the classification feature extractor 111.
[0090] After the classification feature extractor 111 outputs a classification feature map, this map is fed into the classifier 112, which performs classification prediction on the feature map. During classification prediction, the classifier 112 uses a sliding window to select multiple sample regions sequentially on the feature map. When the sliding window's step size is 1, the number of sample regions is equal to the number of pixels in the classification feature map, and each sample region corresponds one-to-one with a pixel in the feature map. In other words, if the classification feature map has a dimension of H×W×256, if the sliding window's step size is 1, and if the target neural network 100110 can simultaneously detect C categories, the classifier 112 outputs H×W vectors, each containing C probability values. These H×W vectors can form a feature map of dimension H×W×C, where the value in the same channel of the feature map represents the probability value of each sample region corresponding to the same category. The channel values of a pixel in the feature map represent the probability values of each category for pixels at the same position in the classification feature map.
[0091] Meanwhile, the process of the localization branch 120 in processing the target feature map includes: using the localization feature extractor 121 to extract localization features from the sample feature map to obtain the localization feature map; and using the regressor 122 to perform regression prediction on the localization feature map to obtain the predicted distance from at least one fourth pixel point to the sample detection box.
[0092] Specifically, similar to the classification feature extractor 111, the localization feature extractor 121 may include multiple cascaded convolutional layers to process the target feature map and obtain a localization feature map. In this embodiment, the target feature map has a dimension of H×W, and the localization feature map has a dimension of H×W×256 (meaning the localization feature map includes H×W pixels, and each pixel has 256 channels of pixel values). This application does not limit the specific processing procedure of the localization feature extractor 121.
[0093] After the localization feature extractor 121 outputs a localization feature map, this map is fed into the regressor 122, which performs regression prediction on the localization feature map. Specifically, when the sliding window's step size is 1, the number of sample regions is equal to the number of pixels included in the localization feature map, and the sample regions correspond one-to-one with the pixels in the localization feature map. That is, if the localization feature map has a dimension of H×W×256, and if the sliding window's step size is equal to 1, and in each sample region there are 4 fourth pixels, distributed at the top left, top right, bottom left, and bottom right corners of the third pixel and adjacent to it, then the regressor 122 outputs H×W vectors, each containing 4 distance values. These H×W vectors can form a feature map with a dimension of H×W×4, where the value in the same channel of this feature map represents the predicted distance value corresponding to the fourth pixel at the same position in each sample region. The channel values of a pixel in the feature map represent the distance values of the fourth pixel in the sample region corresponding to the pixel at the same position in the feature map. The sample region corresponding to the pixel refers to the area selected when the center point of the sliding window coincides with the pixel.
[0094] In this embodiment, combined with Figure 4 Step S260 specifically includes:
[0095] S261: Input the classification feature map into the feature discriminator 130 to obtain the first determination vector corresponding to each first pixel in the classification feature map. The first determination vector includes a first probability value and a second probability value. The first probability value and the second probability value represent the probability that the first pixel comes from the classification feature map and the localization feature map, respectively.
[0096] S262: Input the localization feature map into the feature discriminator 130 to obtain the second determination vector corresponding to each second pixel in the localization feature map. The second determination vector includes a third probability value and a fourth probability value, which represent the probability that the second pixel comes from the classification feature map and the localization feature map, respectively.
[0097] S263: Determine the third loss value based on the first decision vector and the first supervision label corresponding to each first pixel, and the second decision vector and the second supervision label corresponding to each second pixel, wherein the first supervision label and the second supervision label indicate whether the first pixel and the second pixel come from the classification feature map or the localization feature map, respectively.
[0098] S264: The total loss value is obtained based on the first loss value, the second loss value, and the third loss value.
[0099] Specifically, to further decouple the classification and localization tasks, this application also includes a feature discriminator 130. The feature discriminator 130 determines whether the first pixel in the classification feature map comes from the classification branch 110 or the localization branch 120. After the classification feature map is input into the feature discriminator 130, the feature discriminator 130 outputs a vector with two channels for each first pixel in the classification feature map. This vector is the first determination vector corresponding to the first pixel. The first dimension of this vector represents the probability that the first pixel comes from the classification branch 110, and the second dimension represents the probability that the first pixel comes from the localization branch 120. In other words, the first determination vector includes a first probability value and a second probability value, which respectively represent the probability that the first pixel comes from the classification feature map and the localization feature map.
[0100] Similarly, the localization feature map is also input into the feature discriminator 130, which determines whether the second pixel in the localization feature map comes from the classification branch 110 or the localization branch 120. Specifically, after inputting the localization feature map into the feature discriminator 130, the feature discriminator 130 outputs a vector with two channels for each second pixel in the localization feature map. This vector is the second determination vector corresponding to the second pixel. The first dimension of this vector represents the probability that the second pixel comes from the classification branch 110, and the second dimension represents the probability that the second pixel comes from the localization branch 120. In other words, the second determination vector includes a third probability value and a fourth probability value, which respectively represent the probability that the second pixel comes from the classification feature map and the localization feature map.
[0101] In this embodiment, since the features of the two tasks are decoupled, that is, it is only necessary to determine whether the pixel comes from the classification branch 110 or the localization branch 120, this application designs a simple supervision label: if the first pixel comes from the classification feature map, the first supervision label of the first pixel is set to 1, otherwise it is set to 0. Similarly, if the second pixel comes from the classification feature map, the second supervision label of the second pixel is set to 1, otherwise it is set to 0.
[0102] After the above steps, each first pixel corresponds to a first decision vector and a first supervision label (a one-to-one correspondence is formed between the first decision vector and the first supervision label), and each second pixel corresponds to a second decision vector and a second supervision label (a one-to-one correspondence is formed between the second decision vector and the second supervision label), so that the loss can be calculated and the third loss value can be obtained.
[0103] The third loss value can be calculated using any loss function, such as Focal loss, without any restrictions.
[0104] Finally, by combining the first, second, and third loss values, the total loss value can be obtained. The total loss value can be calculated by performing any of the following methods on the first, second, and third loss values: direct summation, weighted summation, or averaging.
[0105] To avoid overfitting of the feature discriminator 130 during training, when calculating the third loss value, all the first and second decision vectors are first mixed together and their order is shuffled before the third loss value is calculated.
[0106] Specifically, the shuffling can be done according to preset rules or randomly; this application does not impose any restrictions. For understanding, an example is provided below:
[0107] Suppose there are 8 first pixels and 8 second pixels. The first decision vectors corresponding to these 8 first pixels are denoted as A1, A2, A3, A4, A5, A6, A7, and A8, respectively. The second decision vectors corresponding to these 8 second pixels are denoted as B1, B2, B3, B4, B5, B6, B7, and B8, respectively. After mixing and shuffling, the order of all decision vectors can be either A1, A2, B1, B2, B3, B4, A3, A4, A5, B5, B6, A6, A7, B7, A8, and B8, or B1, B2, A1, A2, B3, A3, B4, B5, A4, B6, A5, B7, A6, A7, B8, and A8.
[0108] During the process of shuffling the order, the correspondence between the first decision vector and the first supervision label remains unchanged, and the correspondence between the second decision vector and the second supervision label remains unchanged. That is to say, the corresponding first decision vector and the first supervision label always correspond to a first pixel at the same time, and the corresponding second decision vector and the second supervision label always correspond to a second pixel at the same time.
[0109] This embodiment scrambles the order of the first and second decision vectors to avoid overfitting of the feature discriminator 130 during the training of the target neural network 100. However, in other embodiments, the order of the first and second decision vectors may not be scrambled.
[0110] As can be seen from the above, this application sets a feature discriminator 130 in the target neural network 100. The feature discriminator 130 is used to determine whether the feature of each pixel is a classification-aware feature or a localization-aware feature, so that the target neural network 100 can decouple the classification feature and the localization feature at the pixel level, thereby further decoupling the classification task and the localization task, and ensuring the detection accuracy of the trained target neural network 100.
[0111] To facilitate a further understanding of the structure of the target neural network 100 of this application, the following detailed explanation of the processing process of the target feature map by the target neural network 100 is provided with examples:
[0112] In this example, the first pixel is the center pixel of the target region, and at least one second pixel includes a first target pixel, a second target pixel, a third target pixel, and a fourth target pixel. For each target region, the predicted category of the center pixel is directly determined as the target category of the target region, and it is assumed that the sliding window's step size is one pixel.
[0113] After inputting the target feature map of dimension H×W into the target neural network 100, the classification feature extractor 111 extracts features from the target feature map to obtain a classification feature map of dimension H×W×256. Then, the classifier 112 performs classification prediction on the classification feature map. During the prediction process, the classifier 112 uses a sliding window to select multiple target regions on the classification feature map in sequence. At this time, the selected target regions correspond one-to-one with the pixels on the classification feature map. Finally, the classifier 112 outputs H×W C-dimensional vectors, where the first dimension of the vector represents the probability of the target region corresponding to class C1, the second dimension represents the probability of the target region corresponding to class C2, and so on, with the Cth dimension representing the probability of the target region corresponding to class C. C The probability of each category. In other words, the target neural network 100 can perform multi-class prediction, with the number of categories being C.
[0114] After inputting the target feature map of dimension H×W into the target neural network 100, the localization feature extractor 121 extracts features from the target feature map to obtain a localization feature map of dimension H×W×256. Then, the regressor 122 performs regression prediction on the localization feature map. During the prediction process, the regressor 122 slides a sliding window on the localization feature map to sequentially select multiple target regions. At this time, the selected multiple target regions correspond one-to-one with the pixels on the localization feature map. Finally, the regressor 122 outputs H×W 4-dimensional vectors. The first dimension of the vector represents the first predicted distance from the first target pixel in the target region to the left border of the target detection box. The second dimension of the vector represents the second predicted distance from the second target pixel in the target region to the top border of the target detection box. The third dimension of the vector represents the third predicted distance from the third target pixel in the target region to the bottom border of the target detection box. The fourth dimension of the vector represents the fourth predicted distance from the fourth target pixel in the target region to the right border of the target detection box.
[0115] Finally, based on the position of each second target pixel and its corresponding prediction distance for each target region, the target detection box corresponding to each target region can be determined.
[0116] Thus, through the above steps, the target category and target detection box corresponding to each sample region on the target feature map can be obtained. Therefore, after inputting the target feature map of each layer into the target neural network 100, the target category and target detection box corresponding to each sample region in each target feature map can be obtained. Finally, based on the target category and target detection box corresponding to each sample region in all target feature maps, the category and location of each target object in the target image can be obtained, simultaneously achieving the classification and localization tasks.
[0117] See Figure 5 , Figure 5 This is a schematic diagram of one embodiment of the target detection device of this application. The target detection device 200 includes a processor 210, a memory 220 and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230 respectively. The memory 220 stores program data. The processor 210 executes the program data in the memory 220 to implement the steps in any of the above embodiments. The detailed steps can be found in the above embodiments and will not be repeated here.
[0118] The target detection device 200 can be any device with image processing capabilities, such as a computer or mobile phone, and there are no restrictions on it.
[0119] See Figure 6 , Figure 6This is a schematic diagram of another embodiment of the target detection device of this application. The target detection device 300 includes an acquisition module 310, a first determination module 320, a second determination module 330 and a third determination module 340 connected in sequence.
[0120] The acquisition module 310 is used to acquire the target feature map for feature extraction of the target image.
[0121] The first determining module 320 is used to determine the target category corresponding to the target region based on the predicted category corresponding to at least one first pixel in the target region, wherein the target region is the region selected by the sliding window each time it slides on the target feature map.
[0122] The second determining module 330 is used to determine the target detection box corresponding to the target region based on the predicted distance from each of the at least one second pixel point in the target region to the target detection box and the position of the at least one second pixel point, wherein any first pixel point and any second pixel point do not overlap.
[0123] The third determining module 340 is used to determine the category and location of the target object in the target image based on the target category corresponding to the target region and the target detection box.
[0124] The target detection device 300 can be any device with image processing capabilities, such as a computer or mobile phone, and there are no restrictions on it.
[0125] When the target detection device 300 is in operation, it executes the method steps of any of the above embodiments. For detailed steps, please refer to the relevant content above, which will not be repeated here.
[0126] See Figure 7 , Figure 7 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.
[0127] Specifically, the computer-readable storage medium 400 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a device that can store the computer program 410. Alternatively, it can be a server that stores the computer program 410, which can send the stored computer program 410 to other devices for execution, or it can run the stored computer program 410 itself.
[0128] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A target detection method, characterized in that, The method includes: Obtain the target feature map for feature extraction from the target image; The target category corresponding to the target region is determined based on the predicted category corresponding to at least one first pixel in the target region, wherein the target region is the region selected by the sliding window each time it slides on the target feature map; The target detection box corresponding to the target region is determined based on the predicted distance from each of the at least one second pixel in the target region to the target detection box and the position of the at least one second pixel, wherein any first pixel and any second pixel do not overlap; Based on the target category corresponding to the target region and the target detection box, determine the category and location of the target object in the target image; Wherein, the at least one first pixel includes the center pixel of the target region; the at least one second pixel includes a plurality of pixels distributed around the center pixel; The method further includes, before determining the target detection box corresponding to the target region based on the predicted distance from each of the at least one second pixel in the target region to the target detection box and the position of the at least one second pixel: Determine whether the center pixel of the target region is located on the boundary of the target feature map; If it is, then the target detection box is determined based on the predicted distance from the center pixel to each border of the target detection box and the position of the center pixel; Otherwise, the step of determining the target detection box corresponding to the target region based on the predicted distance from each of the at least one second pixel in the target region to the target detection box and the position of the at least one second pixel is performed.
2. The method according to claim 1, characterized in that, The at least one second pixel includes a first target pixel, a second target pixel, a third target pixel, and a fourth target pixel located at the upper left, upper right, lower left, and upper right corners of the center pixel, respectively, and the first target pixel, the second target pixel, the third target pixel, and the fourth target pixel are adjacent to the center pixel. The step of determining the target detection box corresponding to the target region based on the predicted distance from each of at least one second pixel in the target region to the target detection box and the position of the at least one second pixel includes: The left border of the target detection box is determined based on the first predicted distance from the first target pixel to the left border of the target detection box and the position of the first target pixel. The upper border of the target detection box is determined based on the second predicted distance from the second target pixel to the upper border of the target detection box and the position of the second target pixel. The bottom border of the target detection box is determined based on the third predicted distance from the third target pixel to the bottom border of the target detection box and the position of the third target pixel. The right bounding box of the target detection box is determined based on the fourth predicted distance from the fourth target pixel to the right bounding box of the target detection box and the position of the fourth target pixel.
3. The method according to claim 1, characterized in that, The step of determining the target category corresponding to the target region based on the predicted category corresponding to at least one first pixel in the target region includes: The classification branch in the target neural network is used to predict the predicted category of each of the at least one first pixel in the target region, and the classification branch is used to determine the target category of the target region based on the predicted category of each of the at least one first pixel. The step of determining the target detection box corresponding to the target region based on the predicted distance from each of at least one second pixel in the target region to the target detection box and the position of the at least one second pixel includes: The localization branch in the target neural network is used to predict the predicted distance from each of the at least one second pixel in the target region to the target detection box; The target detection box is determined based on the predicted distance of each of the at least one second pixel point to the target detection box and the position of the at least one second pixel point.
4. The method according to claim 3, characterized in that, The method further includes: Obtain the sample feature map from the sample image for feature extraction; The classification branch is used to predict the predicted category of at least one third pixel in the sample region, and the classification branch is used to determine the sample category of the sample region based on the predicted category of the at least one third pixel. The sample region is the area selected by the sliding window each time it slides on the sample feature map. The localization branch is used to predict the predicted distance from at least one fourth pixel in the sample region to the sample detection box, wherein the sample detection box corresponds to the sample region, and any third pixel does not coincide with any fourth pixel. The first loss value is obtained based on the sample category and label category corresponding to the sample region; The second loss value is obtained based on the predicted distance from each fourth pixel to the sample detection box and the first label distance from each fourth pixel to the sample detection box. A total loss value is generated based on the first loss value and the second loss value; The target neural network is trained based on the total loss value.
5. The method according to claim 4, characterized in that, The classification branch includes a classification feature extractor and a classifier; the localization branch includes a localization feature extractor and a regressor. The step of predicting the predicted category of at least one third pixel in the sample region using the classification branch, and determining the sample category of the sample region based on the predicted category of the at least one third pixel, includes: The classification feature extractor is used to extract classification features from the sample feature map to obtain a classification feature map. The classifier is used to perform classification prediction on the classification feature map to obtain the predicted category of each of the at least one third pixel in the sample region. At the same time, the classifier determines the sample category of the sample region based on the predicted category of each of the at least one third pixel. The step of predicting the distance from at least one fourth pixel in the sample region to the sample detection box using the localization branch includes: The localization feature extractor is used to extract localization features from the sample feature map to obtain a localization feature map. The regression analyzer is used to perform regression prediction on the localization feature map to obtain the predicted distance from the at least one fourth pixel point to the sample detection box.
6. The method according to claim 5, characterized in that, The step of generating a total loss value based on the first loss value and the second loss value includes: The classification feature map is input into the feature discriminator to obtain a first determination vector corresponding to each first pixel in the classification feature map. The first determination vector includes a first probability value and a second probability value. The first probability value and the second probability value represent the probability that the first pixel comes from the classification feature map and the localization feature map, respectively. The localization feature map is input into the feature discriminator to obtain a second determination vector corresponding to each second pixel in the localization feature map. The second determination vector includes a third probability value and a fourth probability value, which respectively represent the probability that the second pixel comes from the classification feature map and the localization feature map. A third loss value is determined based on the first decision vector and first supervision label corresponding to each first pixel, and the second decision vector and second supervision label corresponding to each second pixel, wherein the first supervision label and the second supervision label respectively indicate whether the first pixel and the second pixel come from the classification feature map or the localization feature map; The total loss value is obtained based on the first loss value, the second loss value, and the third loss value.
7. The method according to claim 6, characterized in that, Before determining the third loss value based on the first decision vector and first supervision label corresponding to each first pixel, and the second decision vector and second supervision label corresponding to each second pixel, the method further includes: After arranging all the first decision vectors and all the second decision vectors together, shuffle the order.
8. The method according to claim 4, characterized in that, Before obtaining the second loss value based on the predicted distance from each of the fourth pixels to the sample detection box and the first label distance from each of the fourth pixels to the sample detection box, the method further includes: Obtain the distance from the center pixel of the sample region to the second label of the sample detection box, wherein when the sliding window selects the sample region, the center point of the sliding window coincides with the center pixel of the sample region; Based on the second label distance and the target distance corresponding to the fourth pixel, the first label distance from the fourth pixel to the sample detection box is determined, wherein the target distance corresponding to the fourth pixel is the distance between the fourth pixel and the center pixel of the sample region.
9. A target detection device, characterized in that, The target detection device includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit. The memory stores program data. The processor executes the program data in the memory to implement the steps of the method as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.