Target detection method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a target detection method and device, electronic equipment and a storage medium; including: performing feature extraction on a to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, the feature maps including feature pixels; for each feature map in the plurality of feature maps, performing detection based on a plurality of branches to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of respective training data sets, and the plurality of training data sets each include different categories; obtaining a target detection result based on the branch detection result of the feature pixel, and displaying the target detection result. Embodiments of the present application realize target detection of different category sets through a plurality of branches respectively, and since a plurality of branches of one model are utilized, the number of models is reduced while the accuracy of multi-category detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, and in particular to a target detection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] For a picture, identifying multiple targets of different categories in the picture can help users more quickly and comprehensively obtain information in the picture. In the prior art, when identifying multiple targets of different categories in a picture, the following two methods are usually used:

[0003] 1. A single category training data set is constructed for each category, and an image recognition model is trained using the training data set to obtain multiple image recognition models for identifying targets of each category; 2. Multiple category training data sets are combined into a training data set, and an image recognition model is trained using the training data set, so that the image recognition model is used to identify all categories.

[0004] However, for the prior art 1, when the number of categories to be identified is large, the total number of image recognition models is also large, the total number of model parameters is large, and a large amount of computing resources is consumed. For the prior art 2, because the number of categories to be trained is large, the image recognition model trained has the problem of inaccurate detection. SUMMARY

[0005] The embodiments of the present application provide a target detection method, device, electronic equipment and storage medium, which can reduce the number of model parameters while achieving better detection accuracy.

[0006] The embodiments of the present application provide a target detection method, which comprises: performing feature extraction on a to-be-detected picture to obtain multiple feature maps of different sizes corresponding to the to-be-detected picture, the feature maps comprising feature pixels;

[0007] For each feature map of the multiple feature maps, a detection based on multiple branches is performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the multiple branches are trained by a respective training data set, and the multiple training data sets each comprise different categories.

[0008] A target detection result is obtained based on the branch detection result of the feature pixels, and the target detection result is displayed.

[0009] The embodiments of the present application also provide a target detection device, which comprises:

[0010] The feature map acquisition unit is configured to perform feature extraction on the to-be-detected picture, and acquire a plurality of feature maps of different sizes corresponding to the to-be-detected picture, wherein the feature maps include feature pixels.

[0011] The branch result acquisition unit is configured to perform branch-based detection on each feature map in the plurality of feature maps, and obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of training data sets respectively, and the plurality of training data sets respectively include different categories.

[0012] The target result acquisition unit is configured to obtain a target detection result based on the branch detection result of the feature pixel, and display the target detection result.

[0013] In some embodiments, the branch result acquisition unit is specifically configured to perform convolution processing on the first feature map by using a first branch to obtain a probability value of M categories corresponding to each pixel point on the first feature map, a weight value of each pixel point and a target center point of the pixel point, and N bounding box distance values corresponding to each pixel point, wherein the first branch is any branch in the plurality of branches, the training data set corresponding to the first branch includes the M categories, the first feature map is any feature map in the plurality of feature maps, and M and N are positive integers.

[0014] In some embodiments, the target result acquisition unit includes:

[0015] The probability value determination sub-unit is configured to determine a probability value with a maximum value among the probability values of the M categories corresponding to each pixel point, take the probability value with the maximum value as a binding probability value of the corresponding pixel point, and take a category corresponding to the binding probability value as a binding category of the pixel point.

[0016] The probability value sorting sub-unit is configured to sort each pixel point on the first feature map according to the value of the binding probability value.

[0017] The pixel point determination sub-unit is configured to determine P pixel points with the maximum binding probability value from the pixel points on the first feature map, wherein P is a positive integer.

[0018] The detection box determination sub-module is configured to determine P detection boxes corresponding to the P pixel points respectively according to the N bounding box distance values corresponding to each pixel point, wherein the P detection boxes and the binding categories corresponding to the P pixel points respectively are the target detection result of the first feature map in the first branch.

[0019] In some embodiments, the feature map acquisition unit includes:

[0020] a downsampling processing subunit, configured to perform downsampling processing of different scales on the to-be-detected picture for a first number of times to obtain a first number of downsampling results, the first number of downsampling results having different sizes;

[0021] a primary selection subunit, configured to perform convolution processing on the first number of downsampling results to obtain a primary feature map corresponding to each downsampling result;

[0022] a secondary selection subunit, configured to perform downsampling processing on a primary feature map with the smallest size for a second number of times to obtain a second number of secondary feature maps, wherein the first number of primary feature maps and the second number of secondary feature maps together form the plurality of feature maps with different sizes.

[0023] In some embodiments, the apparatus further includes:

[0024] a training data set acquisition unit, configured to acquire a plurality of training data sets, wherein the plurality of training data sets correspond one-to-one to the plurality of branches, and the plurality of training data sets each include different categories;

[0025] a network training unit, configured to train the branch corresponding to the training data set by using the plurality of training data sets until the entire target detection network meets a preset condition, to obtain a trained target detection network.

[0026] In some embodiments, the network training unit includes:

[0027] a training feature map subunit, configured to, for each branch, perform feature extraction on a training picture from the training data set corresponding to the branch to acquire a plurality of training feature maps with different sizes corresponding to the training picture;

[0028] a training detection result subunit, configured to detect each training feature map in the plurality of training feature maps by using the branch to acquire a training detection result;

[0029] a branch loss value subunit, configured to determine a branch loss value corresponding to the branch according to the training detection result, a prediction result of a target teacher model on the training picture, and a labeled box in the training picture, wherein the target teacher model is a teacher model corresponding to the branch, and the target teacher model is a model that has completed training on the training data set;

[0030] a total loss value subunit, configured to determine the branch loss value corresponding to each branch in the plurality of branches, calculate a weighted sum of the branch loss value corresponding to each branch, and obtain a total loss value;

[0031] A determination first sub-unit is configured to determine that the plurality of branches are trained when the total loss value meets the preset condition.

[0032] A determination second sub-unit is configured to adjust the parameters of each branch according to the total loss value when the total loss value does not meet the preset condition, and iterates until the total loss value meets the preset condition.

[0033] In some embodiments, the training detection result sub-unit is specifically configured to perform convolution processing on each training feature map by the branch to obtain a probability value of M' categories corresponding to each pixel point on the training feature map, a weight value of the each pixel point and a target center point of the each pixel point, and N' bounding box distance values corresponding to the each pixel point; wherein the training data set from which the training picture comes includes the M' categories, and M' and N' are positive integers.

[0034] In some embodiments, the device further comprises:

[0035] A data set division unit is configured to divide a preset data set into the first training data set and the second training data set.

[0036] In some embodiments, the data set division unit comprises:

[0037] A category division sub-unit is configured to divide all Q categories in the preset data set into the first training data set and the second training data set, wherein Q is a positive integer.

[0038] A picture division sub-unit is configured to divide all pictures in the preset data set into pictures in which categories are only contained in the first training data set, pictures in which categories are only contained in the second training data set, and pictures in which categories are contained in both the first training data set and the second training data set.

[0039] A picture equal division sub-unit is configured to equally divide the pictures in which categories are contained in both the first training data set and the second training data set into the first training data set and the second training data set.

[0040] In some embodiments, the device further comprises:

[0041] A first category deletion unit is configured to delete information related to categories of the second training data set from all pictures in the first training data set.

[0042] A second category deletion unit is configured to delete information related to categories of the first training data set from all pictures in the second training data set.

[0043] In some embodiments, the category dividing unit comprises:

[0044] a current category sub-unit configured to obtain a current category divided into the first training data set or the second training data set;

[0045] a to-be-classified category sub-unit configured to obtain, in all pictures in the preset data set, a co-occurrence frequency of each of the unassigned categories of the preset data set and the current category appearing in a same picture, and determine a category with a maximum co-occurrence frequency as a to-be-classified category;

[0046] a first sum sub-unit configured to determine, in all pictures in the preset data set, a first co-occurrence frequency of the to-be-classified category and each of the categories in the first training data set appearing in a same picture, and obtain a sum of a plurality of the first co-occurrence frequencies to obtain a first sum;

[0047] a second sum sub-unit configured to determine, in all pictures in the preset data set, a second co-occurrence frequency of the to-be-classified category and each of the categories in the second training data set appearing in a same picture, and obtain a sum of a plurality of the second co-occurrence frequencies to obtain a second sum;

[0048] a first division sub-unit configured to divide the to-be-classified category into the second training data set as a new current category when the first sum is greater than or equal to the second sum;

[0049] a second division sub-unit configured to divide the to-be-classified category into the first training data set as a new current category when the first sum is less than the second sum;

[0050] a step jump sub-unit configured to jump to the step of obtaining the current category divided into the first training data set or the second training data set until the division of all Q categories in the preset data set is completed.

[0051] In some embodiments, the apparatus further comprises:

[0052] a co-occurrence frequency calculation unit configured to, for each of the Q categories in the preset data set, determine, in all pictures in the preset data set, a co-occurrence frequency of each of the other Q-1 categories appearing in a same picture, to obtain Q-1 co-occurrence frequencies;

[0053] an accumulated value calculation unit configured to calculate an accumulated value of the Q-1 co-occurrence frequencies, the accumulated value corresponding to the category;

[0054] An accumulated value obtaining unit is configured to obtain Q accumulated values corresponding to the Q categories respectively.

[0055] A category storage unit is configured to put the category corresponding to the accumulated value with the largest value among the Q accumulated values into the first training data set.

[0056] The embodiment of the present application further provides an electronic device, including a memory storing a plurality of instructions; and a processor loading the instructions from the memory to perform the steps in any of the target detection methods provided by the embodiment of the present application.

[0057] The embodiment of the present application further provides a computer readable storage medium storing a plurality of instructions, and the instructions are suitable for being loaded by a processor to perform the steps in any of the target detection methods provided by the embodiment of the present application.

[0058] The target detection method provided by the embodiment of the present application performs feature extraction on a to-be-detected picture, obtains a plurality of feature maps with different sizes corresponding to the to-be-detected picture, and the feature maps include feature pixels; for each feature map in the plurality of feature maps, a detection based on a plurality of branches is performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of training data sets corresponding to the branches respectively, and the categories included in the plurality of training data sets are different from each other; a target detection result is obtained based on the branch detection result of the feature pixel, and the target detection result is displayed.

[0059] In the present application, feature extraction is first performed on a to-be-detected picture to obtain a plurality of feature maps corresponding to the to-be-detected picture, and the sizes of the plurality of feature maps are different from each other; then for each feature map, a corresponding branch detection result is obtained based on each branch of a plurality of branches, and a plurality of branch detection results are obtained. Since different branches have a training data set corresponding to each branch, and the categories included in the plurality of training data sets are different from each other, the targets corresponding to the categories presented in each branch detection result are different from each other. The plurality of branch detection results are summarized to obtain a detection result corresponding to a larger category set composed of a plurality of category sets. The embodiment of the present application realizes target detection of different category sets through a plurality of branches respectively, and since a plurality of branches of one model are utilized, the number of models is reduced while the accuracy of the plurality of category detections is improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0061] Figure 1a is a scene schematic diagram of a target detection method provided by an embodiment of the present application;

[0062] Figure 1b is a flow schematic diagram of a target detection method provided by an embodiment of the present application;

[0063] Figure 1c is a result schematic diagram of a target detection network used by a target detection method provided by an embodiment of the present application;

[0064] Figure 1d a module schematic diagram of a target teacher model and a target detection network calculating loss is shown;

[0065] Figure 2a is a schematic diagram of a road condition detection scene to which a target detection method provided by an embodiment of the present application is applied;

[0066] Figure 2b is a flow schematic diagram of a specific implementation of a target detection method provided by an embodiment of the present application;

[0067] Figure 3 is a structural schematic diagram of a target detection device provided by an embodiment of the present application;

[0068] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0069] Figure 5a is a schematic diagram of training numbers and test numbers of two training data sets;

[0070] Figure 5b is a schematic diagram of detection effect display;

[0071] Figure 5c is a schematic diagram of another detection effect display. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0073] The embodiments of the present application provide a target detection method, device, electronic device and storage medium.

[0074] The target detection apparatus can be integrated in an electronic device, which can be a terminal, a server, or the like. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer (PC), or the like. The server can be a single server or a server cluster composed of multiple servers.

[0075] In some embodiments, the target detection apparatus can be integrated in multiple electronic devices, for example, the target detection apparatus can be integrated in multiple servers to implement the target detection method of the present application.

[0076] In some embodiments, the server can also be implemented in the form of a terminal.

[0077] The electronic device can perform feature extraction on a to-be-detected picture to obtain multiple feature maps of different sizes corresponding to the to-be-detected picture, the feature maps including feature pixels. For each feature map of the multiple feature maps, multi-branch-based detection is performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the multiple branches are obtained by training respective training data sets, and the multiple training data sets each include different categories.

[0078] For example, referring to Figure 1a The electronic device can perform feature extraction on a to-be-detected picture to obtain x feature maps, the x feature maps being different in size from each other. For each feature map of the x feature maps, multi-branch-based detection is performed. For the convenience of description, feature map i is taken as an example: feature map i is detected by y branches to obtain y branch detection results. In the y branches, branch 1 can be used to detect a1 to aj, a total of j different categories; branch 2 can be used to detect b1 to bj, a total of j different categories. Any one of the j categories detected by branch 1 is different from any one of the j categories detected by the other branches. In this way, each branch detection result corresponding to feature map i can not have a repeated category.

[0079] Since each feature map obtains y branch detection results, x feature maps can obtain a total of x*y branch detection results. Processing all the x*y branch detection results can obtain a target detection result.

[0080] The following are described in detail respectively. It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0081] In this embodiment, a target detection method is provided, as shown in the figure, the specific process of the target detection method can be as follows: Figure 1b

[0082] 101. Feature extraction is performed on a to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, wherein the feature maps include feature pixels.

[0083] The to-be-detected picture is a picture to be subjected to target detection; the feature pixel is a pixel point in the feature map. The specific process of performing feature extraction on the to-be-detected picture and obtaining a plurality of feature maps can be referred to steps 1011-1013 as follows:

[0084] Step 1011. The to-be-detected picture is subjected to a first number of times of downsampling processing of different scales to obtain a first number of downsampling results, wherein the sizes of the first number of downsampling results are different.

[0085] The downsampling scales of the first number of times of downsampling processing are different. For details, please refer to Figure 1c , which is described by taking 3 as the first number for example:

[0086] The to-be-detected picture is subjected to 3 times of downsampling processing of different scales: the to-be-detected picture can be subjected to 8 times of downsampling to obtain a downsampling result C3; C3 is subjected to 2 times of downsampling to obtain a downsampling result C4; and C4 is subjected to 2 times of downsampling to obtain a downsampling result C5. The specific number of the first number should not be understood as a limitation of the present application, and the specific scale value of the downsampling scale should also not be understood as a limitation of the present application.

[0087] Step 1012. Convolution processing is performed on the first number of downsampling results to obtain a preliminary selected feature map corresponding to each downsampling result.

[0088] The convolution processing can be a convolution operation using a 1*1 convolution kernel. For details, please refer to Figure 1c , which is described by continuing the example: C3, C4, and C5 are subjected to convolution operation with a 1*1 convolution kernel to obtain preliminary selected feature maps P3, P4, and P5. The convolution processing can also be other convolution operation modes, such as a 2*2 convolution kernel, and the specific mode of the convolution processing should not be understood as a limitation of the present application.

[0089] ​In step 1013, the smallest-sized primary feature map is subjected to a second number of times of down-sampling processing to obtain a second number of secondary feature maps, wherein the first number of primary feature maps and the second number of secondary feature maps together constitute the plurality of feature maps of different sizes.

[0090] The down-sampling rates of the second number of times of down-sampling processing are also different from each other. It is assumed that the second number is 2, and the above example is continued to be described:

[0091] The smallest-sized primary feature map P5 is determined from the primary feature maps. P5 can be subjected to 2 times of down-sampling to obtain a down-sampling result, i.e., a secondary feature map P6; P6 can be subjected to 2 times of down-sampling again to obtain a down-sampling result, i.e., a secondary feature map P7. The primary feature maps P3, P4, and P5 and the secondary feature maps P6 and P7 together constitute the plurality of feature maps of different sizes described above. The specific value of the second number should not be understood as a limitation of the present application, and the specific value of the down-sampling rate should also not be understood as a limitation of the present application.

[0092] 102. For each feature map in the plurality of feature maps, a plurality of branch-based detections are performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are trained by a plurality of training data sets respectively, and the plurality of training data sets each include different categories.

[0093] Each feature map is subjected to a plurality of branch-based detections, and for each feature map, the operation process of the plurality of branch-based detections can be the same, so in the following, an example of performing a plurality of branch-based detections on any feature map is described.

[0094] The category refers to the classification to which a picture thing in an image belongs, and the category can include a person, a cat, a dog, a flower, a bicycle, a car, etc. The specific number of categories should not be understood as a limitation of the present application. For the plurality of branches, the operation process of each branch can be the same, but the category detected by each branch is different. Alternatively, each branch can be trained by a respective training data set, and the training data set includes a category set, the category set includes at least one category, and the at least one category is the category that the corresponding branch can detect. For any two branches in the plurality of branches, any category in the category set in the training data set corresponding to one branch is different from any category in the category set in the training data set corresponding to the other branch.

[0095] The data set comprises a class set, a picture set and a label set. The picture set refers to a set of pictures in the data set, and the label set refers to a set of description information of the pictures, which comprises position information of a label box in the picture and class information of the label box. For example, if a picture contains a person and a dog, the description information of the picture can comprise position information of a label box for the person, class information corresponding to the position information, i.e. person, position information of a label box for the dog, and class information corresponding to the position information, i.e. dog.

[0096] In an embodiment, step 102 can specifically comprise the following steps: performing convolution processing on the first feature map by using the first branch to obtain a probability value of M classes corresponding to each pixel point on the first feature map, a weight value of the each pixel point and a target center point of the each pixel point, and N bounding box distance values corresponding to each pixel point.

[0097] The first branch is any branch of the plurality of branches, the training data set corresponding to the first branch comprises the M classes, and the first feature map is any feature map of the plurality of feature maps. M and N are positive integers.

[0098] For example, the plurality of branches can be two branches: Head A and Head B, and the first feature map can be P4, and the first branch can be Head A in a multi-branch head (MTB-Head). Figure 1c

[0099] For P4, convolution operations are performed on two sub-branches HA1 and HA2 of the first branch Head A, respectively.

[0100] For the sub-branch HA1, after P4 is subjected to four convolution operations, an intermediate operation result corresponding to P4 can be obtained. The intermediate operation result is divided into two branches again, and one convolution operation is performed on each branch, so that a probability value Cls of M classes corresponding to each pixel point of P4 and a weight value Ctr of each pixel point of P4 and a target center point corresponding to the pixel point can be obtained. The M classes are classes in a class set in a training data set corresponding to the first branch Head A, i.e. the first branch Head A is trained to detect M classes in an image. The target center point is a center point of an outline of a picture object where the pixel point is located. The weight value of the pixel point and the target center point is negatively correlated with the distance between the pixel point and the target center point, i.e. the closer the distance between the pixel point and the target center point, the greater the weight value of the pixel point and the target center point; the farther the distance between the pixel point and the target center point, the smaller the weight value of the pixel point and the target center point.

[0101] ​For the sub-branch HA2, P4, after four convolution operations, the corresponding intermediate operation result can be obtained. After one more convolution operation on the intermediate operation result, the N bounding box distance values Reg corresponding to each pixel point of P4 can be obtained. Optionally, N can be 4, and the 4 bounding box distance values are l, t, r, and b, where l represents the distance value of the pixel point and the left bounding box of the detection box, t represents the distance value of the pixel point and the top bounding box of the detection box, r represents the distance value of the pixel point and the right bounding box of the detection box, and b represents the distance value of the pixel point and the bottom bounding box of the detection box.

[0102] 103. Obtain a target detection result based on the branch detection result of the feature pixels, and display the target detection result.

[0103] As described above, the branch detection result of the feature pixels is Cls, Ctr, and Reg. To obtain the target detection result from the branch detection result, a post-processing process is also needed.

[0104] Optionally, in a specific embodiment, the post-processing process for obtaining the target detection result from the branch detection result includes the following steps 1031 to 1034:

[0105] 1031. Determine the maximum probability value from the M probability values corresponding to each pixel point, and take the maximum probability value as the binding probability value of the corresponding pixel point. The category corresponding to the binding probability value is the binding category of the pixel point.

[0106] Each pixel point has M probability values corresponding to M categories, and the maximum probability value is selected from the M probability values as the binding probability value of the corresponding pixel point. For example, let's assume that the M categories are cat, dog, person, flower, car, and bicycle, then for each pixel point, the probability values corresponding to the 6 categories cat, dog, person, flower, car, and bicycle are calculated, i.e., 6 probability values: Pr 猫 , Pr 狗 , Pr 人 , Pr 花 , Pr 汽车 , Pr 自行车 . The maximum probability value is selected from the 6 probability values, and let's assume that Pr 汽车 is the maximum probability value of a certain pixel point, then the binding probability value of the pixel point is Pr 汽车 , and the binding category of the pixel point is car.

[0107] Through the above operation process, the binding probability value and the binding category of each pixel point can be obtained.

[0108] 1032. For each pixel on the first feature map, sort them according to the value of the binding probability value.

[0109] 1033. From the pixels on the first feature map, determine the P pixels with the highest binding probability values, where P is a positive integer.

[0110] After determining the binding probability values ​​of all pixels in the first feature map, the binding probability values ​​can be sorted numerically; and based on the sorting, the P pixels with the highest binding probability values ​​can be determined.

[0111] Optionally, the sorting can be in descending order, in which case the pixels corresponding to the first P binding probability values ​​of the sorted binding probability value queue can be retrieved; the sorting can also be in ascending order, in which case the pixels corresponding to the last P binding probability values ​​of the sorted binding probability value queue can be retrieved.

[0112] 1034. For each of the P pixels, based on the N bounding box distance values ​​corresponding to the pixel, determine the P detection boxes corresponding to the P pixels respectively, wherein the binding categories corresponding to the P detection boxes and the P pixels are the target detection results of the first feature map in the first branch.

[0113] After obtaining P pixels, P detection boxes corresponding to each of the P pixels can be determined based on the N bounding box distance values ​​(i.e., Reg) corresponding to each of the P pixels. The detection boxes corresponding to each of the P pixels and the binding categories corresponding to each of the P pixels can be used as the object detection results of the first branch of the first feature map.

[0114] Similar to steps 1031 to 1034, the following steps can be performed: Figure 1c In the five feature maps shown, object detection results are calculated for each of the two branches, resulting in a total of 10 (5*2) object detection results. These 10 results are then processed to obtain the object detection result for the image under test. Optionally, the processing of the 10 object detection results can involve assigning weight values ​​to these 10 results, and the object detection result for the image under test is the weighted sum of the aforementioned 10 object detection results. The specific process of obtaining the object detection result for the image under test based on the branch object detection results should not be construed as a limitation of this application.

[0115] In the target detection method provided in the embodiments of the present application, feature extraction is first performed on a to-be-detected picture to obtain a plurality of feature maps corresponding to the to-be-detected picture, and the sizes of the plurality of feature maps are different from each other; then, for each feature map, a corresponding branch detection result is obtained based on each branch of a plurality of branches, and a plurality of branch detection results are obtained in total. Since different branches are trained by using respective corresponding training data sets, the plurality of training data sets each include different categories, and thus the categories corresponding to the targets presented in each branch detection result are different from each other. The plurality of branch detection results are summarized to obtain a detection result corresponding to a larger category set composed of a plurality of category sets. The embodiments of the present application implement target detection of different category sets by using a plurality of branches respectively, and since the plurality of branches of one model are utilized, the number of models can be reduced while the accuracy of category detection is improved.

[0116] Optionally, before step 101, the target detection method provided in the embodiments of the present application can further include the following step: obtaining a plurality of training data sets, wherein the plurality of training data sets correspond to the plurality of branches one by one, and the categories included in the plurality of training data sets are different from each other; training the branch corresponding to the training data set by using the plurality of training data sets until the entire target detection network meets a preset condition, to obtain a trained target detection network.

[0117] Each branch has a respective corresponding training data set, and the categories included in each training data set are different from each other. When the training data set is used to train the corresponding branch, the category included in the corresponding training data set can be detected more accurately by each branch. The preset condition can be that the entire target detection network converges or the total loss value of the entire target detection network is minimum. The specific condition content of the preset condition should not be understood as a limitation on the present application.

[0118] The step of "training the branch corresponding to the training data set by using the plurality of training data sets until the entire target detection network meets a preset condition, to obtain a trained target detection network" includes the following steps a1 to a6:

[0119] a1, performing feature extraction on a training picture from the training data set corresponding to the branch to obtain a plurality of training feature maps of different sizes corresponding to the training picture.

[0120] a2, detecting each training feature map in the plurality of training feature maps by using the branch to obtain a training detection result.

[0121] During training, the training picture in the training data set can be sent only to the branch corresponding to the training data set, so as to improve the training efficiency.

[0122] Optionally, in one specific embodiment, determining the branch corresponding to the training picture can be performed in the following manner: the training picture can carry an identity from a training data set, and each branch also carries an identity of the corresponding training data set. The training picture can be delivered to the corresponding branch by comparing the identity of the training picture with the identity of the branch.

[0123] Optionally, in another specific embodiment, determining the branch corresponding to the training picture can be performed in the following manner: the training picture itself carries a category to which the picture object belongs. The category in the training picture can be matched with a category set of the training data set corresponding to the branch, so as to determine the branch corresponding to the training picture. The specific manner of determining the branch corresponding to the training picture should not be understood as a limitation of the present application.

[0124] Step a2 specifically comprises: performing convolution processing on each training feature map by the branch to obtain a probability value of M' categories corresponding to each pixel point on the training feature map, a weight value of the each pixel point and a target center point of the each pixel point, and N' bounding box distance values corresponding to the each pixel point.

[0125] The training data set from which the training picture comes includes the M' categories, and M' and N' are both positive integers. The value of N' can be the same as or different from N.

[0126] a3, determining a branch loss value corresponding to the branch according to the training detection result, a prediction result of the training picture by a target teacher model, and a labeled box in the training picture, wherein the target teacher model is a teacher model corresponding to the branch, and the target teacher model is a model trained on the training data set.

[0127] The target teacher model is a model with excellent detection effect on the training data set corresponding to the branch, and the labeled box is a bounding box in the training picture that frames a target object in a picture object. Steps a1 to a3 are steps performed for each branch, so that a branch loss value of each branch can be obtained.

[0128] According to the training detection result, the prediction result of the training picture by the target teacher model, and the labeled box in the training picture, the branch loss value corresponding to the branch can be determined in the following manner:

[0129] The branch loss value L k includes a feature loss L ft , a distillation loss , and a detection loss L gt .

[0130] The branch loss value L k can be calculated according to the formula The branch loss value L is calculated k . Wherein, ω GT , ω kd , ω ft are hyperparameters; the feature loss L ft , the distillation loss and the detection loss L gt are calculated as described below. The feature loss L ft reflects the loss value of the target teacher model supervising the features learned by the backbone network of the target detection network, and the distillation loss reflects the loss value of the target teacher model supervising the prediction results of the corresponding branch. When calculating the branch loss value, the feature loss L ft and the distillation loss are calculated on the basis of the detection loss L gt , which can make the detection accuracy of the trained target detection network higher.

[0131] Please refer to Figure 1d , the calculation process of the feature loss L ft is as follows:

[0132] The teacher output features of the target teacher model on the plurality of feature maps of different sizes are obtained; the branch output features corresponding to the plurality of feature maps of different sizes are obtained; the mean square error of the teacher output features and the branch output features is calculated, wherein the mean square error is the feature loss.

[0133] The feature loss L ft is calculated according to the formula L ft =||F t -H(F tar )||. Wherein, F t is the teacher output feature of the target teacher model on the plurality of feature maps of different sizes. Since the plurality of feature maps of different sizes are obtained by the backbone network, F t is also the output feature of the backbone network of the target teacher model. F tar is the output feature of the training picture by the backbone network of the target detection network. H() is a feature converter, which is used to convert F tar , and H() can be a 1*1 convolution kernel. ||F t -H(F tar )|| represents the mean square error of F t and H(F tar ).

[0134] Please refer to Figure 1d , the calculation process of the distillation loss is as follows:

[0135] The result of the class prediction of the target teacher model is determined, a first quotient of the result of the class prediction of the target teacher model and a hyperparameter temperature is obtained, and a first function value is obtained by operating the first quotient through a preset function; the class detection result obtained by the branch is determined, a second quotient of the class detection result and the hyperparameter temperature is obtained, and a second function value is obtained by operating the second quotient through the preset function; and a KL divergence of the first function value and the second function value is calculated, and the KL divergence is the distillation loss.

[0136] According to the formula The distillation loss is calculated wherein, is a class detection result of the branch of the target detection network for Cls, is a class detection result of the target teacher model for Cls, T is a hyperparameter temperature, σ is a softmax function, and KL-Div is a KL divergence.

[0137] See Figure 1d , the detection loss L gt is calculated as follows:

[0138] The detection loss is calculated according to the probability value of M' classes corresponding to each pixel point on each training feature map, the weight value of the center degree of the each pixel point with itself, the N' bounding box distance value corresponding to the each pixel point, and the labeled box in the training picture.

[0139] According to the formula:

[0140]

[0141] The detection loss L gt is calculated. Wherein, i, j represent the position index of the output in the length and width directions, represents the class output of the target detection network, represents the Reg output of the target detection network, represents the center degree output of the target detection network; represents a class label calculated by a first-order fully convolutional one-stage object detection (FCOS) according to a labeled box of a training sample, represents a Reg label calculated by the FCOS according to the labeled box of the training sample, represents a center degree label calculated by the FCOS according to the labeled box of the training sample; FocalLoss is a focal loss function, IoULoss is an intersection over union loss function, and BCELoss is a binary cross entropy loss function; N posN is the number of positive samples; I is an indicator function, which takes 1 when the indicated value is positive, and 0 when the indicated value is negative or 0.

[0142] a4, determining a branch loss value corresponding to each branch of the plurality of branches, calculating a weighted sum of the branch loss value corresponding to each branch to obtain a total loss value.

[0143] According to the formula The total loss value is calculated, wherein the weight ω of the branch loss value k is a hyperparameter.

[0144] a5, if the total loss value meets the preset condition, it is determined that the plurality of branches is trained.

[0145] a6, if the total loss value does not meet the preset condition, the parameters of each branch are adjusted according to the total loss value, and iteration is performed until the total loss value meets the preset condition.

[0146] The preset condition can be that the total loss value converges or is minimum. The way of adjusting the parameters of each branch according to the total loss value can be as follows: the total loss value is back propagated, and the parameters of each branch are adjusted.

[0147] Obtaining a training data set with the same number of branches as the plurality of branches, and training the corresponding branch using the corresponding training data set can significantly improve the detection and recognition ability of the branch to the picture things of multiple categories in the category set in the training data set. Different branches in the plurality of branches focus on the detection of different category sets, so as to improve the detection and recognition ability of the entire target detection network to multiple categories, and since it is realized through multiple branches of one target detection network, compared with the prior art of using multiple target detection models for multiple category detection, the parameter amount of the model is reduced.

[0148] Optionally, before the step of "obtaining a plurality of training data sets", the target detection method provided by the embodiment of the application can further include: dividing a preset data set into the first training data set and the second training data set.

[0149] The preset data set can be a data set that is easy to obtain and used for training a neural network to improve the recognition accuracy of the neural network. The preset data set can be a Common Objects in Context data set (COCO data set for short) or a Visual Object Classes data set (VOC data set for short).

[0150] The embodiment of the application can divide the COCO dataset into subsets COCO-A and COCO-B, and can also divide the VOC dataset into subsets VOC-A and VOC-B.

[0151] In a specific implementation, the step of "dividing the preset dataset into the first training dataset and the second training dataset" can specifically include steps b1 to b5 as follows:

[0152] b1. Dividing all Q categories in the preset dataset into the first training dataset and the second training dataset, wherein Q is a positive integer.

[0153] Dividing all Q categories into the first training dataset and the second training dataset can make the categories divided into the first training dataset and the second training dataset have no intersection, and the union of the categories divided into the first training dataset and the second training dataset is all Q categories.

[0154] Optionally, in a specific implementation, step b1 includes steps s1 to s6 as follows:

[0155] s1. Obtaining a current category divided into the first training dataset or the second training dataset.

[0156] The current category refers to a category just completed division, which can be divided into the first training dataset or the second training dataset.

[0157] s2. Obtaining, in all pictures in the preset dataset, the co-occurrence times of each of the unassigned categories of the preset dataset and the current category appearing in the same picture, and determining the category with the maximum co-occurrence times, denoted as a to-be-classified category.

[0158] After obtaining the current category, the co-occurrence times of each of the unassigned categories of the preset dataset and the current category appearing in the same picture are obtained for all pictures in the preset dataset.

[0159] Suppose that z categories have been assigned, and Q-z categories in the preset dataset have not been assigned. For each of the Q-z categories, the co-occurrence times of the category and the current category appearing in the same picture in all pictures in the preset dataset can be calculated, and Q-z co-occurrence times are obtained.

[0160] The category corresponding to the maximum co-occurrence times in the Q-z co-occurrence times is obtained, and the category is denoted as a to-be-classified category.

[0161] The above process can be implemented by the formula . Wherein, CA is a first training data set, C B is a second training data set, M i,c is the Q-z common occurrence times described above, is the category corresponding to the common occurrence times with the largest value in the Q-z common occurrence times.

[0162] s3, in all pictures in the preset data set, determine the first common occurrence times of the same picture of the occurrence of each category in the first training data set and the category to be classified, and obtain the sum of a plurality of the first common occurrence times, to obtain a first sum.

[0163] Suppose that all the categories in the first training data set that have been divided are s1, then the first common occurrence times of the same picture of each category in the s1 categories and the category to be classified in all pictures in the preset data set need to be determined, and there are s1 first common occurrence times; then the s1 first common occurrence times are summed to obtain a first sum.

[0164] The first sum can be represented as , wherein M i,c represents the number of common occurrences of category i and the category to be classified C, and category i is a category that has been divided into the first training data set C A .

[0165] s4, in all pictures in the preset data set, determine the second common occurrence times of the same picture of the occurrence of each category in the second training data set and the category to be classified, and obtain the sum of a plurality of the second common occurrence times, to obtain a second sum.

[0166] Suppose that all the categories in the second training data set that have been divided are s2, then the second common occurrence times of the same picture of each category in the s2 categories and the category to be classified in all pictures in the preset data set need to be determined, and there are s2 second common occurrence times; then the s2 second common occurrence times are summed to obtain a second sum.

[0167] The second sum can be represented as , wherein M i,c represents the number of common occurrences of category i and the category to be classified C, and category i is a category that has been divided into the second training data set C B .

[0168] s5, if the first sum is greater than or equal to the second sum, the category to be classified is divided into the second training data set as a new current category, and jumps to step s1 until the division of all Q categories in the preset data set is completed.

[0169] s6, if the first sum is less than the second sum, then the class to be classified is divided into the first training data set as a new current class, and jump to step s1, until the division of all Q classes in the preset data set is completed.

[0170] If then C B =C B ∪{c};

[0171] If then C A =C A ∪{c}.

[0172] By repeatedly performing steps s1 to s6, the division of all Q classes in the preset data set can be completed.

[0173] Before performing the loop of steps s1 to s6, the first class to be divided needs to be calculated. The calculation method of the first class to be divided is as follows: for each of the Q classes in the preset data set: in all pictures in the preset data set, determine the number of times each of the other Q-1 classes appears together with the class in the same picture, obtaining Q-1 common occurrence times; calculate the cumulative value of the Q-1 common occurrence times, which corresponds to the class; obtain Q cumulative values corresponding to the Q classes respectively; the class corresponding to the largest cumulative value in the Q cumulative values is put into the first training data set.

[0174] For example, the coco data set is taken as an example for description, the coco data set includes 80 classes, and the following steps are performed for each of the 80 classes:

[0175] Taking the "person" class in the 80 classes as an example, the common occurrence times of the "person" class appearing together with other 79 classes in the same picture are determined, and the other 79 classes include: bicycle, car, motorcycle, airplane, etc. That is, the common occurrence times of the person appearing together with the bicycle in the same picture, the common occurrence times of the person appearing together with the car in the same picture, the common occurrence times of the person appearing together with the motorcycle in the same picture, the common occurrence times of the person appearing together with the airplane in the same picture, etc. There are 79 common occurrence times, and the cumulative value is obtained by accumulating the above 79 common occurrence times, which corresponds to the "person" class.

[0176] Through similar operation, the accumulated value corresponding to each of the 80 categories is calculated respectively, and then the category corresponding to the largest accumulated value is selected, which is recorded as the first divided category, and the category is divided into the first training data set.

[0177] In addition to the above-mentioned category division method, the embodiments of the present application can also divide all Q categories in the preset data set into two groups in a random manner, thereby dividing all Q categories. The specific manner of dividing the categories of the preset data set should not be understood as a limitation of the present application.

[0178] b2, divide all pictures in the preset data set into pictures whose categories are only contained in the first training data set, pictures whose categories are only contained in the second training data set, and pictures whose categories are contained in both the first training data set and the second training data set.

[0179] In addition to the pictures, the preset data set also includes annotation boxes for training the model and annotation description information of the categories corresponding to the annotation boxes.

[0180] After the categories are divided into the first training data set and the second training data set in step b1, the pictures are further divided according to the categories, which can be divided according to the case that the categories reflected by the annotation description information in each picture fall into the first training data set and the second training data set.

[0181] The pictures can be divided into: the categories reflected by all annotation description information of the pictures fall completely within the scope of the first training data set, the categories reflected by all annotation description information of the pictures fall completely within the scope of the second training data set, and the categories reflected by all annotation description information of the pictures cross the two training data sets.

[0182] Continue to illustrate the above example: the 80 categories included in the coco data set can be divided into the first training data set and the second training data set, wherein the first training data set is divided into 40 categories, and the second training data set is divided into another 40 categories.

[0183] Let's assume the first training dataset includes 40 categories such as bicycles, cars, motorcycles, and trains, and the second training dataset includes 40 categories such as cats, dogs, cows, and sheep. If all the annotations and descriptions of an image indicate that the category is either a car or a train, then the category represented by that image is completely covered by the first training dataset. If all the annotations and descriptions of an image indicate that the category is either a dog or a cow, then the category represented by that image is completely covered by the second training dataset. If all the annotations and descriptions of an image indicate that the category is either a cow, a sheep, or a train, then the category represented by that image spans both the first and second training datasets.

[0184] b3. Distribute the images whose categories are included in both the first training dataset and the second training dataset equally to the first training dataset and the second training dataset.

[0185] Images whose category is only included in the first training dataset are assigned to the first training dataset, images whose category is only included in the second training dataset are assigned to the second training dataset, and images whose category spans both training datasets can be evenly distributed between the two training datasets.

[0186] b4. For all images in the first training dataset, delete any information that relates to the categories in the second training dataset.

[0187] b5. For all images in the second training dataset, delete all information that is related to the category of the first training dataset.

[0188] After the images are partitioned, the images in the first training dataset are only used to train the detection accuracy for the categories included in the first training dataset. Therefore, for the annotation descriptions of the images in the first training dataset, only the annotation descriptions related to the categories included in the first training dataset are retained, and all information related to the categories of the second training dataset is deleted. Similarly, for the second training dataset, all information related to the categories of the first training dataset is deleted. After deleting the category-related information, the first and second training datasets are obtained after the partitioning is complete.

[0189] By removing annotation descriptions, we can avoid interference from annotation descriptions that do not belong to a certain branch's category recognition. This allows the split training dataset to help the corresponding branch to train more focused on its category recognition, thereby improving the branch's detection accuracy for the categories included in the training dataset.

[0190] The method provided in the embodiment of the present application can first train the target detection model by using the divided training data set, and after the training is completed, the trained target detection model is used to perform feature extraction on the to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, and the feature maps include feature pixels; for each feature map in the plurality of feature maps, a detection based on a plurality of branches is performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of training data sets corresponding to the plurality of branches respectively, and the plurality of training data sets each include different categories; a target detection result is obtained based on the branch detection result of the feature pixel, and the target detection result is displayed. And in the process of training the target detection model, the loss values respectively obtained by the target teacher model supervising the backbone network and the branch network of the target detection network are added: feature loss L ft and distillation loss The introduction of the two loss values makes the detection accuracy of the trained target detection model higher.

[0191] In the embodiment of the present application, since different branches are trained based on the training data sets corresponding to the branches respectively, and the category sets included in each training data set are different, the categories corresponding to the targets presented in each branch detection result are different. By summarizing the plurality of branch detection results, a detection result corresponding to a larger category set composed of a plurality of category sets can be obtained. The embodiment of the present application realizes target detection of different category sets by a plurality of branches respectively, and since the plurality of branches of one model are used, the number of models is reduced while the accuracy of multi-category detection is improved.

[0192] The target detection method provided in the embodiment of the present application can be applied to a scene in which multiple categories of picture objects in an image need to be detected, such as road condition detection, video monitoring, map AI, and the like. Details can be referred to in Figure 2a For the road condition detection scene, it is necessary to detect not only Figure 2a the driving indication information of the road surface in the left image, such as the indication arrow, but also Figure 2a the road sign information beside the road in the right image. The method described in the above embodiment will be further described in detail below.

[0193] In the embodiment, the method of the present application will be described in detail by taking two branches as an example.

[0194] As shown in Figure 2b , a target detection method specifically includes the following steps:

[0195] 201, performing downsampling processing of a first number of different magnifications on a to-be-detected picture to obtain a first number of downsampling results.

[0196] 202. Perform convolution processing on the first number of downsampling results to obtain the first number of initial selected feature maps.

[0197] 203. Perform a second number of downsampling operations on the initial selected feature map with the smallest size to obtain the second number of secondary selected feature maps, wherein the first number of initial selected feature maps and the second number of secondary selected feature maps are used together as feature maps of different sizes.

[0198] 204. For each feature map, based on the first branch detection, obtain the probability values ​​of M1 categories corresponding to each feature pixel in the feature map, the weight value of each feature pixel and its own target center point, and the N1 bounding box distance values ​​corresponding to each feature pixel, wherein the first branch is trained by the first training dataset, and the category set of the first training dataset includes M1 categories.

[0199] 205. Obtain the largest probability value among the M1 probability values ​​corresponding to the loudness of each feature. This probability value is the binding probability value of the corresponding feature pixel, and the category corresponding to the binding probability value is the binding category of the feature pixel.

[0200] 206. For each feature map, sort the binding probability values ​​of a feature map in descending order and obtain the feature pixels corresponding to the first P binding probability values. Based on the N1 bounding box distance values ​​corresponding to each feature pixel, determine the P detection boxes corresponding to the P feature pixels respectively. The binding categories corresponding to the P detection boxes and the P pixels are respectively the target detection results of each feature map in the first branch.

[0201] 207. For each feature map, based on the second branch detection, obtain the probability values ​​of M2 categories corresponding to each feature pixel in the feature map, the weight value of each feature pixel and its own target center point, and the N2 bounding box distance values ​​corresponding to each feature pixel. The second branch is trained by the second training dataset, and the category set of the second training dataset includes M2 categories.

[0202] 208. Obtain the largest probability value among the M2 probability values ​​corresponding to the loudness of each feature. This probability value is the binding probability value of the corresponding feature pixel, and the category corresponding to the binding probability value is the binding category of the feature pixel.

[0203] 209. For each feature map, sort the binding probability values ​​of a feature map in descending order and obtain the feature pixels corresponding to the first P binding probability values. Based on the N2 bounding box distance values ​​corresponding to each feature pixel, determine the P detection boxes corresponding to the P feature pixels respectively. The binding categories corresponding to the P detection boxes and the P pixels are respectively the target detection results of each feature map in the second branch.

[0204] 210、According to the target detection result of each feature map in the first branch and the target detection result of each feature map in the second branch, the target detection result of the picture to be detected is obtained, and the target detection result is displayed.

[0205] As can be seen from the above, the method provided in the embodiments of the present application can train the first branch and the second branch of the target detection model respectively by using the first training data set and the second training data set. After the training is completed, the target detection model is used to extract features of a picture to be detected, to obtain a plurality of feature maps of different sizes corresponding to the picture to be detected, and the feature maps include feature pixels. For each feature map in the plurality of feature maps, detection is performed based on the first branch and the second branch respectively, to obtain the branch detection result of each feature pixel corresponding to the first branch and the branch detection result of each feature pixel corresponding to the second branch, wherein the categories detected by the first branch and the second branch are different from each other. The target detection result is obtained based on the two branch detection results, and the target detection result is displayed.

[0206] In the embodiments of the present application, the first branch is trained based on the first training data set, and the second branch is trained based on the second training data set, and the category sets included in the first training data set and the second training data set are different from each other. Therefore, the targets corresponding to the categories presented in the first branch detection result and the second branch detection result are different from each other. By summarizing the first branch detection result and the second branch detection result, the detection result corresponding to a larger category set composed of a plurality of category sets can be obtained. The embodiments of the present application realize target detection of different category sets by two branches respectively. Since two branches of one model are used, the number of models is reduced, and the accuracy of multiple category detection is improved.

[0207] In order to better implement the above method, the embodiments of the present application also provide a target detection device. The target detection device can be integrated in an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers.

[0208] For example, in the present embodiment, the target detection device is integrated in an electronic device, and the method of the embodiments of the present application is described in detail.

[0209] For example, as shown in FIG. 1, the target detection device can include: Figure 3

[0210] The feature map acquisition unit 301 is configured to extract features of a picture to be detected, and obtain a plurality of feature maps of different sizes corresponding to the picture to be detected, wherein the feature maps include feature pixels.​

[0211] The branch result acquisition unit 302 is configured to, for each of the feature maps, perform branch-based detection to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the branches are obtained by training respective training data sets, and the training data sets each include different categories.

[0212] The target result acquisition unit 303 is configured to obtain a target detection result based on the branch detection result of the feature pixel and display the target detection result.

[0213] In some embodiments, the branch result acquisition unit 302 is specifically configured to perform convolution processing on the first feature map by using a first branch to obtain probability values of M categories corresponding to each pixel point on the first feature map, a weight value of each pixel point and a target center point of the pixel point, and N bounding box distance values corresponding to each pixel point, wherein the first branch is any branch of the plurality of branches, the training data set corresponding to the first branch includes the M categories, the first feature map is any feature map of the plurality of feature maps, and M and N are positive integers.

[0214] In some embodiments, the target result acquisition unit 303 includes:

[0215] The probability value determination sub-unit is configured to determine a probability value with the largest value among the probability values of the M categories corresponding to each pixel point, and take the probability value with the largest value as a binding probability value of the corresponding pixel point, wherein the category corresponding to the binding probability value is a binding category of the pixel point.

[0216] The probability value sorting sub-unit is configured to sort each pixel point on the first feature map according to the value of the binding probability value.

[0217] The pixel point determination sub-unit is configured to determine P pixel points with the largest binding probability value from the pixel points on the first feature map, wherein P is a positive integer.

[0218] The detection frame determination sub-module is configured to determine P detection frames corresponding to the P pixel points respectively according to the N bounding box distance values corresponding to each pixel point, wherein the P detection frames and the binding categories corresponding to the P pixel points respectively are the target detection result of the first feature map in the first branch.

[0219] In some embodiments, the feature map acquisition unit 301 includes:

[0220] The downsampling processing subunit is configured to perform downsampling processing of different scales on the to-be-detected picture for a first number of times to obtain a first number of downsampling results, wherein the sizes of the first number of downsampling results are different from each other;

[0221] The primary selection subunit is configured to perform convolution processing on the first number of downsampling results to obtain a primary selection feature map corresponding to each downsampling result.

[0222] The secondary selection subunit is configured to perform downsampling processing on the primary selection feature map with the smallest size for a second number of times to obtain a second number of secondary selection feature maps, wherein the first number of primary selection feature maps and the second number of secondary selection feature maps together form the plurality of feature maps with different sizes.

[0223] In some embodiments, the apparatus further comprises:

[0224] The training data set acquisition unit is configured to acquire a plurality of training data sets, wherein the plurality of training data sets correspond to the plurality of branches one by one, and the plurality of training data sets each include different categories from each other;

[0225] The network training unit is configured to train the branches corresponding to the training data sets by using the plurality of training data sets until the entire target detection network meets a preset condition to obtain a trained target detection network.

[0226] In some embodiments, the network training unit comprises:

[0227] The training feature map subunit is configured to, for each branch, perform feature extraction on a training picture from the training data set corresponding to the branch to acquire a plurality of training feature maps with different sizes corresponding to the training picture.

[0228] The training detection result subunit is configured to detect each training feature map in the plurality of training feature maps by using the branch to acquire a training detection result.

[0229] The branch loss value subunit is configured to determine a branch loss value corresponding to the branch according to the training detection result, a prediction result of a target teacher model on the training picture, and a labeled box in the training picture, wherein the target teacher model is a teacher model corresponding to the branch, and the target teacher model is a model that has completed training on the training data set.

[0230] The total loss value subunit is configured to determine the branch loss value corresponding to each branch in the plurality of branches, calculate a weighted sum of the branch loss values corresponding to each branch, and obtain a total loss value.

[0231] A determination first sub-unit is configured to determine that the plurality of branches are trained when the total loss value meets the preset condition.

[0232] A determination second sub-unit is configured to adjust the parameters of each branch according to the total loss value when the total loss value does not meet the preset condition, and iterates until the total loss value meets the preset condition.

[0233] In some embodiments, the training detection result sub-unit is specifically configured to perform convolution processing on each training feature map by the branch to obtain a probability value of M' categories corresponding to each pixel point on the training feature map, a weight value of the each pixel point and a target center point of the each pixel point, and N' bounding box distance values corresponding to the each pixel point; wherein the training data set from which the training picture comes includes the M' categories, and M' and N' are positive integers.

[0234] In some embodiments, the device further comprises:

[0235] A data set division unit is configured to divide a preset data set into the first training data set and the second training data set.

[0236] In some embodiments, the data set division unit comprises:

[0237] A category division sub-unit is configured to divide all Q categories in the preset data set into the first training data set and the second training data set, wherein Q is a positive integer.

[0238] A picture division sub-unit is configured to divide all pictures in the preset data set into pictures in which categories are only contained in the first training data set, pictures in which categories are only contained in the second training data set, and pictures in which categories are contained in both the first training data set and the second training data set.

[0239] A picture equal division sub-unit is configured to equally divide the pictures in which categories are contained in both the first training data set and the second training data set into the first training data set and the second training data set.

[0240] In some embodiments, the device further comprises:

[0241] A first category deletion unit is configured to delete information related to categories of the second training data set from all pictures in the first training data set.

[0242] A second category deletion unit is configured to delete information related to categories of the first training data set from all pictures in the second training data set.

[0243] In some embodiments, the category dividing unit comprises:

[0244] a current category sub-unit configured to obtain a current category divided into the first training data set or the second training data set;

[0245] a to-be-classified category sub-unit configured to obtain, in all pictures in the preset data set, a co-occurrence frequency of each of the categories in the preset data set which are not assigned and the current category appearing in a same picture, and determine a category with a maximum co-occurrence frequency as a to-be-classified category;

[0246] a first sum sub-unit configured to determine, in all pictures in the preset data set, a first co-occurrence frequency of the to-be-classified category and each of the categories in the first training data set appearing in a same picture, and obtain a sum of a plurality of the first co-occurrence frequencies to obtain a first sum;

[0247] a second sum sub-unit configured to determine, in all pictures in the preset data set, a second co-occurrence frequency of the to-be-classified category and each of the categories in the second training data set appearing in a same picture, and obtain a sum of a plurality of the second co-occurrence frequencies to obtain a second sum;

[0248] a first division sub-unit configured to divide the to-be-classified category into the second training data set as a new current category when the first sum is greater than or equal to the second sum;

[0249] a second division sub-unit configured to divide the to-be-classified category into the first training data set as a new current category when the first sum is less than the second sum;

[0250] a step jump sub-unit configured to jump to the step of obtaining the current category divided into the first training data set or the second training data set until the division of all Q categories in the preset data set is completed.

[0251] In some embodiments, the apparatus further comprises:

[0252] a co-occurrence frequency calculation unit configured to, for each of the Q categories in the preset data set, determine, in all pictures in the preset data set, a co-occurrence frequency of each of the other Q-1 categories appearing in a same picture to obtain Q-1 co-occurrence frequencies;

[0253] an accumulated value calculation unit configured to calculate an accumulated value of the Q-1 co-occurrence frequencies, the accumulated value corresponding to the category;

[0254] The cumulative value acquisition unit is used to obtain the Q cumulative values ​​corresponding to the Q categories respectively;

[0255] The category storage unit is used to put the category corresponding to the largest accumulated value among the Q accumulated values ​​into the first training dataset.

[0256] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0257] As can be seen from the above, the object detection device in this embodiment can first train the object detection model using the divided training dataset. After training, the trained object detection model is used to extract features from the image to be detected, obtaining multiple feature maps of different sizes corresponding to the image to be detected. The feature maps include feature pixels. For each feature map in the multiple feature maps, detection based on multiple branches is performed to obtain the branch detection result for each feature pixel in the feature map corresponding to each branch. The multiple branches are trained using their respective training datasets, and the categories included in the multiple training datasets are different from each other. The object detection result is obtained based on the branch detection result of the feature pixels, and the object detection result is displayed. Therefore, this embodiment can improve the accuracy of multi-category detection while reducing the number of models.

[0258] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0259] In some embodiments, the target detection device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the target detection method of this application.

[0260] In this embodiment, the electronic device will be used as an example for detailed description, such as... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0261] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4The electronic device structure shown in the figures is not intended to limit the electronic device, which can include more or fewer components than shown, or have components combined together or arranged differently. Among others:

[0262] The processor 401 is the control center of the electronic device, which connects various parts of the electronic device through various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the entire electronic device. In some embodiments, the processor 401 can include one or more processing cores; in some embodiments, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0263] The memory 402 can be used to store software programs and modules, and the processor 401 executes various functions and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access for the processor 401 to the memory 402.

[0264] The electronic device also includes a power supply 403 for supplying power to various components, and in some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include one or more direct current or alternating current power supplies, a recharging system, a power supply failure detection circuit, a power supply converter or inverter, a power supply state indicator, and any other components.

[0265] The electronic device can also include an input module 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0266] The electronic device can also include a communication module 405, which in some embodiments can include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing the user with wireless broadband Internet access. For example, the communication module 405 can be used to help the user send and receive emails, browse web pages, and access streaming media, etc.

[0267] Although not shown, the electronic device can also include a display unit, etc., which will not be described here. In particular, in the present embodiment, the processor 401 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and run the application program stored in the memory 402 by the processor 401, thereby realizing various functions, such as:

[0268] Feature extraction is performed on the to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, the feature maps including feature pixels; for each feature map in the plurality of feature maps, a detection based on a plurality of branches is performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of training data sets corresponding to the branches respectively, and the plurality of training data sets each include different categories; a target detection result is obtained based on the branch detection result of the feature pixel, and the target detection result is displayed.

[0269] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described here.

[0270] As can be seen from the above, in the embodiments of the present application, different branches are trained based on the training data sets corresponding to the branches respectively, and each training data set includes different category sets, so that the targets corresponding to the categories presented in each branch detection result are different. By summarizing the plurality of branch detection results, a detection result corresponding to a larger category set composed of a plurality of category sets can be obtained. The embodiments of the present application realize target detection of different category sets through a plurality of branches respectively, and since a plurality of branches of one model are utilized, the number of models is reduced while the accuracy of category detection is improved.

[0271] Please refer to Figure 5a In Figure 5a , the training number and the test number of two training data sets in an embodiment are introduced.

[0272] For training data sets COCO_A and COCO_B:

[0273] For COCO_A, 57357 images, 245039 descriptions and 32 categories were trained, and 4158 images, 18539 descriptions and 32 categories were tested.

[0274] For COCO_B, 59909 images, 243320 descriptions and 48 categories were trained, and 4225 images, 18242 descriptions and 48 categories were tested.

[0275] For VOC_A and VOC_B training data sets:

[0276] For VOC_A, 8739 images, 19096 descriptions and 17 categories were trained, and 4087 images, 8005 descriptions and 17 categories were tested.

[0277] For VOC_B, 7812 images, 19096 descriptions and 3 categories were trained, and 2687 images, 6971 descriptions and 3 categories were tested.

[0278] After training and testing the above training data sets in the above quantities, the comparison results are shown in Figure 5b and Figure 5c .

[0279] Please refer to Figure 5b , wherein Single represents Prior Art 1: a single category training data set is constructed for different categories respectively, and an image recognition model is trained using the training data set to obtain a plurality of image recognition models for identifying the target of each category; Brutal represents Prior Art 2: a plurality of category training data sets are combined into a training data set, and an image recognition model is trained using the training data set, so that the image recognition model is used to identify all categories; MTB represents the case where the target teacher model is not used when training the target detection network; Ours represents the target detection network provided by the present application. AP represents the ratio of the area of the intersection of the detection box and the labeled box to the area of the union of the detection box and the labeled box, AP 50 represents the case where the ratio of the area of the intersection of the detection box and the labeled box to the area of the union of the detection box and the labeled box exceeds 50%, AP 75 represents the case where the ratio of the area of the intersection of the detection box and the labeled box to the area of the union of the detection box and the labeled box exceeds 75%, AP, AP 50 , AP 75The greater the value, the higher the accuracy.

[0280] Please refer to Figure 5b From the values, it is clear that the detection accuracy of the target detection network provided by the embodiments of the present application is higher than that when the target teacher model is not used to train the target detection network, and is also higher than that of prior art 1 and prior art 2.

[0281] Please refer to Figure 5c , Figure 5c The first two rows except the table header represent that the backbone network of the target teacher model is R101 and R50, respectively, and the third row represents that the backbone network of the target detection network is R50 and the target detection network does not calculate the feature loss L ft and the distillation loss The fourth row represents that the backbone network of the target teacher model is R101, the backbone network of the target detection network is R50, and the target detection network calculates the feature loss L ft ; the fifth row represents that the backbone network of the target teacher model is R101, the backbone network of the target detection network is R50, and the target detection network calculates the distillation loss The sixth row represents that the backbone network of the target teacher model is R101, the backbone network of the target detection network is R50, and the target detection network calculates the feature loss L ft and the distillation loss The seventh row represents that the backbone network of the target teacher model is R50, the backbone network of the target detection network is R50, and the target detection network calculates the feature loss L ft and the distillation loss From Figure 5c It can be seen that whether the backbone network used by the target teacher model is the same as the backbone network of the target detection network, the accuracy of detection will be higher after calculating the feature loss L ft or the distillation loss

[0282] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or controlled by related hardware by instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0283] Therefore, the embodiments of the present application provide a computer readable storage medium, which stores a plurality of instructions capable of being loaded by a processor to execute the steps in any target detection method provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0284] ​Feature extraction is performed on a to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, the feature maps including feature pixels; for each feature map in the plurality of feature maps, multi-branch-based detection is performed to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training respective training data sets, and the plurality of training data sets each include different categories.

[0285] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or the like.

[0286] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method provided in any of the optional implementation manners of the target detection aspect or the model training aspect provided in the above embodiments.

[0287] Due to the instructions stored in the storage medium, the steps of any of the target detection methods provided in the embodiments of the present application can be performed, and thus the beneficial effects of any of the target detection methods provided in the embodiments of the present application can be achieved. Details are described in the above embodiments, and thus will not be described here.

[0288] The above describes in detail a target detection method, device, electronic device and computer readable storage medium provided in the embodiments of the present application. The principle and implementation manner of the present application are described by applying specific examples. The above embodiment descriptions are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed. In summary, the content of the present application should not be understood as a limitation.

Claims

1. A target detection method characterized by, The method comprises: feature extraction is performed on a to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, wherein the feature maps comprise feature pixels; for each of the plurality of feature maps, multi-branch-based detection is performed to obtain a branch detection result of each of the feature pixels in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of training data sets, and the plurality of training data sets each comprise different categories; the multi-branch-based detection for each of the plurality of feature maps to obtain the branch detection result of each of the feature pixels in the feature map corresponding to each branch comprises: performing convolution processing on a first feature map by using a first branch to obtain probability values of M categories corresponding to each pixel point on the first feature map, a weight value of each pixel point and a target center point of the pixel point, and N bounding box distance values corresponding to each pixel point, wherein the first branch is any branch in the plurality of branches, the training data set corresponding to the first branch comprises the M categories, the first feature map is any feature map in the plurality of feature maps, and M and N are positive integers; obtaining a target detection result based on the branch detection result of the feature pixels and displaying the target detection result.

2. The method of claim 1, wherein, The method for obtaining a target detection result based on the branch detection result of the feature pixels comprises: determining a probability value with the maximum value among the probability values of the M categories corresponding to each pixel point, taking the probability value with the maximum value as a binding probability value of the corresponding pixel point, and taking a category corresponding to the binding probability value as a binding category of the pixel point; sorting each pixel point on the first feature map according to the value of the binding probability value; determining P pixel points with the maximum binding probability value from the pixel points on the first feature map, wherein P is a positive integer; for each of the P pixel points, determining P detection boxes corresponding to the P pixel points respectively according to the N bounding box distance values corresponding to the pixel points, wherein the P detection boxes and the binding categories corresponding to the P pixel points respectively are the target detection result of the first feature map in the first branch.

3. The method of claim 1, wherein, The method for performing feature extraction on a to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture comprises: performing downsampling processing on the to-be-detected picture for a first number of times at different magnifications to obtain a first number of downsampling results, wherein the sizes of the first number of downsampling results are different; performing convolution processing on each of the first number of downsampling results to obtain a preliminary feature map corresponding to each downsampling result; performing downsampling processing on a preliminary feature map with the smallest size for a second number of times to obtain a second number of secondary feature maps, wherein the first number of preliminary feature maps and the second number of secondary feature maps together constitute the plurality of feature maps of different sizes.

4. The method of claim 1, wherein, Before the method for performing feature extraction on a to-be-detected picture to obtain a plurality of feature maps of different sizes corresponding to the to-be-detected picture, the method further comprises: obtaining a plurality of training data sets, wherein the plurality of training data sets correspond to the plurality of branches one by one, and each of the plurality of training data sets comprises different categories; training the branches corresponding to the training data sets by using the plurality of training data sets until the entire target detection network meets a preset condition, to obtain a trained target detection network.

5. The method of claim 4, wherein, The training of the branches corresponding to the training data sets by using the plurality of training data sets until the entire target detection network meets a preset condition, to obtain a trained target detection network, comprises: for each branch: performing feature extraction on a training picture from the training data set corresponding to the branch to obtain a plurality of training feature maps of different sizes corresponding to the training picture; detecting each training feature map in the plurality of training feature maps by using the branch to obtain a training detection result; determining a branch loss value corresponding to the branch according to the training detection result, a prediction result of a target teacher model on the training picture, and a labeled box in the training picture, wherein the target teacher model is a teacher model corresponding to the branch, and the target teacher model is a model trained on the training data set; determining a branch loss value corresponding to each of the plurality of branches, calculating a weighted sum of the branch loss values corresponding to each of the branches to obtain a total loss value; if the total loss value meets the preset condition, it is determined that the plurality of branches are trained; if the total loss value does not meet the preset condition, adjusting the parameters of each branch according to the total loss value, and iterating until the total loss value meets the preset condition.

6. The method of claim 5, wherein, The detection of each training feature map by using the branch to obtain a training detection result comprises: performing convolution processing on each training feature map by using the branch to obtain probability values of M' categories corresponding to each pixel point on the training feature map, a weight value of each pixel point and a target center point of itself, and N' distance values of a bounding box corresponding to each pixel point; wherein the training data set from which the training picture comes comprises the M' categories, and M' and N' are positive integers.

7. The method of claim 4, wherein, The plurality of branches comprises a first branch and a second branch, the first branch corresponds to a first training data set, the second branch corresponds to a second training data set, and any category in the first training data set is different from any category in the second training data set; Before the plurality of training data sets are obtained, the method further comprises: dividing a preset data set into the first training data set and the second training data set.

8. The method of claim 7, wherein, The division of the preset data set into the first training data set and the second training data set comprises: dividing all Q categories in the preset data set into the first training data set and the second training data set, wherein Q is a positive integer. divide all pictures in the preset data set into pictures of a category contained only in the first training data set, pictures of a category contained only in the second training data set, and pictures of a category contained in both the first training data set and the second training data set; evenly divide the pictures of a category contained in both the first training data set and the second training data set to the first training data set and the second training data set.

9. A target detection apparatus characterized by comprising: comprise: a feature map acquisition unit, configured to perform feature extraction on a to-be-detected picture, and acquire a plurality of feature maps of different sizes corresponding to the to-be-detected picture, wherein the feature maps comprise feature pixels; a branch result acquisition unit, configured to, for each feature map in the plurality of feature maps, perform detection based on a plurality of branches to obtain a branch detection result of each feature pixel in the feature map corresponding to each branch, wherein the plurality of branches are obtained by training a plurality of training data sets respectively, and the categories included in the plurality of training data sets are different from each other; and the detection based on the plurality of branches to obtain the branch detection result of each feature pixel in the feature map corresponding to each branch, for each feature map in the plurality of feature maps, comprises: performing convolution processing on a first feature map by using a first branch to obtain a probability value of M categories corresponding to each pixel point on the first feature map, a weight value of the each pixel point and a target center point of the each pixel point, and N bounding box distance values corresponding to each pixel point, wherein the first branch is any branch in the plurality of branches, the training data set corresponding to the first branch comprises the M categories, the first feature map is any feature map in the plurality of feature maps, and M and N are positive integers; a target result acquisition unit, configured to obtain a target detection result based on the branch detection result of the feature pixels, and display the target detection result.

Images