Object detection methods and their model training methods, equipment, and storage media

Abstract

This application discloses an object detection method and its model training method, device, and storage medium. The object detection model training method includes: acquiring sample images, each sample image being labeled with at least one object type; using the object detection model to determine target region features for each object type in the sample images, where the target region features represent image data features within the target region, and the target region is the area where the target type is located in the sample image; comparing the target region features for each object type with reference region features for several types to obtain multiple comparison results, where the reference region features include reference region features for each type from several historical sample images; determining a target loss based on the multiple comparison results; and adjusting the parameters in the object detection model using the target loss. This approach can improve the accuracy of subsequent object detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a target detection method, a training method for the target detection model, an apparatus, and a storage medium. Background Technology

[0002] Deep learning-based image processing methods have become a hot technology in recent years. Target localization and detection technologies, as mainstream methods in this field, have been widely applied in areas such as intelligent transportation, smart healthcare, smart city management, and smart home services, bringing safer, more comfortable, and convenient services to people's lives. Mainstream target detection technologies require labeling the location and category of targets in the training data. Then, convolutional neural networks are used to extract image or video features and perform classification and regression tasks to obtain the category and location information of targets in the image, thereby enabling the localization and identification of the required targets in the project. This type of target detection method relies on carefully labeled data for training. The labeling of this data requires a significant amount of manpower and time. In some specialized fields, such as smart healthcare and industrial defect detection, the targets in the images require even more specialized personnel to label them correctly, which undoubtedly increases the cost of labeling. Summary of the Invention

[0003] This application provides at least one object detection method, a training method and device for the same model, and a storage medium.

[0004] This application provides a training method for an object detection model, comprising: acquiring sample images, wherein the sample images are labeled with at least one object type contained in the sample images; using an object detection model to determine target region features in the sample images for each object type, wherein the target region features of the target type are used to represent image data features within the target region, and the target region is the region in the sample image where the target corresponding to the target type is located; comparing the target region features of each target type with reference region features of several types to obtain multiple comparison results, wherein the reference region features of several types include reference region features of each type in several historical sample images; determining a target loss based on the multiple comparison results; and adjusting the parameters in the object detection model using the target loss.

[0005] This application provides a target detection method, comprising: acquiring a target image; performing target detection on the target image using a target detection network to obtain an initial detection result corresponding to the target image, the initial detection result including the confidence that a target is contained in a target region contained in the target image; the target detection network is trained by the above-mentioned training method; and determining the target detection result of the target image based on the confidence corresponding to each target region.

[0006] This application provides a training apparatus for an object detection model, comprising: a sample image acquisition module for acquiring sample images, wherein the sample images are labeled with at least one object type contained in the sample images; a region feature acquisition module for determining target region features of each object type in the sample images using the object detection model, wherein the target region features of the target type are used to represent image data features within the target region, and the target region is the region in the sample image where the target corresponding to the target type is located; a feature comparison module for comparing the target region features of each object type with reference region features of several types to obtain multiple comparison results, wherein the reference region features of several types include reference region features of each type in several historical sample images; a loss determination module for determining a target loss based on the multiple comparison results; and a parameter adjustment module for adjusting the parameters in the object detection model using the target loss.

[0007] This application provides a target detection device, comprising: a target image acquisition module for acquiring a target image; a detection module for performing target detection on the target image using a target detection network to obtain an initial detection result corresponding to the target image, the initial detection result including the confidence that a target region contained in the target image contains a target; the target detection network is trained by the above-described training method; and a processing module for determining the target detection result of the target image based on the confidence corresponding to each target region.

[0008] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described target detection method or the above-described target detection model training method.

[0009] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described target detection method or the training method for the above-described target detection model.

[0010] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0013] Figure 1 This is a flowchart illustrating an embodiment of the target detection method of this application;

[0014] Figure 2 This is a schematic diagram of a sub-process of step S12, illustrating an embodiment of the training method for the object detection model of this application.

[0015] Figure 3 This is a schematic diagram of the target detection model of this application;

[0016] Figure 4 This is a schematic diagram of the location information attention mechanism network structure shown in the training method of the object detection model of this application;

[0017] Figure 5 This is a flowchart illustrating an embodiment of the target detection method of this application;

[0018] Figure 6 This is another flowchart illustrating an embodiment of the target detection method of this application;

[0019] Figure 7 This is a schematic diagram of the structure of an embodiment of the training device for the object detection model of this application;

[0020] Figure 8 This is a schematic diagram of the structure of an embodiment of the target detection device of this application;

[0021] Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0022] Figure 10 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0023] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0024] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0025] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the target detection method of this application.

[0027] like Figure 1 As shown, the target detection method provided in this disclosure embodiment may include the following steps:

[0028] Step S11: Obtain sample images.

[0029] The sample images are labeled with at least one target type they contain. The target type refers to the type of target. For example, the target type can be any target to be detected, such as a human body, a face, a vehicle, or a sheep. For instance, if a sample image contains a vehicle, then the sample image can be labeled with an identifier indicating that it contains a vehicle. For example, the number of sample images can be single or multiple; for example, a batch of sample images can be used to train the target detection model.

[0030] Step S12: Use the object detection model to determine the target region features of each target type in the sample image.

[0031] In this context, the target region feature of the target type is used to represent the image data features within the target region. The target region is the area in the sample image where the target corresponding to the target type is located. For example, the target detection model first determines the target region in the sample image where the target corresponding to the target type is located, and then determines the image data features of the target region as the target region feature.

[0032] Step S13: Compare the target region features of each target type with the reference region features of several types to obtain multiple comparison results.

[0033] "Several" refers to one or more types. Among these "several types," there may be types identical to the target type, or there may be no types identical to the target type. The reference region features for each type include reference region features from several historical sample images. For example, the "several types" may include sheep, and the target type is a vehicle. From appearance and other aspects, sheep and vehicles differ significantly in images. If the comparison results between the extracted region features show that they are quite similar, it's clear that there's a problem with feature extraction. Therefore, determining the target loss based on the comparison results between the two can increase the feature gap between classes, making subsequent target detection results more accurate. Alternatively, if the "several types" includes vehicles, and the target type is also a vehicle, the differences between the two in images should be small. If the comparison results between the extracted region features show that they are quite far apart, it's clear that there's a problem with feature extraction. Therefore, determining the target loss based on the comparison results between the two can reduce the feature gap within classes, making subsequent target detection results more accurate.

[0034] Step S14: Determine the target loss based on multiple comparison results.

[0035] Specifically, the target loss can be determined based on the comparison results between the target region features of the target category and the reference region features of each category.

[0036] Step S15: Adjust the parameters in the target detection model using the target loss.

[0037] The specific method for adjusting the parameters in the model using the target loss can be any method, such as backpropagation; no specific limitations are made here.

[0038] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0039] In some embodiments, the sample images include the original image and the enhanced image obtained by pixel enhancement processing of the original image. Optionally, there may be multiple sample images. For example, there may be multiple original images. In some application scenarios, for a sample dataset I, the sample dataset contains several original images and their corresponding category labels, where C is the total number of target categories in the sample dataset. In the category labels, the category position of the target contained in the original image is 1, and the category position of the target not contained in the original image is 0. For example, if the original image contains category A, then in the label of category A, the category position corresponding to the original image is 1. An original image may contain one or more categories, and the target of the same category may contain one or more. During training, the data is fed into the network for training according to the set batch size N. The original images and enhanced images are trained on the model in the same way, which will be described uniformly below.

[0040] Before each batch of data is fed into the network for training, each original image undergoes random pixel enhancement to generate a corresponding enhanced image, forming an image pair. Pixel enhancement methods include Gaussian blur, random brightness variations, adding random salt-and-pepper noise, or a random combination of several methods. For example, there are multiple original images, and each original image corresponds to one enhanced image.

[0041] In some embodiments, step S11 above may include the following steps:

[0042] Multiple original images are acquired, and it is determined whether the number of target original images is greater than or equal to a preset number. Here, a target original image is an original image containing a target category whose number is less than or equal to a preset value. For example, if the preset value is 1, and the original image only indicates that category A exists in the image (meaning the original image contains 1 target category), then that original image is the target original image.

[0043] In some embodiments, in response to the number of target original images being greater than or equal to a preset number, a reference original image containing other target categories is stitched together with the target original image to obtain the final original image, and an enhanced image of the reference original image is stitched together with an enhanced image of the target original image to obtain the final enhanced image.

[0044] For example, if the target category in the original image is category A, then the other categories can be any category other than category A, such as category B or category C. In some embodiments, the target original image can be stitched together using several other reference original images. The number of reference original images can be preset. For example, to ensure the integrity of the stitched image, three reference original images can be used to stitch the target original image together. Furthermore, the target categories contained in the several reference original images can be the same or all different target categories.

[0045] In some application scenarios, for original images containing only one type of target (target original images), random data stitching is performed based on probability if the image is a target original image and satisfies the probability of stitching enhancement. Specifically, the probability is the ratio of the number of target original images to the total number of original images in a batch, which is the preset value mentioned above. Three target original images containing different types of targets will be randomly selected from the same batch of data and randomly stitched together in a 2x2 order to form a multi-class image. In other words, the reference original image can also be other target original images. Furthermore, the label corresponding to the final original image obtained through stitching is a fusion of the labels of the four original images before stitching. Before stitching, the four original images to be stitched are each randomly enhanced to generate corresponding enhanced images, and then the enhanced images are stitched together. Finally, the generated final original image and / or final enhanced image are Gaussian blurred to eliminate stitching boundary effects.

[0046] In some application scenarios, for original images containing multiple target categories, random pixel enhancement is directly performed to generate corresponding enhanced images, which are then fed into the network as comparison data for learning. All enhanced images are generated online before each epoch iteration. Thus, a batch of original image data generates a batch of enhanced image data to form image pairs, which are then fed into the backbone network for learning. During the learning process, the weights of the entire backbone network are shared.

[0047] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating a sub-process of step S12 in an embodiment of the training method for the object detection model of this application. Figure 2 As shown, in some embodiments, step S12 above may include the following steps:

[0048] Step S121: Use the object detection model to extract initial features from the sample images to obtain initial feature maps for each object type.

[0049] Please see Figure 3 , Figure 3This is a schematic diagram of the target detection model in this application. (See attached diagram.) Figure 3 As shown, the original data and the generated augmented data are fed into the backbone network for feature extraction and semantic information learning. Figure 3 The diagram shows the last three feature extraction modules of the backbone network: F1, F2, and F3. F1, F2, and F3 are cascaded together. The outputs of F1, F2, and F3 can all be considered as initial feature maps. For example, the set of initial feature maps belongs to R. H×W×(Cm) Where m is a hyperparameter representing the number of channels in the initial feature map for a certain category, i.e., the number of initial feature maps corresponding to each target type. H and W are the height and width of the initial feature map, and C represents the total number of target categories contained in all sample images, i.e., the total number of target categories contained in the sample images in the entire training set. In other words, in some scenarios, the object detection model can determine the initial feature map, the first feature map, and the second feature map for all categories in the sample image during the detection process.

[0050] Step S122: Perform advanced feature extraction on the initial feature maps of each target type to obtain the first feature map and the second feature map of each target type.

[0051] The first and second feature maps respectively contain the location information of the target region. For example... Figure 3 In the middle, the set of first feature maps can be Figure 3 In LIA1, the set of second feature maps can be Figure 3 LIA2 in the model. The location information of the target region may differ between the first and second feature maps; for example, the location information in the second feature map may be more accurate.

[0052] In some embodiments, the method for obtaining the first feature map and the second feature map in step S122 may be: performing a first feature extraction on the initial feature map to obtain a first feature map for each target type; and performing a second feature extraction on the first feature map to obtain a second feature map for each target type. Wherein, the number of channels in the first feature map for each target type is greater than the number of channels in the second feature map for the corresponding target type. Wherein, the number of channels in the first feature map for each target type is less than the number of channels in the initial feature map for each target type.

[0053] For example, a 1×1 convolution is used to map the number of channels of the initial feature map to C×k, generating the first feature map set LIA1∈R corresponding to each sample image. H×W×()H and W are the height and width of the first feature map, C represents the total number of target categories contained in all sample images, and k is a hyperparameter representing the number of channels in the first feature map for each target type. Optionally, k is less than or equal to m. In other words, by performing first feature extraction on the initial feature map, the goal is to compress the feature map of each target category into a set of first feature maps with k channels.

[0054] One method for obtaining the first feature map is to use the first feature extraction module in the target detection model to extract the first feature from the initial feature map, thereby obtaining the first feature map for each target type.

[0055] Optionally, the first feature extraction module can be designed based on a location information attention mechanism. This first feature extraction module can be seamlessly embedded into any backbone network for further extraction of location information.

[0056] The second feature map can be obtained by using the third feature extraction module in the object detection model to extract second features from the first feature map, thus obtaining second feature maps for each object type. The third feature extraction module can be designed based on a location information attention mechanism. This third feature extraction module can be seamlessly embedded into any backbone network for further extraction of location information.

[0057] For example, the features of every k channels in every C×k channel of the generated LIA1 are taken as a feature unit, and 1×1 convolutions are performed on each of them to compress the number of channels from k to 1, generating a second feature map set LIA2∈R. H×W×C Where C is the number of target categories. Through the second feature extraction, the location information of each target category is compressed into one channel, and the feature map of each channel contains the location information of the corresponding target category.

[0058] Step S123: Process each second feature map separately to generate a region mask corresponding to the target type.

[0059] In some embodiments, before executing step S123, the following steps may also be performed: Using the second feature maps of each target type, sample detection results of the sample image are obtained. The sample detection results include the confidence that the target contained in each target region of the sample image is of the target type. Then, from the second feature maps of each target type, the second feature map whose confidence satisfies a preset confidence condition is selected as the final second feature map. Based on this, each second feature map is processed separately to generate a region mask corresponding to the target type. Specifically, this can be achieved by processing each final second feature map separately to generate a region mask corresponding to the relevant target type.

[0060] As mentioned above, the second feature map set contains second feature maps corresponding to C categories. The sample detection result can then obtain the confidence level of each category in the sample image, where the target is the corresponding category. The confidence level can be a prediction probability. For example, the prediction label corresponding to each prediction probability is determined based on the magnitude of the category prediction probability. For instance, if the category prediction probability is greater than or equal to a preset probability, the prediction label corresponding to that category prediction probability is determined to be 1; otherwise, it is 0. From the second feature maps of each target type, the method of selecting the second feature map whose confidence level meets the preset confidence condition as the final second feature map can be to select the second feature map corresponding to the prediction label 1 as the final second feature map. The number of final second feature maps selected is less than or equal to the number of target categories contained in the sample image. For example, if only four target categories are labeled in the sample image, the maximum number of final second feature maps selected is four. If the number of prediction labels 1 is more than four, the corresponding final second feature maps can be selected according to the prediction probability from high to low. Furthermore, for cases where the label contains a target category but the classification confidence given by the target detection model is low, it is considered that the current network's learning of semantic features is not complete enough; therefore, the final second feature map is not determined for that target category. The determined final second feature map is used to generate the region mask corresponding to the relevant type.

[0061] Specifically, the method of processing each second feature map to generate the region mask corresponding to the target type can be as follows: perform binarization processing on each second feature map to generate the region mask corresponding to each second feature map.

[0062] The final second feature map can be considered as the location information activation map of the relevant category. The location activation map is then binarized according to a predefined threshold υ, with values ​​less than υ set to 0 and values ​​greater than υ set to 1, generating a mask M for the corresponding category. c ∈R H×W .

[0063] Step S124: For each target type, process the first feature map corresponding to the target type using a region mask to obtain the target region features corresponding to the target type.

[0064] Specifically, step S124 can be: performing masked average pooling on the first feature map using a region mask to obtain the target region features.

[0065] In other embodiments, masked average pooling (MAP) can be performed on the first feature maps corresponding to each category to obtain the target region features of the target category. As mentioned above, the total number of categories in the entire training set is C. If the sample image contains category d, masked average pooling is performed to obtain the target region features f of target category d. d The method can be referenced from Formula 1:

[0066]

[0067] Among them, M c (i,j) represents the mask M c ∈R H×W The mask at position (i,j) in the middle. Where LIA1∈R H×W×() LIA1(i,j) is the pixel value at position (i,j) in the first feature map of a certain channel. k indicates that the number of channels in the first feature map of target category d is k. The first feature map of each channel needs to be processed to obtain f. d ∈R C×k This method enables the activation of the target region in the forward pass of the target detection model.

[0068] In some embodiments, before performing step S12, the following steps may be performed: Third feature extraction is performed on the initial feature maps of each target type to obtain reference feature maps for each target type. The reference feature maps contain the location information of the target regions. For each target type, the reference feature map of the target type is processed using a region mask to obtain the reference region features corresponding to the target type. Then, the reference region features of several types are updated using the reference region features corresponding to each target type in the sample image. The updated reference region features of several types contain the reference region features of each target type in the sample image. Based on this, step S12 may specifically include the following steps: comparing the target region features of each target type with the updated reference region features of several types to obtain multiple comparison results.

[0069] In some embodiments, the specific method for obtaining the reference feature map may be: using the second feature extraction module in the object detection model to perform a third feature extraction on the initial feature map to obtain reference feature maps for each target type. Optionally, the structure of the first feature extraction module can be the same as that of the second feature extraction module. Specifically, the initial feature map for the third feature extraction and the initial feature map used to obtain the target region features are obtained by the same feature extraction module in the backbone network. For example, the initial feature map output by the feature extraction module F3 is used to determine the target region features and the reference region features. That is, the network structures of the first feature extraction module and the second feature extraction module connected to F3 are the same.

[0070] That is, the set of reference feature maps of various types obtained by the second feature extraction module is LIA′1∈R. H×W×(Ck) The method of processing the reference feature map using a region mask to obtain the reference region features can be referred to Formula 1 above, and will not be repeated here.

[0071] The feature matrix consists of several types of regional features. Each row in the feature matrix corresponds to a set of regional features of one type, and different rows correspond to different types. The comparison results include a similarity matrix. The method described above for comparing the target regional features of each target type with reference regional features of several types to obtain multiple comparison results can be: calculating the similarity between the target regional features of each target type and the feature matrix to obtain a similarity matrix. In this similarity matrix, each row corresponds to the similarity between the target regional features of one target type and the historical regional features of each type. Based on this, the method described above for determining the target loss based on multiple comparison results can be: determining the target loss based on the similarity matrix.

[0072] For example, in the same batch of data, the reference region features in all sample images containing category d will participate in feature caching. middle The construction of . In each Epoch iteration, It stores the reference region features of the entire Epoch image to form a feature matrix, which serves as a contrastive learning correlation matrix for the region features. This matrix is ​​used to calculate the similarity matrix Q between the target region features in each sample image and the reference region features in the images throughout the entire Epoch.

[0073] Assume that the final batch of data will generate region-level feature vectors f corresponding to n sample images, and assume that the current feature cache... m is the number of all feature vectors f′, where m ≥ n. For a batch of data, the generated... Can calculate and The similarity Q is calculated using the following formula:

[0074] Q = M Batch ·M T ∈R n×m Formula 2;

[0075] Where T represents transpose. Each row in Q represents a target region feature of the current batch of data. The similarity of all reference region features is calculated. Optionally, if the similarity is greater than a preset similarity, the two regions are considered to belong to the same category, and Q is binarized, setting the similarity of features belonging to the same category to 1; otherwise, it is set to 0.

[0076] The target loss can be the cross-entropy loss. The loss is calculated using the cross-entropy loss function (CE Loss), the gradient is calculated and backpropagated, and the network parameters are updated.

[0077] In this way, the object detection model can integrate all previously learned region-level semantic features while learning the semantic features of the current batch of data. This reduces the similarity between features of the same type and increases the differences between features of different types, allowing the model to learn the semantic information of the entire dataset more comprehensively and accurately. Furthermore, each iteration of the model references all learned semantic information, preventing the specific characteristics of a particular batch of data from affecting the learning of semantic features.

[0078] In some embodiments, the structure of the first feature extraction module described above can be the same as the structure of the second feature extraction module. For example... Figure 3 In this process, the structure of the first feature extraction module E3 is the same as that of the second feature extraction module. Based on this, step S15 may include the following steps: adjusting the parameters of the first feature extraction module using the target loss through backpropagation to obtain updated parameters for the first feature extraction module. Then, fusing the updated parameters of the first feature extraction module with the parameters to be updated in the second feature extraction module, and using the fused parameters to update the second feature extraction module.

[0079] For the parameters of the second feature extraction module, momentum updates are used. The timing of momentum updates can be configured. After each replacement of all data by E3, the second feature extraction module is updated through backpropagation. The second feature extraction module can be updated together with E3, or it can be updated after several iterations. The update process can be referred to Formula 3:

[0080] θ′←γθ′+(1-γ)θ Formula 3;

[0081] Where θ represents the updated parameters after the current iteration of E3, and θ′ represents the parameters of the second feature extraction module. Parameter updates in the second feature extraction module do not require backpropagation, thus eliminating the need for extensive GPU computation and storage.

[0082] As described above, in some embodiments, the sample image includes the original image and an enhanced image obtained by pixel enhancement processing of the original image. Then, the second feature map includes the first sub-feature map of each target type corresponding to the original image and the second sub-feature map of each target type corresponding to the enhanced image. Based on this, the training method of the target detection model provided in this embodiment may further include the following steps: fusing the first sub-feature maps of each target type to obtain a first foreground feature map containing the positional information of the foreground in the original image. The foreground in the original image is a set of target regions. And fusing the second sub-feature maps of each target type to obtain a second foreground feature map containing the positional information of the foreground in the enhanced image. The foreground in the enhanced image is a set of target regions. Then, based on the positional difference of the foreground between the first foreground feature map and the second feature map, a positional difference loss is determined. Based on obtaining the positional difference loss, step S15 above may specifically include: adjusting the parameters in the target detection model using the target loss and the positional difference loss.

[0083] For example, the first foreground feature map can be obtained by adding the corresponding LIA2 values ​​of the original image along the channel direction to fuse the positional information of targets corresponding to all categories. Then, using the Sigmoid function, all positional information is normalized to the range {0,1} to generate the first foreground feature map LIA3. LIA3 is a category-independent foreground positional information activation map; that is, the positional information contained in LIA3 is the positional information of the foreground, i.e., LIA3∈R. H×W The method for obtaining the second foreground feature map is similar and will not be repeated here. The second feature maps can be fused by directly adding them together or by performing a weighted fusion.

[0084] To better understand the process of changing the number of channels in generating a second feature map from a first feature map, and then generating the corresponding first or second foreground feature map, please refer to [reference needed]. Figure 4 , Figure 4 This is a schematic diagram illustrating the structure of the location information attention mechanism network used in the training method of the object detection model in this application. Specifically, the first feature extraction module and the third feature extraction module belong to the location information attention mechanism network. For example... Figure 4 As shown, the first feature map set LIA1∈R H×W×() That is, the number of channels in the first feature map of each category in the first feature map set is k, and the total number of channels is Ck. The second feature map set is LIA2∈R. H×W×CThat is, each category of the second feature map in the second feature map set has 1 channel, and the total number of channels is C. Then, the second feature maps of each category are fused to obtain the foreground feature map LIA3∈R. H×W This means the foreground feature map is category-independent and has a total of 1 channel. The formula for obtaining the foreground feature map from the initial feature map can be found in Formula 4:

[0085]

[0086] in, This represents the process of mapping the initial feature map F to the final LIA3.

[0087] In some embodiments, there are multiple initial feature maps. These multiple initial feature maps are obtained by multiple cascaded first feature extraction modules in the object detection model, as described above. The first feature extraction modules may include F1, F2, and F3, and the initial feature maps may include the initial feature map corresponding to F1, the initial feature map corresponding to F2, and the initial feature map corresponding to F3. There are also multiple second feature maps. That is, the second feature maps include the initial feature map corresponding to F1, the second feature map corresponding to F2, and the second feature map corresponding to F3. The number of first foreground feature maps and second foreground feature maps is the same as the number of initial feature maps, and the first foreground feature maps and second foreground feature maps correspond one-to-one. Specifically, the first foreground feature maps include the first foreground feature map corresponding to F1, the first foreground feature map corresponding to F2, and the first foreground feature map corresponding to F3. The second foreground feature maps include the second foreground feature map corresponding to F1, the second foreground feature map corresponding to F2, and the second foreground feature map corresponding to F3. Specifically, F1 corresponds to the first foreground feature map and the second foreground feature map, F2 corresponds to the first foreground feature map and the second foreground feature map, and F3 corresponds to the first foreground feature map and the second foreground feature map. Based on this, the method for determining the positional difference loss based on the positional differences between the first and second foreground feature maps can be as follows: determine the positional difference loss based on the differences between each first foreground feature map and its corresponding second foreground feature map.

[0088] The positional difference loss between the first foreground feature map and the second foreground feature map can be calculated using the L2 loss. For example, the L2 loss can be calculated using Formula 5:

[0089]

[0090] Among them, F k and Let k represent the initial feature maps of the original image and the enhanced image, respectively, where k = 1, 2, 3, and when k = 1, ... This represents the positional difference loss corresponding to F1, when k=2. This represents the positional difference loss corresponding to F2, when k=3. Let F3 represent the positional difference loss. (i,j) represents the position of a pixel on the foreground feature map. This represents the first foreground feature map. This represents the second foreground feature map. W and H represent the width and height of the foreground feature map, respectively.

[0091] By mutually constraining the feature activation maps (first foreground feature map and second foreground feature map) generated at different network depths, the network is guided to learn more abstract semantic features, rather than being limited to shallow features such as edges, colors, and textures.

[0092] In some embodiments, the training method for the object detection model provided in this embodiment may further include the following steps: determining the sample detection results of the sample image using the object detection model. The sample detection results include the confidence that the target contained in each target region of the sample image is of the target type. Determining the binary cross-entropy loss corresponding to the sample detection results. Based on this, the parameters in the object detection model can be adjusted by combining the target loss and the binary cross-entropy loss.

[0093] The sample detection results can be obtained from the second feature map. For example, the LIA2 obtained from the last feature extraction module (such as F3 above) is subjected to Global Average Pooling (GAP), and then a fully connected layer is used to compress the generated feature vector into a vector of length C as the final confidence level. The confidence level can be the class prediction probability, ultimately yielding C class prediction probabilities. The C class prediction probabilities can be obtained by referring to Equations 6 and 7.

[0094]

[0095]

[0096] in, This represents the process of mapping the initial feature map F to the final LIA2. This indicates that the content within the parentheses will undergo global average pooling (GAP) processing. This indicates that a fully connected layer is used to compress the generated feature vector into a vector of length C. For example, the predicted label is determined based on the magnitude of the predicted probability for each category. For instance, if the predicted probability of a category is greater than or equal to a preset probability, the predicted label corresponding to that category's predicted probability is determined to be 1; otherwise, it is 0.

[0097] Since the sample images only contain class labels, the binary cross-entropy loss function (BCE Loss) is used to calculate the loss. For details on how to calculate the binary cross-entropy loss, please refer to Formula 8:

[0098]

[0099] Where C represents the total number of data categories, i.e., the length of the label vector y, and y and These represent the predicted label and the true class label output by the object detection model, respectively. As mentioned above, when labeling sample images, if class A exists in the sample image, the corresponding position in the label for class A is 1; otherwise, the corresponding position is 0. In other words, each true label can be either 0 or 1.

[0100] In some embodiments, the target loss, positional difference loss, and binary cross-entropy loss are used together to adjust the model parameters. The parameters can be adjusted sequentially using each target loss, or by fusing the losses and using the fused loss to adjust the parameters. Specific details regarding how to combine the various losses to adjust the model parameters are not specified here.

[0101] In some application scenarios, when performing forward inference on a trained object detection model, it is only necessary to generate LIA1 and LIA2, select the final second feature map based on the classification confidence, and then threshold the final second feature map by setting a threshold τ to segment out the regions with high confidence and restore them to the original image to generate the minimum bounding rectangle as the final detection result.

[0102] In this embodiment, the original image and the enhanced image are simultaneously fed into the object detection model for learning. At different stages of the model, corresponding positional information attention maps LIA1, LIA2, and LIA3 are generated. LIA3 is a class-agnostic foreground positional information activation map. The original image and the enhanced image are constrained by positional difference loss at different depths of LIA3 in the model to maximize the preservation of the target's positional information in the network. Simultaneously, the deepest layer (e.g., F3) features LIA1 and LIA2 are used to generate region-level semantic features for the construction of a Memory Bank (reference region features for several categories). The Memory Bank divides the semantic information of different categories into different groups as a reference for semantic information. After the Memory Bank is constructed, it guides the model's subsequent learning of semantic features, maximizing inter-class differences and minimizing intra-class differences. Finally, after the model training is completed, the target feature activation map (second feature map) to be activated is directly selected based on the deepest layer feature LIA2 and the classification confidence. After the feature activation map is thresholded, it is applied back to the original image to achieve the target detection task of the corresponding category.

[0103] The target detection method can be executed by a target detection device, such as a terminal device, server, or other processing device. The terminal device can be a monitoring device in a security system, a network video recorder, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the target detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0104] Please see Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of the target detection method of this application.

[0105] like Figure 5 As shown, the target detection method provided in this embodiment includes the following steps:

[0106] Step S21: Obtain the target image.

[0107] The target image can be acquired either by the device executing the target detection method or by transmission from other devices. The target image can be any image containing the target. The target can be any object to be detected, such as a human body or a vehicle.

[0108] Step S22: Use an object detection network to perform object detection on the target image to obtain the initial detection result corresponding to the target image. The initial detection result includes the confidence that the target region contained in the target image contains the target.

[0109] The method for obtaining the initial detection results can be found in the training method embodiment of the target detection model, and will not be described in detail here.

[0110] Step S23: Determine the target detection result of the target image based on the confidence level corresponding to each target region.

[0111] For example, if the confidence level is high, it is determined that the target exists in the target image.

[0112] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0113] For a better understanding of the object detection process, please refer to [link / reference]. Figure 6 , Figure 6 This is another flowchart illustrating an embodiment of the target detection method of this application.

[0114] like Figure 6 As shown, the target image is first input into the feature extraction module in the backbone network for general feature extraction, and the output of the last feature extraction module is used as the initial feature. Then, it passes through the first feature extraction module E3 to obtain the first feature map set. Then, feature extraction is performed on the first feature map set to obtain the second feature map set. Then, GAP and fully connected layers are applied to the second feature map set to obtain a vector of length C as the confidence level for each category. Then, the category with a confidence level higher than the preset confidence level is taken as the final category and the second feature map corresponding to the final category is determined. Then, the second feature map is thresholded according to the set threshold τ to segment the region with high confidence. The segmented second feature map is then restored to the original image to generate the minimum bounding rectangle, and the target detection result is obtained.

[0115] The target detection method can be executed by a target detection device, such as a terminal device, server, or other processing device. The terminal device can be a monitoring device in a security system, a network video recorder, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the target detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0116] Please see Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the training device for the target detection model of this application. The training device 30 for the target detection model includes a sample image acquisition module 31, a region feature acquisition module 32, a feature comparison module 33, a loss determination module 34, and a parameter adjustment module 35. The sample image acquisition module 31 is used to acquire sample images, which are marked with at least one target type contained in the sample images; the region feature acquisition module 32 is used to determine the target region features of each target type in the sample images using the target detection model, where the target region features of the target type are used to represent the image data features within the target region, and the target region is the region where the target corresponding to the target type is located in the sample image; the feature comparison module 33 is used to compare the target region features of each target type with the reference region features of several types to obtain multiple comparison results, where the reference region features of several types include the reference region features of each type in several historical sample images; the loss determination module 34 is used to determine the target loss based on the multiple comparison results; and the parameter adjustment module 35 is used to adjust the parameters in the target detection model using the target loss.

[0117] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0118] The functions of each module can be found in the implementation examples of the target detection method, and will not be repeated here.

[0119] Please see Figure 8 , Figure 8This is a schematic diagram of an embodiment of the target detection device of this application. The target detection device 40 includes a target image acquisition module 41, a detection module 42, and a processing module 43. The target image acquisition module 41 is used to acquire a target image; the detection module 42 is used to perform target detection on the target image using a target detection network to obtain an initial detection result corresponding to the target image, the initial detection result including the confidence that the target region contained in the target image contains a target; the target detection network is trained by the above-described training method; the processing module 43 is used to determine the target detection result of the target image based on the confidence corresponding to each target region.

[0120] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0121] The functions of each module can be found in the implementation examples of the target detection method, and will not be repeated here.

[0122] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in any of the above-described target detection method embodiments. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, monitoring equipment, microcomputers, and servers. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0123] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described target detection method embodiments. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0124] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0125] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described embodiments of the target detection method.

[0126] The above approach reduces the amount of data labeling by training the object detection model using only sample images labeled with the categories containing the target, eliminating the need to label the target's location. Furthermore, during training, target region features are extracted from the sample images, and these features are compared with reference region features from other images for each category. This reduces intra-category feature differences and increases inter-category feature differences, resulting in more accurate object detection results from the subsequent object detection model.

[0127] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0128] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A training method for an object detection model, characterized in that, include: Obtain a sample image, the sample image being labeled with at least one target type contained therein; The target region features for each target type in the sample image are determined using a target detection model. The target region features for each target type are used to represent the image data features within the target region. The target region is the region in the sample image where the target corresponding to the target type is located. The target region features of each target type are compared with reference region features of several types to obtain multiple comparison results. The reference region features of several types include reference region features of each type in several historical sample images. Based on the multiple comparison results, the target loss is determined; The parameters in the target detection model are adjusted using the target loss. The step of using a target detection model to determine the target region features of each target type in the sample image includes: The target detection model is used to extract initial features from the sample images to obtain initial feature maps for each target type. Advanced feature extraction is performed on the initial feature maps of each target type to obtain a first feature map and a second feature map of each target type, wherein the first feature map and the second feature map respectively contain the location information of the target region; Each of the second feature maps is processed to generate a region mask corresponding to the target type; For each target type, the first feature map corresponding to the target type is processed using the region mask to obtain the target region feature corresponding to the target type; Before comparing the target region features of each target type with the reference region features of several types to obtain multiple comparison results, the method further includes: A third feature extraction is performed on the initial feature map of each target type to obtain a reference feature map of each target type, wherein the reference feature map contains the location information of the target region; For each target type, the reference feature map of the target type is processed using the region mask to obtain the reference region features corresponding to the target type; The reference region features of the several types are updated using the reference region features corresponding to each target type in the sample image. The updated reference region features of the several types include the reference region features of each target type in the sample image. The step involves comparing the target region features of each target type with reference region features of several types to obtain multiple comparison results, including: The target region features of each target type are compared with the updated reference region features of several types to obtain multiple comparison results.

2. The method according to claim 1, characterized in that, The step of performing advanced feature extraction on the initial feature maps of each target type to obtain a first feature map and a second feature map for each target type includes: The initial feature map is subjected to first feature extraction to obtain a first feature map for each of the target types; Perform second feature extraction on the first feature map to obtain a second feature map for each of the target types; In this case, the number of channels in the first feature map of each target type is greater than the number of channels in the second feature map of the corresponding target type.

3. The method according to claim 1, characterized in that, Before processing each of the second feature maps to generate the region mask corresponding to the target type, the method further includes: Using the second feature map of each of the target types, the sample detection result of the sample image is obtained, and the sample detection result includes the confidence that the target contained in each of the target regions of the sample image is of the target type; From the second feature maps of each target type, select the second feature map whose confidence level meets the preset confidence condition as the final second feature map; The step of processing each of the second feature maps to generate a region mask corresponding to the target type includes: Each of the final second feature maps is processed to generate a region mask corresponding to the target type.

4. The method according to claim 1, characterized in that, The step of processing each of the second feature maps to generate a region mask corresponding to the target type includes: Each of the second feature maps is binarized to generate a region mask corresponding to each of the second feature maps; For each target type, the first feature map corresponding to the target type is processed using the region mask to obtain the target region features corresponding to the target type, including: The target region features are obtained by performing masked average pooling on the first feature map using the region mask.

5. The method according to claim 1, characterized in that, The step of performing advanced feature extraction on the initial feature maps of each target type to obtain a first feature map and a second feature map for each target type includes: The first feature extraction module in the target detection model is used to perform first feature extraction on the initial feature map to obtain a first feature map for each of the target types; The step of performing third feature extraction on the initial feature maps of each target type to obtain reference feature maps for each target type includes: The second feature extraction module in the target detection model is used to perform a third feature extraction on the initial feature map to obtain a reference feature map for each target type. The structure of the first feature extraction module is the same as that of the second feature extraction module. The step of adjusting the parameters in the target detection model using the target loss includes: The parameters of the first feature extraction module are adjusted using the target loss through backpropagation to obtain the updated parameters of the first feature extraction module. The updated parameters from the first feature extraction module are fused with the parameters to be updated in the second feature extraction module, and the fused parameters are used to update the second feature extraction module.

6. The method according to any one of claims 1-4, characterized in that, The sample image includes an original image and an enhanced image obtained by pixel enhancement processing of the original image. The second feature map includes a first sub-feature map of each target type corresponding to the original image and a second sub-feature map of each target type corresponding to the enhanced image. The method further includes: The first sub-feature maps of each target type are fused to obtain a first foreground feature map containing the position information of the foreground in the original image, wherein the foreground in the original image is a set of target regions; and the second sub-feature maps of each target type are fused to obtain a second foreground feature map containing the position information of the foreground in the enhanced image, wherein the foreground in the enhanced image is a set of target regions. Based on the positional differences of the foreground in the first foreground feature map and the second feature map, the positional difference loss is determined; The step of adjusting the parameters in the target detection model using the target loss includes: The parameters in the target detection model are adjusted using the target loss and the position difference loss.

7. The method according to claim 6, characterized in that, The number of initial feature maps is multiple, and the multiple initial feature maps are obtained by multiple cascaded first feature extraction modules in the target detection model. The number of second feature maps is multiple, and the number of first foreground feature maps and second foreground feature maps is the same as the number of initial feature maps, and the first foreground feature maps and second foreground feature maps correspond one-to-one. The step of determining the position difference loss based on the positional difference of the foreground in the first foreground feature map and the second feature map includes: The positional difference loss is determined based on the differences between each of the first foreground feature maps and the corresponding second foreground feature maps.

8. The method according to claim 6, characterized in that, The number of sample images is multiple, and the acquisition of sample images includes: Acquire multiple original images and determine whether the number of target original images is greater than or equal to a pre-defined number, wherein the target original images are original images containing a number of target categories that are less than or equal to a preset value; In response to the number of target original images being greater than or equal to a preset number, a reference original image containing other target categories is stitched together with the target original image to obtain the final original image, and an enhanced image of the reference original image is stitched together with an enhanced image of the target original image to obtain the final enhanced image.

9. The method according to any one of claims 1-4, characterized in that, Several types of regional features form a feature matrix. Each row in the feature matrix corresponds to a set of region features of a certain type, and different rows correspond to different types. The target region features of each target type are compared with reference region features of several types to obtain multiple comparison results, including: The similarity between the target region features of each target type and the feature matrix is ​​calculated to obtain a similarity matrix. Each row of the similarity matrix corresponds to the similarity between the target region features of a target type and the historical region features of each type. The comparison result includes the similarity matrix. The determination of the target loss based on the multiple comparison results includes: The target loss is determined based on the similarity matrix.

10. The method according to any one of claims 1-4, characterized in that, The method further includes: The target detection model is used to determine the sample detection result of the sample image, and the sample detection result includes the confidence that the target contained in each target region of the sample image is of the target type; Determine the binary cross-entropy loss corresponding to the sample detection result; The step of adjusting the parameters in the target detection model using the target loss includes: The parameters in the target detection model are adjusted by combining the target loss and the binary cross-entropy loss.

11. A target detection method, characterized in that, Acquire the target image; The target image is subjected to target detection using a target detection network to obtain an initial detection result corresponding to the target image. The initial detection result includes the confidence that the target region contained in the target image contains the target. The target detection network is trained by any one of the training methods of claims 1-10. The target detection result of the target image is determined based on the confidence level corresponding to each target region.

12. An electronic device, characterized in that, It includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 11.

Images