A target detection method and device, electronic equipment and storage medium

Abstract

The application relates to the computer technical field and provides a target detection method and device, electronic equipment and a storage medium. A target basic feature map is obtained by inputting a to-be-processed image into a target basic feature extraction network of a target detection model; target position information is obtained by inputting the target basic feature map into a target positioning branch network of the target detection model; a target classification feature map corresponding to each classification level is obtained by inputting the target basic feature map into a target hierarchical classification branch network of the target detection model; for each target classification branch network corresponding to a classification level, the target classification feature map corresponding to the classification level and the target classification feature map corresponding to a lower classification level are subjected to fusion processing, and target classification information corresponding to the classification level is predicted based on the fused fusion target classification feature map. The application can simultaneously meet the target detection requirements of different granularities and can improve the classification accuracy of coarse granularity.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a target detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Vision-based object detection methods can identify and locate objects in a scene, and have significant application value in fields such as human-computer interaction, intelligent transportation, and augmented reality. Currently, object detection mainly relies on binary or multi-class classification techniques, where each object category is equal and independent. However, the granularity requirements for object categories vary across different application scenarios. For example, when detecting a hand, some scenarios require detecting the hand's location, while others require identifying the gesture type. Existing solutions propose retraining the detection model or directly using a fine-grained detection model to meet different granularity requirements. However, retraining the detection model is time-consuming when modifying the model's output categories, and using a fine-grained detection model adds unnecessary computation when classifying fine-grained detection results. Therefore, there is an urgent need for an object detection model that can simultaneously meet different granularity requirements. Summary of the Invention

[0003] To address the issue that existing target detection models cannot meet the requirements of different granularities, this application provides a target detection method, apparatus, electronic device, and storage medium:

[0004] According to a first aspect of this application, a target detection method is provided, comprising:

[0005] The image to be processed is obtained and input into the target basic feature extraction network of the target detection model for feature extraction processing to obtain the target basic feature map.

[0006] The target basic feature map is input into the target localization branch network of the target detection model. The target localization branch network performs position prediction processing on the target basic feature map to obtain the target position information.

[0007] The target basic feature map is input into the target hierarchical classification branch network of the target detection model. The target hierarchical classification branch network performs feature extraction processing on the target basic feature map to obtain the target classification feature map corresponding to each classification level.

[0008] For each classification level, the target classification branch network is fused with the target classification feature map of the next lower classification level. Based on the fused target classification feature map, the target classification information of the classification level is predicted.

[0009] Among them, target location information and target classification information are used as target detection information.

[0010] Furthermore, the training steps for the object detection model include:

[0011] Obtain sample images and their annotation information; the annotation information includes the location information of the targets in the sample images and the annotation classification information at different classification levels;

[0012] The sample image is input into the basic feature extraction network of the model to be trained for feature extraction processing to obtain the basic feature map;

[0013] The basic feature map is input into the localization branch network of the model to be trained. The localization branch network performs position prediction processing on the basic feature map, and the localization loss is determined based on the predicted position information and the labeled position information.

[0014] The basic feature map is input into the hierarchical classification branch network of the model to be trained. The hierarchical classification branch network performs feature extraction processing on the basic feature map to obtain the classification feature map corresponding to each classification level.

[0015] For each classification branch network corresponding to each classification level, the classification feature map corresponding to the classification level is fused with the classification feature map corresponding to the next lower classification level, and the predicted classification information corresponding to the classification level is predicted based on the fused feature map.

[0016] Based on the predicted classification information and corresponding labeled classification information for each classification level, determine the classification loss for each classification level;

[0017] Backpropagation is performed based on the sum of the localization loss and the classification loss corresponding to each classification level. The model parameters of the model to be trained are adjusted until the preset training termination condition is met, and the training ends to obtain the object detection model. During the backpropagation process, the weights of the classification feature maps corresponding to each classification level are adjusted.

[0018] Furthermore, based on the localization loss and the classification loss corresponding to each classification level, the model parameters of the model to be trained are adjusted, including:

[0019] Based on the classification loss corresponding to each classification level, adjust the model parameters of the classification branch network corresponding to each classification level;

[0020] Based on the localization loss, adjust the model parameters of the localization branch network;

[0021] The model parameters of the basic feature extraction network are adjusted based on the sum of the localization loss and the classification loss corresponding to each classification level.

[0022] Furthermore, for the target classification branch network corresponding to each classification level, the target classification feature map corresponding to each classification level is fused with the target classification feature map corresponding to the next lower classification level. After predicting the target classification information corresponding to each classification level based on the fused target feature map, the process also includes:

[0023] The output of the target classification branch network corresponding to at least one classification level is suppressed to obtain suppressed target detection information.

[0024] Furthermore, the target basic feature map includes multi-scale target basic feature maps;

[0025] The target's basic feature map is input into the target localization branch network of the target detection model. The target localization branch network performs position prediction processing on the target's basic feature map to obtain the target's position information, including:

[0026] For any scale of the target basic feature map in the multi-scale target basic feature map, the target basic feature map of any scale is input into the target localization branch network. The target localization branch network performs position prediction processing on the target basic feature map of any scale to obtain the candidate position information corresponding to the target basic feature map of any scale.

[0027] Maximum suppression processing is performed on multiple candidate location information corresponding to the multi-scale target feature map to obtain the target location information.

[0028] According to a second aspect of this application, a target detection device is provided, comprising:

[0029] The first processing module is used to acquire the image to be processed, input the image to be processed into the target basic feature extraction network of the target detection model for feature extraction processing, and obtain the target basic feature map.

[0030] The second processing module is used to input the target basic feature map into the target localization branch network of the target detection model, and perform position prediction processing on the target basic feature map through the target localization branch network to obtain the target position information.

[0031] The third processing module is used to input the target basic feature map into the target hierarchical classification branch network of the target detection model, and to perform feature extraction processing on the target basic feature map through the target hierarchical classification branch network to obtain the target classification feature map corresponding to each classification level.

[0032] The first prediction module is used to fuse the target classification feature map corresponding to the classification level with the target classification feature map corresponding to the next lower classification level for each classification level target classification branch network, and predict the target classification information corresponding to the classification level based on the fused target classification feature map.

[0033] Among them, target location information and target classification information are used as target detection information.

[0034] Furthermore, it also includes:

[0035] The acquisition module is used to acquire sample images and their annotation information; the annotation information includes the location information of the targets in the sample images and the annotation classification information of different classification levels.

[0036] The first extraction module is used to input the sample image into the basic feature extraction network of the model to be trained for feature extraction processing to obtain the basic feature map;

[0037] The second prediction module is used to input the basic feature map into the localization branch network of the model to be trained, perform position prediction processing on the basic feature map through the localization branch network, and determine the localization loss based on the predicted position information and the labeled position information.

[0038] The second extraction module is used to input the basic feature map into the hierarchical classification branch network of the model to be trained, and to perform feature extraction processing on the basic feature map through the hierarchical classification branch network to obtain the classification feature map corresponding to each classification level.

[0039] The third prediction module is used to fuse the classification feature map corresponding to each classification level with the classification feature map corresponding to the next lower classification level for each classification branch network, and predict the predicted classification information corresponding to the classification level based on the fused feature map.

[0040] The determination module is used to determine the classification loss for each classification level based on the predicted classification information and the corresponding labeled classification information.

[0041] The adjustment module is used to perform backpropagation based on the localization loss and the classification loss corresponding to each classification level, and adjust the model parameters of the model to be trained until the preset training termination condition is met to end the training and obtain the object detection model; during the backpropagation process, the weights of the classification feature maps corresponding to each classification level are adjusted.

[0042] Furthermore, an adjustment module is used to adjust the model parameters of the classification branch network corresponding to each classification level based on the classification loss corresponding to each classification level.

[0043] Based on the localization loss, adjust the model parameters of the localization branch network;

[0044] The model parameters of the basic feature extraction network are adjusted based on the sum of the localization loss and the classification loss corresponding to each classification level.

[0045] Furthermore, it also includes:

[0046] The suppression module, used for each classification level's target classification branch network, fuses the target classification feature map of that level with the target classification feature map of the next lower level. After predicting the target classification information for that level based on the fused target classification feature map, it also includes:

[0047] The output of the target classification branch network corresponding to at least one classification level is suppressed to obtain suppressed target detection information.

[0048] Furthermore, the target basic feature map includes multi-scale target basic feature maps;

[0049] The second processing module is used to input any scale target basic feature map into the target localization branch network for any scale target basic feature map in the multi-scale target basic feature map, and perform position prediction processing on any scale target basic feature map through the target localization branch network to obtain the candidate position information corresponding to any scale target basic feature map.

[0050] Maximum suppression processing is performed on multiple candidate location information corresponding to the multi-scale target feature map to obtain the target location information.

[0051] According to a third aspect of this application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the target detection method of the first aspect of this application.

[0052] According to a fourth aspect of this application, a computer storage medium is provided, which stores at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the target detection method of the first aspect of this application.

[0053] According to a fifth aspect of this application, a computer program product is provided, comprising at least one instruction or at least one program segment, wherein the at least one instruction or at least one program segment is loaded and executed by a processor to implement the target detection method of the first aspect of this application.

[0054] This application provides a target detection method, apparatus, electronic device, and storage medium, which has the following technical effects: An image to be processed is acquired and input into the target basic feature extraction network of a target detection model for feature extraction processing to obtain a target basic feature map; the target basic feature map is input into the target localization branch network of the target detection model, and the target location information is obtained by the target localization branch network; the target location information is obtained by the target location branch network; the target hierarchical classification branch network of the target detection model is input into the target hierarchical classification branch network, and the target classification feature map is obtained by the target hierarchical classification branch network for feature extraction processing to obtain a target classification feature map corresponding to each classification level; for each classification level, the target classification feature map corresponding to the classification level is fused with the target classification feature map corresponding to the next lower classification level, and the target classification information corresponding to the classification level is predicted based on the fused target classification feature map; wherein, the target location information and the target classification information are used as target detection information. This application can simultaneously meet target detection requirements of different granularities by setting different levels of classification branch networks. Furthermore, by fusing the target classification feature map corresponding to the classification level with the target classification feature map corresponding to the next lower classification level, and predicting the target classification information corresponding to the classification level based on the fused target classification feature map, features of fine-grained classification tasks can be introduced into coarse-grained classification tasks, thereby improving the accuracy of coarse-grained classification. Attached Figure Description

[0055] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0057] Figure 2 This is a schematic flowchart of a target detection method provided in an embodiment of this application;

[0058] Figure 3 This is a flowchart illustrating a training method for an object detection model provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of a training method for an object detection model provided in an embodiment of this application;

[0060] Figure 5 This is a training diagram of a classification branch network provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application.

[0062] Figure 7 This is a schematic diagram of the hardware structure of an electronic device for implementing the target detection method provided in this application embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely one embodiment of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0064] The term "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of this application. In the description of the embodiments of this application, it should be understood that the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," and "third," etc., may explicitly or implicitly include one or more of that feature. Furthermore, the terms "first," "second," and "third," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "comprising," "having," and "being," and any variations thereof, are intended to cover non-exclusive inclusion.

[0065] It is understood that in the specific embodiments of this application, image data and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0066] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application. The application environment may include a data acquisition device 10 and a server 20. The data acquisition device 10 and the server 20 can be directly or indirectly connected via wired or wireless communication.

[0067] In some possible embodiments, the acquisition device 10 can send image data to the server 20. The server can provide target detection services, detecting the location and classification information of targets in the image data.

[0068] The acquisition device 10 may include at least one of the following hardware devices capable of capturing images: camera, webcam, video camera, etc.

[0069] Server 20 can be a standalone physical server, a service cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server may include network communication units, processors, and memory, etc. The server can provide image transmission services.

[0070] In some possible implementations, both the acquisition device 10 and the server 20 can be node devices in a blockchain system, capable of sharing the acquired and generated information with other node devices in the blockchain system, thus realizing information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which consists of multiple blocks, and adjacent blocks are related, ensuring that any data tampering in any block can be detected by the next block, thereby preventing data tampering in the blockchain and ensuring the security and reliability of the data in the blockchain.

[0071] The following describes a specific embodiment of a target detection method according to this application. Figure 2 This is a flowchart illustrating a target detection method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one of many execution orders and does not represent the only execution order. In actual execution, the methods can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0072] Specifically, such as Figure 2 As shown, the target detection method may include:

[0073] S201: Obtain the image to be processed, input the image to be processed into the target basic feature extraction network of the target detection model for feature extraction processing, and obtain the target basic feature map.

[0074] In this embodiment, the image to be processed can be acquired by an electronic device, or the electronic device can acquire the image to be processed from other devices, such as camera devices, monitoring devices, etc. In some implementations, the image to be processed can be a frame from a video.

[0075] In this embodiment, a target detection model can be constructed based on a convolutional neural network for target detection. The target detection model may include a feature encoding module based on downsampling, a feature decoding module based on multi-scale information fusion, and a detection module. In the detection module, the target localization branch network and the target classification branch network can be decoupled. The target localization branch network is responsible for detecting the position of the target in the image, while the target classification branch network is responsible for detecting the category to which the target belongs. To achieve multi-level category detection, i.e., multi-granularity category detection, the target classification branch network can be further decoupled, setting up target-level classification branch networks, i.e., classification branch networks with different granularities.

[0076] Figure 3 This is a flowchart illustrating a training method for an object detection model provided in an embodiment of this application. In practical applications, the object detection model can be obtained by training the model to be trained using the following steps:

[0077] S301: Obtain the sample image and its annotation information; the annotation information includes the annotation location information of the target in the sample image and the annotation classification information of different classification levels.

[0078] In this embodiment, sample images from a sample image set can be acquired by an electronic device, or the electronic device can acquire sample images from other devices, such as camera devices, monitoring devices, etc. In some implementations, a sample image may be a frame from a video.

[0079] In this embodiment, the sample image can carry annotation location information and annotation classification information for different classification levels. That is, the sample image can carry the coordinate data (x, y, w, h) of the target's annotation anchor box and the annotation category label of the target's membership level. (x, y) can represent the horizontal and vertical coordinates of the center point of the annotation anchor box, w can represent the width of the annotation anchor box, and h can represent the height of the annotation anchor box. The annotation location information and the annotation classification information for different classification levels can be manually annotated by technicians. The use of manual annotation is mainly to maximize the accuracy of image annotation, so that during the later training of the model, adverse consequences such as the trained model failing to meet requirements due to annotation errors can be avoided as much as possible.

[0080] S303: Input the sample image into the basic feature extraction network of the model to be trained for feature extraction processing to obtain the basic feature map.

[0081] In this embodiment, feature extraction processing can be performed on sample images in the basic feature extraction network of the model to be trained to obtain multiple feature maps of different scales. The feature maps of each scale are then fused with the feature maps of adjacent scales. The feature maps obtained by the fusion process are then convolved again, and the feature maps obtained by the convolution process are then stitched together to obtain a multi-scale basic feature map.

[0082] S305: Input the basic feature map into the localization branch network of the model to be trained, perform position prediction processing on the basic feature map through the localization branch network, and determine the localization loss based on the predicted position information and the labeled position information.

[0083] In this embodiment, for any scale of the multi-scale basic feature map, the basic feature map of any scale can be input into the localization branch network. The localization branch network performs position prediction processing on the basic feature map of any scale, and determines the localization loss based on the predicted position information and labeled position information obtained from the position prediction processing. Optionally, the basic feature map of any scale can be input into the localization branch network, and the localization branch network performs position prediction processing on the basic feature map of any scale to obtain the coordinate data (x',y',w',h') of the predicted anchor box of the target. Then, the localization loss can be determined according to the overlap rate corresponding to the coordinate data (x,y,w,h) of the labeled anchor box and the coordinate data (x',y',w',h') of the predicted anchor box. Here, the overlap rate is the ratio of the intersection to the union of the predicted anchor box and the standard anchor box in target detection, also known as the intersection over union (IOU). The ideal state is complete overlap, i.e., an overlap rate of 1.

[0084] S307: Input the basic feature map into the hierarchical classification branch network of the model to be trained, and perform feature extraction processing on the basic feature map through the hierarchical classification branch network to obtain the classification feature map corresponding to each classification level.

[0085] In this embodiment, for any scale basic feature map in the multi-scale basic feature map, the basic feature map of any scale is input into the hierarchical classification network branch, and the hierarchical classification network branch performs feature extraction processing on the basic feature map of any scale to obtain the classification feature map corresponding to each classification level.

[0086] S309: For each classification branch network corresponding to each classification level, the classification feature map corresponding to the classification level is fused with the classification feature map corresponding to the next lower classification level, and the predicted classification information corresponding to the classification level is predicted based on the fused feature map.

[0087] Generally, coarse-grained object classification focuses on extracting more abstract, high-level features of objects, ignoring differentiated detailed features, while fine-grained object classification focuses on differentiated detailed features. Therefore, introducing features from fine-grained classification tasks into coarse-grained classification tasks can improve the accuracy of coarse-grained classification, while introducing features from coarse-grained classification tasks into fine-grained classification tasks is detrimental to the accuracy of fine-grained classification. Based on these considerations, a gradient controller can be set up to fuse the classification feature maps corresponding to the next lower classification level with those corresponding to the next lower classification level. The fused feature map is then used to predict the classification information corresponding to the next lower classification level, enabling the features from fine-grained classification tasks to contribute to the classification accuracy of coarse-grained classification tasks while preventing the features from coarse-grained tasks from affecting the fine-grained classification tasks. By utilizing the mutual assistance between classification features of different granularities, the detection accuracy of object detection models can be improved.

[0088] S311: Based on the predicted classification information and corresponding labeled classification information for each classification level, determine the classification loss for each classification level.

[0089] In this embodiment, the classification loss for each classification level can be determined based on the difference between the confidence level of the predicted class label and the true value of the corresponding labeled class label at each granularity. For example, the true value of the labeled class label can be 1, the confidence level of the predicted class label can be 0.8, and the classification loss can be determined based on commonly used L1 loss functions (i.e., MAE, Mean Absolute Error), L2 loss functions (i.e., MSE, Mean Square Error), or the smooth_L1 loss function. Of course, other regression loss functions can also be used to determine the value.

[0090] S313: Backpropagation is performed based on the localization loss and the classification loss corresponding to each classification level to adjust the model parameters of the model to be trained until the preset training termination condition is met to end the training and obtain the object detection model; the weights of the classification feature maps corresponding to each classification level are adjusted during the backpropagation process.

[0091] In this embodiment, the model parameters of the classification branch network corresponding to each classification level can be adjusted based on the classification loss corresponding to each classification level, and the model parameters of the localization branch network can be adjusted based on the localization loss. Furthermore, the model parameters of the basic feature extraction network can be adjusted based on the sum of the localization loss and the classification loss corresponding to each classification level, until the loss meets a certain condition or the number of iterations meets a certain condition, at which point training stops, resulting in an object detection model composed of the target localization branch network and the target level classification branch network. During backpropagation, by adjusting the weights of the classification feature maps corresponding to each classification level, rather than adjusting the weights of the classification feature maps corresponding to each classification level and the classification feature maps corresponding to the higher-level classification level, the gradient flow of the target classification branch in the coarse-grained task cannot be propagated to the fine-grained task during backpropagation. The preset training termination condition can be that the number of iterations reaches a preset iteration threshold, the loss value is less than a preset loss threshold, or the difference between the loss values ​​of two adjacent iterations is less than a preset difference value. When the preset training termination condition is met, the training of the model to be trained ends, and the model to be trained corresponding to the model parameters at the end of training is taken as the object detection model.

[0092] S203: Input the target basic feature map into the target localization branch network of the target detection model, and perform position prediction processing on the target basic feature map through the target localization branch network to obtain the target position information.

[0093] In this embodiment, the target basic feature map may include multi-scale target basic feature maps. In practical applications, for any scale target basic feature map in the multi-scale target basic feature map, the target basic feature map of any scale can be input into the target localization branch network. The target localization branch network performs position prediction processing on the target basic feature map of any scale to obtain candidate position information corresponding to the target basic feature map of any scale. Then, maximum suppression processing is performed on the multiple candidate position information corresponding to the multi-scale target feature map to obtain the target position information.

[0094] S205: Input the target basic feature map into the target hierarchical classification branch network of the target detection model, and perform feature extraction processing on the target basic feature map through the target hierarchical classification branch network to obtain the target classification feature map corresponding to each classification level.

[0095] In this embodiment, the target basic feature map may include multi-scale target basic feature maps. In practical applications, for any scale target basic feature map in the multi-scale target basic feature map, the target basic feature map can be input into the target hierarchical classification branch network. The target hierarchical classification branch network performs feature extraction processing on the target basic feature map to obtain the target classification feature map corresponding to each classification level.

[0096] S207: For the target classification branch network corresponding to each classification level, the target classification feature map corresponding to the classification level is fused with the target classification feature map corresponding to the next lower classification level, and the target classification information corresponding to the classification level is predicted based on the fused target classification feature map; among them, the target location information and the target classification information are used as target detection information.

[0097] In this embodiment, after obtaining the target detection information, the output results at granularities that are not needed by the user can be suppressed. Optionally, the output of the target classification branch network corresponding to at least one classification level can be suppressed to obtain suppressed target detection information.

[0098] The target detection method provided in this application can simultaneously meet target detection requirements of different granularities by setting up classification branch networks at different levels. Furthermore, by fusing the target classification feature map corresponding to the classification level with the target classification feature map corresponding to the next lower classification level, and predicting the target classification information corresponding to the classification level based on the fused target classification feature map, features from fine-grained classification tasks can be introduced into coarse-grained classification tasks, thereby improving the accuracy of coarse-grained classification.

[0099] The following example illustrates the steps described above. Figure 4 This is a schematic diagram of a training method for an object detection model provided in an embodiment of this application. Sample images captured by a camera can be input into the feature encoding module based on downsampling and the feature decoding module based on multi-scale information fusion in the model to be trained to extract basic features, resulting in a basic feature map with N scales. For each basic feature map at scale n (n = 1, 2... N), the localization branch network in the detection module performs position prediction processing on the basic feature map at each scale n, obtaining candidate position information of the target in the basic feature map at each scale n. Simultaneously, the hierarchical classification branch network in the detection module performs category prediction processing on the basic feature map at each scale n, obtaining information on the category to which the target belongs at different granularities f (f = 1, 2... F) in the basic feature map at each scale n. Next, maximum suppression processing can be performed on the candidate position information corresponding to the basic feature maps at multiple scales to obtain predicted position information. To achieve multi-granularity category detection, the classification branch network can be further decoupled, setting classification branch networks of different granularities. Figure 5This is a training diagram of a classification branch network provided in an embodiment of this application. Solid lines represent forward propagation paths, and dashed lines represent backward propagation paths. For each scale n of the basic feature map, feature extraction processing can be performed on the basic feature map of that scale using classification branch networks of different granularities to obtain the classification feature map corresponding to each granularity of the classification branch network. Then, an adaptive feature adjustment module can adjust the weights of the classification branch networks of the corresponding granularities, introducing the classification features of the fine-grained classification branch network into the coarse-grained classification branch network. For example, the classification features of the classification branch network corresponding to granularity 2 and the classification features of the classification branch network corresponding to granularity 3 are introduced into the classification branch network corresponding to granularity 1, where granularity 1 is greater than granularity 2 and granularity 3. However, during the backward propagation stage, the classification features of the coarse-grained classification branch network are not passed to the fine-grained classification branch network. By introducing features of the fine-grained classification task into the coarse-grained classification task, the accuracy of the coarse-grained classification can be improved.

[0100] This application also provides a target detection device in its embodiments. Figure 6 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application, as shown below. Figure 6 As shown, the target detection device may include:

[0101] The first processing module 601 is used to acquire the image to be processed, input the image to be processed into the target basic feature extraction network of the target detection model for feature extraction processing, and obtain the target basic feature map.

[0102] The second processing module 603 is used to input the target basic feature map into the target localization branch network of the target detection model, and perform position prediction processing on the target basic feature map through the target localization branch network to obtain the target position information.

[0103] The third processing module 605 is used to input the target basic feature map into the target hierarchical classification branch network of the target detection model, and to perform feature extraction processing on the target basic feature map through the target hierarchical classification branch network to obtain the target classification feature map corresponding to each classification level.

[0104] The first prediction module 607 is used to fuse the target classification feature map corresponding to the classification level with the target classification feature map corresponding to the next lower classification level for each classification level target classification branch network, and predict the target classification information corresponding to the classification level based on the fused target classification feature map.

[0105] Among them, target location information and target classification information are used as target detection information.

[0106] In this embodiment of the application, the target detection device further includes:

[0107] The acquisition module is used to acquire sample images and their annotation information; the annotation information includes the location information of the targets in the sample images and the annotation classification information of different classification levels.

[0108] The first extraction module is used to input the sample image into the basic feature extraction network of the model to be trained for feature extraction processing to obtain the basic feature map;

[0109] The second prediction module is used to input the basic feature map into the localization branch network of the model to be trained, perform position prediction processing on the basic feature map through the localization branch network, and determine the localization loss based on the predicted position information and the labeled position information.

[0110] The second extraction module is used to input the basic feature map into the hierarchical classification branch network of the model to be trained, and to perform feature extraction processing on the basic feature map through the hierarchical classification branch network to obtain the classification feature map corresponding to each classification level.

[0111] The third prediction module is used to fuse the classification feature map corresponding to each classification level with the classification feature map corresponding to the next lower classification level for each classification branch network, and predict the predicted classification information corresponding to the classification level based on the fused feature map.

[0112] The determination module is used to determine the classification loss for each classification level based on the predicted classification information and the corresponding labeled classification information.

[0113] The adjustment module is used to adjust the model parameters of the model to be trained based on the localization loss and the classification loss corresponding to each classification level, until the preset training termination condition is met to end the training and obtain the object detection model.

[0114] In this embodiment of the application, the adjustment module is used to adjust the model parameters of the classification branch network corresponding to each classification level based on the classification loss corresponding to each classification level.

[0115] Based on the localization loss, adjust the model parameters of the localization branch network;

[0116] The model parameters of the basic feature extraction network are adjusted based on the sum of the localization loss and the classification loss corresponding to each classification level.

[0117] In this embodiment of the application, the target detection device further includes:

[0118] The suppression module, used for each classification level's target classification branch network, fuses the target classification feature map of that level with the target classification feature map of the next lower level. After predicting the target classification information for that level based on the fused target classification feature map, it also includes:

[0119] The output of the target classification branch network corresponding to at least one classification level is suppressed to obtain suppressed target detection information.

[0120] In this embodiment of the application, the target basic feature map includes a multi-scale target basic feature map;

[0121] The second processing module is used to input any scale target basic feature map into the target localization branch network for any scale target basic feature map in the multi-scale target basic feature map, and perform position prediction processing on any scale target basic feature map through the target localization branch network to obtain the candidate position information corresponding to any scale target basic feature map.

[0122] Maximum suppression processing is performed on multiple candidate location information corresponding to the multi-scale target feature map to obtain the target location information.

[0123] The apparatus and method embodiments in this application are based on the same application concept.

[0124] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the target detection method provided in the above method embodiments.

[0125] Figure 7 This is a schematic diagram of the hardware structure of an electronic device for implementing the target detection method provided in this application embodiment. The electronic device can participate in or include the target detection device provided in this application embodiment. Figure 7 As shown, the electronic device may include one or more (shown as 701a and 701b in the figure) processors 701 (processor 701 may include, but is not limited to, microprocessors 701 MCUs or programmable logic devices FPGAs, etc.), memory 703 for storing data, and transmission devices 705 for communication functions. In addition, it may also include: a display, input / output interfaces (I / O interfaces), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and / or a power supply. Those skilled in the art will understand that... Figure 7 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown.

[0126] It should be noted that the aforementioned one or more processors 701 and / or other data processing circuits are generally referred to as "data processing circuits" in this application. The data processing circuit can be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuit can be a single, independent processing module, or it can be integrated, in whole or in part, into any other element within an electronic device (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor 701 control (e.g., selection of a variable resistor termination path connected to an interface).

[0127] The memory 703 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the target detection method in this embodiment. The processor 701 implements the target detection method described above by running the software programs and modules stored in the memory 703 and executing various functional applications and data processing. The memory 703 may include high-speed random access memory, and may also include non-volatile random access memory 703, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory 703. In some possible embodiments, the memory 703 may further include remotely configured memories 703 relative to the processing, which can be connected to electronic devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0128] The transmission device 705 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the electronic device. In one example, the transmission device 705 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 705 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0129] The display can be, for example, a touchscreen liquid crystal display (LED), which allows users to interact with the user interface of an electronic device (or mobile device).

[0130] This application provides a computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a target detection method in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the target detection method provided in the above method embodiment.

[0131] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0132] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, while this specification describes specific embodiments, other embodiments are also within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in the order shown in different embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific order or sequence of connections to achieve the desired results; in some implementations, parallel processing of multiple tasks is possible or may be advantageous.

[0133] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, embodiments of apparatus and electronic devices are described simply because they are based on similar method embodiments; relevant parts can be referred to the descriptions of the method embodiments.

[0134] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A target detection method, characterized in that, include: The image to be processed is obtained and input into the target basic feature extraction network of the target detection model for feature extraction processing to obtain the target basic feature map; The target basic feature map is input into the target localization branch network of the target detection model, and the target localization branch network performs position prediction processing on the target basic feature map to obtain the target position information; The target basic feature map is input into the target hierarchical classification branch network of the target detection model. The target hierarchical classification branch network is used to perform feature extraction processing on the target basic feature map to obtain the target classification feature map corresponding to each classification level. For each classification level, the target classification branch network is fused with the target classification feature map of the next lower classification level, and the target classification information of the classification level is predicted based on the fused target classification feature map. The target location information and the target classification information are used as target detection information. Feature extraction is performed on the target basic feature map by classifying branch networks of different granularities to obtain the classification feature map corresponding to each granularity of the classification branch network. The weights of the classification branch networks of the corresponding granularities are adjusted by an adaptive feature adjustment module, and the classification features of the fine-grained classification branch network are introduced into the classification network of the coarse-grained classification branch network.

2. The method according to claim 1, characterized in that, The training steps of the object detection model include: Obtain sample images and their annotation information; the annotation information includes the annotation location information of targets in the sample images and the annotation classification information of different classification levels; The sample image is input into the basic feature extraction network of the model to be trained for feature extraction processing to obtain the basic feature map; The basic feature map is input into the localization branch network of the model to be trained, and the localization branch network performs position prediction processing on the basic feature map. The localization loss is determined based on the predicted position information and the labeled position information. The basic feature map is input into the hierarchical classification branch network of the model to be trained, and the basic feature map is processed by the hierarchical classification branch network to obtain the classification feature map corresponding to each classification level. For each classification branch network corresponding to each classification level, the classification feature map corresponding to the classification level is fused with the classification feature map corresponding to the next lower classification level, and the predicted classification information corresponding to the classification level is predicted based on the fused feature map. Based on the predicted classification information and corresponding labeled classification information corresponding to each classification level, the classification loss corresponding to each classification level is determined; Backpropagation is performed based on the localization loss and the classification loss corresponding to each classification level to adjust the model parameters of the model to be trained until the preset training termination condition is met to end the training and obtain the target detection model; the weights of the classification feature maps corresponding to each classification level are adjusted during the backpropagation process.

3. The method according to claim 2, characterized in that, The backpropagation based on the localization loss and the classification loss corresponding to each classification level, to adjust the model parameters of the model to be trained, includes: Based on the classification loss corresponding to each classification level, adjust the model parameters of the classification branch network corresponding to each classification level; Based on the localization loss, adjust the model parameters of the localization branch network; Based on the sum of the localization loss and the classification loss corresponding to each classification level, the model parameters of the basic feature extraction network are adjusted.

4. The method according to claim 1, characterized in that, The step of fusing the target classification feature map corresponding to each classification level with the target classification feature map corresponding to the next lower classification level, and predicting the target classification information corresponding to the classification level based on the fused target classification feature map, further includes: The output of the target classification branch network corresponding to at least one classification level is suppressed to obtain suppressed target detection information.

5. The method according to claim 1, characterized in that, The target basic feature map includes a multi-scale target basic feature map; The step of inputting the target basic feature map into the target localization branch network of the target detection model, and performing position prediction processing on the target basic feature map through the target localization branch network to obtain target position information includes: For any scale target basic feature map in the multi-scale target basic feature map, the target basic feature map of any scale is input into the target localization branch network, and the target localization branch network performs position prediction processing on the target basic feature map of any scale to obtain the candidate position information corresponding to the target basic feature map of any scale. The target location information is obtained by performing maximum suppression processing on multiple candidate location information corresponding to the multi-scale target basic feature map.

6. A target detection device, characterized in that, include: The first processing module is used to acquire the image to be processed, input the image to be processed into the target basic feature extraction network of the target detection model for feature extraction processing, and obtain the target basic feature map. The second processing module is used to input the target basic feature map into the target localization branch network of the target detection model, and perform position prediction processing on the target basic feature map through the target localization branch network to obtain target position information. The third processing module is used to input the target basic feature map into the target hierarchical classification branch network of the target detection model, and to perform feature extraction processing on the target basic feature map through the target hierarchical classification branch network to obtain the target classification feature map corresponding to each classification level. The first prediction module is used to fuse the target classification feature map corresponding to the classification level with the target classification feature map corresponding to the next lower classification level for each classification level target classification branch network, and predict the target classification information corresponding to the classification level based on the fused target classification feature map. The target location information and the target classification information are used as target detection information. Feature extraction is performed on the target basic feature map by classifying branch networks of different granularities to obtain the classification feature map corresponding to each granularity of the classification branch network. The weights of the classification branch networks of the corresponding granularities are adjusted by an adaptive feature adjustment module, and the classification features of the fine-grained classification branch network are introduced into the classification network of the coarse-grained classification branch network.

7. The apparatus according to claim 6, characterized in that, Also includes: The acquisition module is used to acquire sample images and the annotation information of the sample images; The annotation information includes the annotation location information of the target in the sample image and the annotation classification information of different classification levels; The first extraction module is used to input the sample image into the basic feature extraction network of the model to be trained for feature extraction processing to obtain a basic feature map. The second prediction module is used to input the basic feature map into the localization branch network of the model to be trained, perform position prediction processing on the basic feature map through the localization branch network, and determine the localization loss based on the predicted position information and the labeled position information. The second extraction module is used to input the basic feature map into the hierarchical classification branch network of the model to be trained, and to perform feature extraction processing on the basic feature map through the hierarchical classification branch network to obtain the classification feature map corresponding to each classification level. The third prediction module is used to fuse the classification feature map corresponding to the classification branch network corresponding to each classification level with the classification feature map corresponding to the next lower classification level, and predict the predicted classification information corresponding to the classification level based on the fused feature map. The determination module is used to determine the classification loss corresponding to each classification level based on the predicted classification information and the corresponding labeled classification information corresponding to each classification level; The adjustment module is used to adjust the model parameters of the model to be trained based on the localization loss and the classification loss corresponding to each classification level, until the preset training termination condition is met to end the training and obtain the target detection model.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the target detection method as described in any one of claims 1-5.

9. A computer storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the target detection method as described in any one of claims 1-5.

10. A computer program product, characterized in that, The computer program product includes at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the target detection method as described in any one of claims 1-5.

Images