Target detection method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses a target detection method and device, electronic equipment and a computer readable storage medium. The application can obtain a to-be-detected image, the to-be-detected image comprising at least one target object; performing feature extraction on the to-be-detected image to obtain a plurality of head features of different scales; performing branch feature extraction on the head features to obtain branch features corresponding to a plurality of branches, the branch features being features corresponding to target objects of different categories in the to-be-detected image; performing feature interaction processing on the branch features corresponding to the plurality of branches to obtain branch interaction features corresponding to the branches; and detecting the target objects in the to-be-detected image based on the branch interaction features corresponding to the branches. In the application, different branches are used to detect target objects of different categories in the to-be-detected image, and all objects of interest in the to-be-detected image can be detected. Therefore, the performance of target detection can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a target detection method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In recent years, research on object detection has encompassed various fields, such as security detection, autonomous driving, traffic detection, drone scene analysis, and robot vision. Object detection is a computer technology related to computer vision and image processing, used to detect objects of a certain category (such as people, buildings, or cars) in images or videos. For example, deep learning networks are used to extract features of objects from input images or videos, and these features are then used to classify objects and determine their locations within the images or videos.

[0003] Currently, object detection methods struggle to detect all objects of interest in an image or video, resulting in poor object detection performance. Summary of the Invention

[0004] This application provides a target detection method, apparatus, electronic device, and computer-readable storage medium that can improve the performance of target detection.

[0005] This application provides a target detection method, including:

[0006] Acquire an image to be detected, which includes at least one target object;

[0007] Feature extraction is performed on the image to be detected to obtain head features at multiple different scales;

[0008] Branch features are extracted from the head features to obtain branch features corresponding to multiple branches. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0009] Perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0010] The target object in the image to be detected is detected based on the branch interaction features corresponding to the branches.

[0011] This application also provides a target detection device, comprising:

[0012] An acquisition unit is used to acquire an image to be detected, wherein the image to be detected includes at least one target object;

[0013] The feature extraction unit is used to extract features from the image to be detected, and obtain head features at multiple different scales;

[0014] The branch feature extraction unit is used to extract branch features from the header features to obtain branch features corresponding to multiple branches. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0015] The interaction unit is used to perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0016] The detection unit is used to detect target objects in the image to be detected based on the branch interaction features corresponding to the branches.

[0017] In some embodiments, the interaction unit is specifically used for:

[0018] Attention-gathering interaction networks;

[0019] An attention interaction network is used to process the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to each branch.

[0020] In some embodiments, the attention interaction network includes an attention subnetwork and a feature extraction subnetwork, and the interaction unit is specifically used for:

[0021] The branch features corresponding to multiple branches are fused to obtain the branch fusion features;

[0022] A feature extraction subnetwork is used to extract features from the branch fusion features, resulting in multiple new branch features;

[0023] An attention subnetwork is used to adjust the attention of the new branch features to obtain the adjusted new branch features.

[0024] The adjusted new branch features and the corresponding branch features are subjected to residual processing to obtain the branch interaction features corresponding to the branches.

[0025] In some embodiments, the branch feature includes a first sub-branch feature and a second sub-branch feature, and the branch fusion feature includes a first sub-branch fusion feature and a second sub-branch fusion feature. The interaction unit is specifically used for:

[0026] The first sub-branch features corresponding to multiple branches are fused to obtain the first sub-branch fused features.

[0027] The features of the second sub-branch corresponding to multiple branches are fused to obtain the fused features of the second sub-branch.

[0028] In some embodiments, the target detection device is further configured to:

[0029] Obtain the output networks of the first and second sub-branches corresponding to the branch;

[0030] The branch interaction features corresponding to the branches include the branch interaction features corresponding to the first sub-branch feature and the branch interaction features corresponding to the second sub-branch feature. The detection unit is specifically used for:

[0031] The first sub-branch output network is used to perform convolution operation on the branch interaction features corresponding to the first sub-branch features to obtain the category and centrality of the pixels in the image to be detected.

[0032] The second sub-branch output network is used to perform convolution operation on the branch interaction features corresponding to the second sub-branch features to obtain the distance regression value of the pixel in the image to be detected.

[0033] The target object in the image is detected based on the category, centrality, and distance regression values ​​of the pixels in the image to be detected.

[0034] In some embodiments, the branch feature extraction unit is specifically used for:

[0035] Obtain the branch extraction network, and use the branch extraction network to extract branch features from the head features to obtain branch features corresponding to multiple branches;

[0036] The branches include a first branch and a second branch. The branch extraction network includes a first branch extraction network and a second branch extraction sub-network. The branch extraction network is used to extract branch features from the head features, resulting in branch features corresponding to multiple branches, including:

[0037] The first branch extraction network is used to extract branch features from the head features to obtain the first branch features corresponding to the first branch.

[0038] The second branch extraction network is used to extract branch features from the head features to obtain the second branch features corresponding to the second branch.

[0039] In some embodiments, the feature extraction unit is specifically used for:

[0040] Obtain the backbone network and feature pyramid network;

[0041] A backbone network is used to extract features from the image to be detected, resulting in multiple backbone features;

[0042] Feature pyramid network is used to extract features from multiple backbone features, resulting in head features of different scales.

[0043] In some embodiments, the target detection device is further configured to:

[0044] Multiple training sets and a preset object detection model are obtained. The multiple training sets correspond one-to-one with the multiple branches in the preset object detection model. The labels of the training images in the multiple training sets are different. The labels are used to distinguish different categories of target objects in the training images.

[0045] The target detection model is obtained by training the pre-defined target detection model using training images.

[0046] In some embodiments, each branch in the preset target detection model includes a preset first sub-branch output network and a preset second sub-branch output network, and the target detection device is further configured to:

[0047] A preset target detection model is used to detect target objects in the training images, and the first output result of the preset first sub-branch output network and the second output result of the preset second sub-branch output network are obtained for the branch.

[0048] Based on the first output result, the second output result corresponding to the branch, and the label of the training image, determine the first loss value and the second loss value corresponding to the branch;

[0049] Based on the first and second loss values ​​corresponding to the branch, determine the loss value corresponding to the branch;

[0050] Based on the loss value corresponding to the branch, determine the loss value of the preset target detection model;

[0051] Based on the loss value of the preset target detection model, the parameters of the preset target detection model are updated, and the process returns to the step of using the preset target detection model to detect target objects on the training image. The process is iterated until the loss value of the preset target detection model meets the preset conditions, and the target detection model is obtained.

[0052] In some embodiments, the target detection device is further configured to:

[0053] Determine the rate of change of loss corresponding to each branch;

[0054] Determine the loss weight corresponding to each branch based on the rate of change of loss for that branch.

[0055] Based on the loss weight and loss value corresponding to each branch, the loss value of the preset target detection model is determined.

[0056] In some embodiments, the target detection device is further configured to:

[0057] Determine the historical average loss value corresponding to the current iteration sliding window branch, and the historical average loss value of the target detection model preset in the current iteration sliding window. The current iteration sliding window represents the period from the nth iteration to the mth iteration.

[0058] Determine the historical average loss value corresponding to the previous iteration sliding window branch. The previous iteration sliding window represents the period from the njth iteration to the mjth iteration, where j, n, and m are all positive integers.

[0059] Based on the historical average loss value corresponding to the previous iteration sliding window branch, the historical average loss value corresponding to the current iteration sliding window branch, and the historical average loss value of the current iteration sliding window preset target detection model, the loss change rate corresponding to the branch is determined.

[0060] In some embodiments, the target detection device is further configured to:

[0061] Determine the historical average loss value of the target detection model in the current iteration sliding window;

[0062] Based on the rate of change of loss corresponding to each branch, determine the loss weight corresponding to that branch, including:

[0063] The loss magnitude corresponding to the branch is determined based on the historical average loss value corresponding to the current iteration sliding window branch and the historical average loss value of the current iteration sliding window preset target detection model.

[0064] The loss weight corresponding to each branch is determined based on the magnitude of the loss and the rate of change of the loss corresponding to each branch.

[0065] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the target detection methods provided in this application.

[0066] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the target detection methods provided in this application.

[0067] This application can acquire an image to be detected, which includes at least one target object; perform feature extraction on the image to be detected to obtain multiple header features at different scales; perform branch feature extraction on the header features to obtain multiple branch features corresponding to branches, where the branch features are features corresponding to different categories of target objects in the image to be detected; perform feature interaction processing on the branch features corresponding to multiple branches to obtain branch interaction features corresponding to the branches; and detect the target object in the image to be detected based on the branch interaction features corresponding to the branches.

[0068] In this application, after obtaining the initial features of the image to be detected, features of different categories of target objects are extracted through different branches, and feature interaction processing is performed on the features of different branches to discover potential information between different categories of target objects, which helps to detect all objects of interest in the image to be detected. This improves the performance of object detection. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1a This is a flowchart illustrating the target detection method provided in this application;

[0071] Figure 1b This is a schematic diagram of the target detection model provided in this application;

[0072] Figure 1c This is a schematic diagram of the structure of the attention interaction network provided in this application;

[0073] Figure 2a This is another flowchart illustrating the target detection method provided in this application;

[0074] Figure 2b This is a schematic diagram of the object detection visualization provided in this application;

[0075] Figure 2c This is a schematic diagram of the detection results of the target object provided in this application;

[0076] Figure 3 This is a schematic diagram of a target detection device provided in this application;

[0077] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0078] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] This application provides a target detection method, apparatus, electronic device, and computer-readable storage medium.

[0080] Specifically, the target detection device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server can also be implemented as a terminal.

[0081] In some embodiments, the target detection device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the target detection method of this application.

[0082] For example, the electronic device can acquire an image to be detected, which includes at least one target object; perform feature extraction on the image to be detected to obtain multiple header features at different scales; perform branch feature extraction on the header features to obtain multiple branch features corresponding to branches, where the branch features are features corresponding to different categories of target objects in the image to be detected; perform feature interaction processing on the branch features corresponding to multiple branches to obtain branch interaction features corresponding to the branches; and detect the target object in the image to be detected based on the branch interaction features corresponding to the branches.

[0083] In this embodiment, after obtaining the initial features of the image to be detected, the electronic device extracts features of different categories of target objects through different branches, and performs feature interaction processing on the features of different branches to discover potential information between different categories of target objects, which helps to detect all objects of interest in the image to be detected. This improves the performance of object detection.

[0084] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0085] Artificial intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use that knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. AI technology mainly includes several major areas such as computer vision, speech processing, natural language processing, machine learning, and deep learning.

[0086] Computer vision (CV) is a technology that uses computers to perform operations such as recognition, measurement, and further processing of target images, replacing the human eye. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, virtual reality, augmented reality, and simultaneous localization and mapping (SLAM), such as image processing techniques like image coloring and image outline extraction.

[0087] In this embodiment, a target detection method is provided, such as Figure 1a As shown, the specific process of this target detection method can be as follows:

[0088] 110. Obtain the image to be detected, which includes at least one target object.

[0089] The image to be detected can be any image, such as a frame from a video or an image from the test set of a public detection dataset. The size of the image is also not limited; for example, it can be 800×1024. In some embodiments, the image can be obtained in various ways. For example, it can be obtained from a database via a network, such as the public detection datasets COCO (Common Objects in Context) and PASCAL VOC (The PASCAL VisualObject Classes); it can be read from local memory; or it can be acquired in real-time using a camera, and so on.

[0090] The target object can be any object. For example, if the image to be detected is an image of a road collected by an autonomous vehicle, the target object can include, but is not limited to, cars, people, road signs, traffic light signs, bicycles, etc.; if the image to be detected is an image of a lawn, the target object can include, but is not limited to, people, trees, pets, etc.

[0091] 120. Perform feature extraction on the image to be detected to obtain head features at multiple different scales.

[0092] The features mentioned in this application can be data on a certain feature distribution on the image to be detected, and can be displayed as a feature map or a heat map.

[0093] In some embodiments, the electronic device may acquire a backbone network and feature pyramid networks (FPNs).

[0094] A backbone network is used to extract features from the image to be detected, resulting in multiple backbone features. The backbone network can be a network structure such as VGG (Visual Geometry Group), ResNet, or DenseNets. In some embodiments, the backbone network can be used to downsample the image to be detected, obtaining multiple backbone features; this downsampling can be maximum sampling or average sampling; and the multiple features can be two or more. For example, if the image to be detected is 800×1024, the backbone network can be used to downsample to obtain three backbone features, and the sizes of these backbone features can include 100×128, 50×64, and 25×32.

[0095] A feature pyramid network is used to extract features from multiple backbone features, resulting in multiple head features at different scales, also known as header features. These multiple head features can be two or more, for example, up to five different scales. Sampling through the feature pyramid network reduces the size of the feature maps. Feature maps obtained in the direction of decreasing size are called low-level features, and those obtained in the opposite direction are called high-level features. High-level features focus more on semantic information and less on detailed information, while low-level features contain more detailed information, thus allowing feature maps at each scale to have strong semantic information.

[0096] 130. Extract branch features from the head features to obtain branch features corresponding to multiple branches. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0097] Different branches can be used to detect different categories of target objects in the image to be detected.

[0098] Electronic devices can employ a branch extraction network to extract branch features from the initial features, resulting in branch features corresponding to multiple branches. This branch extraction network can be a convolutional neural network, for example, four convolutional layers with 256 channels. The number of branches corresponds to the categories of the target objects that can be detected; for example, if two categories of target objects can be detected, then the number of branches can be two. There are no restrictions on the classification rules for the target objects, and each category can include multiple subcategories. For example, during training, the categories of objects labeled in each training set can be considered as the same category; alternatively, categories can be divided based on classifications such as vehicles and traffic light signs.

[0099] In some embodiments, the branch includes a first branch and a second branch, and the branch extraction network includes a first branch extraction network and a second branch extraction sub-network. Therefore, using the branch extraction network to extract branch features from the header features to obtain branch features corresponding to multiple branches may include: using the first branch extraction network to extract branch features from the header features to obtain the first branch feature corresponding to the first branch; and using the second branch extraction network to extract branch features from the header features to obtain the second branch feature corresponding to the second branch.

[0100] In some embodiments, each branch may further include a first sub-branch and a second sub-branch. The branch extraction network may have two convolutional layers with 256 channels each, and each sub-branch corresponds to one convolutional layer with 256 channels. Therefore, for each branch:

[0101] The first sub-branch features are obtained by using the convolutional layer corresponding to the first sub-branch to extract branch features from the header features; the second sub-branch features are obtained by using the convolutional layer corresponding to the second sub-branch to extract branch features from the header features.

[0102] 140. Perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0103] Electronic devices can acquire attention interaction networks; these networks are used to process the branch features corresponding to multiple branches, resulting in branch interaction features. This embodiment, by adding attention interaction networks between different branches, provides cross-branch potential information for each branch, which helps discover potential relationships between different categories of target objects.

[0104] In some embodiments, the attention interaction network may include an attention subnetwork and a feature extraction subnetwork. The attention interaction network can be used to perform feature interaction processing on the branch features corresponding to multiple branches, including but not limited to:

[0105] i. Perform feature fusion processing on the branch features corresponding to multiple branches to obtain branch fusion features; this can fuse the latent semantic information between branch features. There are no restrictions on the fusion method; for example, branch fusion features can be obtained by concatenating multiple branch features.

[0106] ii. A feature extraction sub-network is used to extract features from the branch fusion features, resulting in multiple new branch features. The feature extraction sub-network can be two convolutional layers with a kernel of 3, meaning that two adaptive convolutional operations with a kernel of 3 are performed on the branch fusion features to generate new branch features suitable for each branch, containing latent semantic information.

[0107] iii. An attention sub-network is used to adjust the attention of the new branch features, resulting in the adjusted new branch features. The attention sub-network can be a Convolutional Block Attention Module (CBAM), etc. For example, CBAM can be used to adjust the attention of the new branch features. CBAM can include a channel attention module and a spatial attention module. This can be achieved by performing max pooling and global average pooling on each channel of the new branch features, then inputting each into a three-layer perceptron. The outputs of the perceptrons are summed and then input into a Rectified Linear Unit (ReLU) activation function to obtain the feature map of the channel attention module. Two feature maps, one with global max pooling and one with global average pooling, are calculated along the channel dimension. These two feature maps are concatenated, followed by another convolution operation. Finally, a sigmoid loss function is used to obtain the feature map of the spatial attention module. The feature maps of the channel attention module and the spatial attention module are then applied to the new branch features to adjust them. Attention subnetworks can focus more on important information in new branch features and suppress unimportant information, so that the adjusted new branch features contain key information about the relevant target object.

[0108] iv. Perform residual processing on the adjusted new branch features and the corresponding branch features to obtain the branch interaction features corresponding to the branches. Performing residual processing on the features output by the attention sub-network and the original branch features can further enhance the information of the relevant target objects.

[0109] In some embodiments, each branch may further include a first sub-branch and a second sub-branch, and the branch features corresponding to each branch include the first sub-branch features and the second sub-branch features. The corresponding branch fusion features also include the first sub-branch fusion features and the second sub-branch fusion features. Therefore, feature fusion processing can also be performed on the first sub-branch features corresponding to multiple branches to obtain the first sub-branch fusion features. Feature fusion processing can also be performed on the second sub-branch features corresponding to multiple branches to obtain the second sub-branch fusion features.

[0110] Then, for each branch: the corresponding feature extraction sub-network is used to extract features from the fused features of the first sub-branch to obtain multiple new first sub-branch features; the corresponding feature extraction sub-network is used to extract features from the fused features of the second sub-branch to obtain multiple new second sub-branch features.

[0111] The attention of the new first sub-branch features is adjusted using the corresponding attention sub-network to obtain the adjusted new first sub-branch features; the attention of the new second sub-branch features is adjusted using the corresponding attention sub-network to obtain the adjusted new second sub-branch features.

[0112] Perform residual processing on the adjusted new first sub-branch feature and the corresponding first sub-branch feature to obtain the branch interaction feature corresponding to the first sub-branch; perform residual processing on the adjusted new second sub-branch feature and the corresponding second sub-branch feature to obtain the branch interaction feature corresponding to the second sub-branch.

[0113] Optionally, the branch interaction features after residual processing can be normalized using a normalized subnetwork for subsequent object detection. For example, a 3×3 group-normalization convolution operation and a ReLU activation function can be used to refine the branch interaction features after residual processing for subsequent object detection.

[0114] Steps 130 and 140 are illustrated using only the branch corresponding to the header feature at a certain scale as an example. It is understood that steps 130 and 140 can also be performed on the branches corresponding to header features at other scales to obtain the branch interaction features corresponding to each header feature.

[0115] 150. Detect target objects in the image to be detected based on the branch interaction features corresponding to the branches.

[0116] In this embodiment, the output results of each branch corresponding to each header feature can be combined to detect the target object in the image to be detected. The output result corresponding to each branch can be the category, centrality, and distance regression value of the pixel in the image to be detected for each branch. There are no restrictions on the specific form of the detected target object. For example, the target object can be marked with a bounding box and its category can be indicated; or the target object can be marked with bounding boxes of different colors; or all pixels of the target object can be marked in the form of a heatmap, and so on.

[0117] For each branch: the output networks of the first and second sub-branches corresponding to the branch can be obtained; the first sub-branch output network is used to perform convolution operations on the branch interaction features corresponding to the first sub-branch features to obtain the class (Cls) and centrality (Ctr) of the pixels in the image to be detected; the second sub-branch output network is used to perform convolution operations on the branch interaction features corresponding to the second sub-branch features to obtain the distance regression value (Reg) of the pixels in the image to be detected. It can be understood that the class, centrality, and distance regression value of the pixels in the image to be detected can be obtained by establishing the relationship between the position of the pixels in the branch interaction features and the position of the pixels in the image to be detected. Assuming the target object is labeled with a bounding box, the class of a pixel can represent the class of the target object corresponding to that pixel; the centrality is the distance between each pixel and the center point of the target (the center point of the bounding box), which can be used to reduce predicted points that are far from the center point of the target (predicted points belonging to the target object); the distance regression value can be the distance of a pixel within the bounding box from the left, top, right, and bottom edges of the bounding box.

[0118] Based on the category, centrality, and distance regression values ​​of pixels in the image to be detected, target objects in the image can be detected. That is, by combining the category, centrality, and distance regression values ​​of pixels in the image to be detected for each branch corresponding to each header feature, all target objects of interest in the image to be detected can be detected.

[0119] In some embodiments, before acquiring the image to be detected, the electronic device may train a preset object detection model to obtain an object detection model. This object detection model may include, but is not limited to, all the networks involved in steps 110-150 above; that is, the object detection model may include a backbone network, a feature pyramid network, a branch extraction network, an attention interaction network, and branch output networks (a first sub-branch output network and a second sub-branch output network), etc. Figure 1b The diagram shown is a structural schematic of the target detection model provided in this embodiment. The cube in the accompanying drawings can represent features output by the network, such as... Figure 1b As shown, the first feature output of each convolutional layer in the Feature Pyramid Network is input into an attention feature interaction head (MIB-Head). This embodiment uses a two-branch configuration, referred to as the first branch and the second branch for ease of description. The first branch includes a first sub-branch and a second sub-branch. The attention feature interaction head can include a first branch extraction network, a second branch extraction network, an attention interaction network corresponding to the first sub-branch, and an attention interaction network corresponding to the second sub-branch, as well as a first branch output network and a second branch output network. Figure 1c As shown, Figure 1cThis is a schematic diagram of the attention interaction network provided in this application; the attention interaction network may include a feature extraction subnetwork, an attention subnetwork, and a normalization subnetwork.

[0120] The specific implementation method for training the preset target detection model is not limited, and may include, but is not limited to, the following steps:

[0121] S10. Obtain multiple training sets and preset object detection models.

[0122] The preset object detection model is an object detection model that has not yet been trained. Multiple training sets correspond one-to-one with multiple branches in the preset object detection model. The labels of the training images in the multiple training sets are different, and the labels are used to distinguish different categories of target objects in the training images. Training a branch to obtain the network can also be called a task; therefore, each branch corresponds to a task, and thus the tasks and training sets also correspond one-to-one.

[0123] In some embodiments, the number of training sets is 2 (i.e., the number of branches is 2). Since it is difficult to detect all objects of interest when labels are missing, the object detection model solves the problem of detecting all objects of interest when labels are missing. Therefore, in this embodiment, the dataset COCO can be split into training sets COCO_A and COCO_B, with COCO_A corresponding to the first branch and COCO_B corresponding to the second branch. The labels on the training images in COCO_A only indicate objects of the first category, but the training images include objects of the second category; the first category can include 32 subcategories. The labels on the training images in COCO_B only indicate objects of the second category, but include objects of the first category; the second category can include 48 subcategories. The objects corresponding to the first and second categories do not overlap.

[0124] S20. Train the preset target detection model using training images to obtain the target detection model.

[0125] In some embodiments, training a preset target detection model using training images may include, but is not limited to, the following steps:

[0126] 1) Use a preset target detection model to detect target objects in the training images, and obtain the first output result of the preset first sub-branch output network and the second output result of the preset second sub-branch output network corresponding to the branch.

[0127] In some embodiments, training parameters can be customized according to the actual application. For example, the batch size can be any integer; for instance, the batch size can be equal to 8, where 4 of the 8 training images can come from COCO_A and the other 4 from COCO_B. Typically, related target objects from different datasets will appear in the same training image, such as a television or a mouse.

[0128] A backbone network is used to extract features from the training images, resulting in backbone features for multiple training images. A feature pyramid network is then used to extract features from these backbone features, resulting in header features at different scales for the multiple training images. An attention feature interactor corresponding to each scale of header feature is then trained. This process is illustrated using an example of an attention feature interactor, which may include, but is not limited to:

[0129] For each branch in the attention feature interactor: the convolutional layer corresponding to the first sub-branch is used to extract branch features from the header features to obtain the corresponding first sub-branch features; the convolutional layer corresponding to the second sub-branch is used to extract branch features from the header features to obtain the corresponding second sub-branch features.

[0130] In the corresponding attention interaction network, feature fusion processing can be performed on the features of the first sub-branch corresponding to the two branches to obtain the first sub-branch fused features; feature fusion processing can be performed on the features of the second sub-branch corresponding to the two branches to obtain the second sub-branch fused features. Thus, the latent semantic information between features of different target objects in different datasets can be fused.

[0131] In the corresponding attention interaction network, a corresponding feature extraction subnetwork can be used to extract features from the fused features of the first sub-branch, resulting in new first sub-branch features corresponding to the first branch and new first sub-branch features corresponding to the second branch; similarly, a corresponding feature extraction subnetwork can be used to extract features from the fused features of the second sub-branch, resulting in new second sub-branch features corresponding to the first branch and new second sub-branch features corresponding to the second branch. This can generate new branch features suitable for each branch, containing latent semantic information.

[0132] Then, for each branch: the corresponding attention subnetwork can be used to adjust the attention of the new first subbranch features, resulting in adjusted new first subbranch features; the corresponding attention subnetwork can be used to adjust the attention of the new second subbranch features, resulting in adjusted new second subbranch features. The attention subnetwork can focus more on important information in the new branch features and suppress unimportant information, so that the adjusted new branch features contain key information about the relevant target object.

[0133] Residual processing is performed on the adjusted new first sub-branch features and their corresponding first sub-branch features to obtain the branch interaction features corresponding to the first sub-branch; residual processing is also performed on the adjusted new second sub-branch features and their corresponding second sub-branch features to obtain the branch interaction features corresponding to the second sub-branch. This approach can enhance features with relevant semantic information while suppressing easily confused features.

[0134] In some embodiments, branch interaction features can be filtered based on the identifiers (e.g., image IDs) of the training images in COCO_A and COCO_B. After filtering the branch interaction features corresponding to each sub-branch, the features of the corresponding training images in the dataset are retained, so that the branch corresponding to each dataset only calculates the loss of the training images with the true labels (ground truth). For example, the first sub-branch and the second sub-branch of the first branch only include the branch interaction features corresponding to the training images in COCO_A. Figure 1c As shown, the features input to the attention interaction network are the features corresponding to 2N training images, where 2N is the batch size, H and W represent the feature size, and C represents the channel. After filtering, the branch interaction features corresponding to the first branch are the features corresponding to the N training images in COCO_A, and the branch interaction features corresponding to the second branch are the features corresponding to the N training images in COCO_B.

[0135] The branch interaction features corresponding to the first sub-branch after screening are normalized using a normalized sub-network to obtain the normalized branch interaction features corresponding to the first sub-branch; the branch interaction features corresponding to the second sub-branch after screening are normalized using a normalized sub-network to obtain the normalized branch interaction features corresponding to the second sub-branch.

[0136] Finally, based on the branch interaction features corresponding to the first sub-branch and the branch interaction features corresponding to the second sub-branch, the first output result of the preset first sub-branch output network and the second output result of the preset second sub-branch output network corresponding to each branch are obtained.

[0137] 2) Determine the first loss value and the second loss value corresponding to the branch based on the first output result, the second output result, and the label of the training image.

[0138] In some embodiments, the loss value can be calculated based on the loss function.

[0139] 3) Determine the loss value corresponding to the branch based on the first and second loss values ​​corresponding to the branch.

[0140] In some embodiments, the first and second loss values ​​of each branch can be added together to obtain the loss value corresponding to each branch. Then, the loss values ​​corresponding to the first branches in all attention feature interactors can be added together to obtain the loss values ​​corresponding to all first branches; and the loss values ​​corresponding to the second branches in all attention feature interactors can be added together to obtain the loss value of the second branch.

[0141] In some embodiments, a preset weight can be set for each attention feature interactor, and a weighted sum can be performed according to the preset weight when determining the loss value of the first branch and the loss value of the second branch.

[0142] 4) Determine the loss value of the preset target detection model based on the loss value corresponding to the branch.

[0143] This embodiment proposes a dynamic loss balancing strategy based on an iterative sliding window. This strategy considers both the rate of change of loss and the magnitude of loss within a given number of iterations. If the loss value of a branch decreases rapidly (i.e., the rate of change of loss is smaller), or if the loss value of a branch is smaller, it indicates that the branch has learned well during training. Therefore, a smaller loss weight can be assigned to that branch, shifting the focus of the entire object detection model's learning towards the more difficult branches. This balances the weight of the loss value across each branch, resulting in more efficient and stable backpropagation performance on each branch.

[0144] In some embodiments, determining the rate of change of loss corresponding to a branch includes determining the historical average loss value corresponding to the branch of the current iteration sliding window, and the historical average loss value of the preset target detection model of the current iteration sliding window, where the current iteration sliding window represents the iteration from the nth iteration to the mth iteration. The historical average loss value corresponding to the branch of the previous iteration sliding window is determined, where the previous iteration sliding window represents the iteration from the njth iteration to the mjth iteration; j can represent the sliding step size, and the value of mn can be called the window size of the iteration sliding window. j, n, and m are all positive integers, and their values ​​can be customized according to the actual application. Based on the historical average loss value corresponding to the branch of the previous iteration sliding window, the historical average loss value corresponding to the branch of the current iteration sliding window, and the historical average loss value of the preset target detection model of the current iteration sliding window, the rate of change of loss corresponding to the branch is determined.

[0145] In some embodiments, the historical average loss value of the preset target detection model of the current iterative sliding window can also be determined; the loss magnitude corresponding to the branch can be determined based on the historical average loss value corresponding to the branch of the current iterative sliding window and the historical average loss value of the preset target detection model of the current iterative sliding window; and the loss weight corresponding to the branch can be determined based on the loss magnitude corresponding to the branch and the loss change rate corresponding to the branch.

[0146] For example, the parameters of the loss weights can be calculated using the following formula:

[0147]

[0148] in, Indicates the first During the next iteration The parameters of the loss weights corresponding to the branches; This indicates the tasks corresponding to different branches; Indicates the number of iterations. It equals m+1; Indicates the current iteration sliding window. This represents the previous iteration of the sliding window; This represents the historical average loss value corresponding to the sliding window branch in the previous iteration; This represents the historical average loss value corresponding to the current iteration sliding window branch; This represents the historical average loss value of the target detection model preset in the current iteration sliding window;

[0149] Indicates the first During the next iteration The rate of change of loss corresponding to the branch; Indicates the first During the next iteration The magnitude of the loss corresponding to the branch.

[0150] The loss weight for each branch is determined based on its rate of change and magnitude (i.e., the parameters of the loss weight). For example, the loss weight for each branch can be calculated using the following formula:

[0151]

[0152] in, Indicates the total number of tasks (total number of branches); This indicates the temperature for adjusting task weight flexibility, which can be set according to the actual application. The larger the value, the closer the loss weights of each branch become.

[0153] The loss value of the preset target detection model is determined based on the loss weights and loss values ​​corresponding to the branches. In some embodiments, the loss values ​​corresponding to the branches can be weighted and summed to obtain the loss value of the preset target detection model.

[0154] 5) Update the parameters of the preset object detection model based on its loss value, and return to the step of using the preset object detection model to detect objects on the training image. Iterate until the loss value of the preset object detection model satisfies a preset condition, thus obtaining the object detection model. The preset condition can be that the preset object detection model has converged.

[0155] The object detection scheme provided in this application can be applied to various computer vision scenarios. For example, it can be used in real-life detection scenarios, and so on.

[0156] As shown above, this application can obtain multiple training sets, each corresponding one-to-one with a branch in a pre-defined object detection model. The labels on the training images in each training set are different, and these labels are used to distinguish different categories of target objects in the training images. When training the pre-defined object detection model using these training images, attention feature interactors are added between different branches to provide cross-branch potential information for each branch, helping to discover potential relationships between target objects from different datasets. Then, a dynamic loss balancing strategy based on an iterative sliding window is proposed. By simultaneously considering the rate of change of loss and the magnitude of loss within a certain number of iterations, more efficient and stable backpropagation performance is achieved on each branch. Finally, the trained object detection network is used to perform object detection on the image to be detected, detecting all objects of interest in the image, thereby improving the performance of object detection.

[0157] The target detection method described in the above embodiments will be further described in detail below.

[0158] like Figure 2a As shown, the specific process of a target detection method is as follows:

[0159] 210. Obtain multiple training sets and preset object detection models.

[0160] The dataset is a subset generated based on COCO and PASCAL VOC. The partitioning of the training and test sets is shown in Table 1.

[0161] Table 1: Dataset splitting results on COCO and PASCAL VOC.

[0162]

[0163] The VOC training set is partitioned from the training and validation sets of VOC2007 and VOC2012. In VOC_A, the training images are labeled with 17 subcategories, while in VOC_B, the training images are labeled with 3 subcategories. The test set is based on the VOC2007 test set. COCO is based on the COCO2017 dataset. In COCO_A, the training images are labeled only with the first category of objects, but the training images include objects from the second category; the first category can include 32 subcategories. In COCO_B, the training images are labeled only with the second category of objects, but the first category of objects is included; the second category can include 48 subcategories.

[0164] 220. Train the preset target detection model using training images to obtain the target detection model.

[0165] The backbone network is ResNet50. For the dynamic loss equalization strategy based on iterative sliding windows, the window size and stride are set to 15,5 during training on the COCO dataset; and to 20,1 during training on the VOC dataset. The specific implementation of this step can be found in step S20, and will not be elaborated here.

[0166] 230. Obtain the image to be detected, which includes at least one target object.

[0167] The image to be detected can be an image from the test set in step 210.

[0168] 240. The backbone network and feature pyramid network in the target detection model are used to extract features from the image to be detected, and multiple head features at different scales are obtained.

[0169] 250. The branch extraction network in the target detection model is used to extract branch features from the head features, and the branch features corresponding to multiple branches are obtained. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0170] 260. The attention interaction network in the object detection model is used to perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0171] 270. Detect target objects in the image to be detected based on the branch interaction features corresponding to the branches.

[0172] Steps 230 to 270 can be referred to as described in steps 110 to 150, and will not be repeated here. After detecting the target objects in the images to be detected in the test set, the detection results can be calculated and compared with common solutions, as shown in Table 2. "Single" means training each training set with a single model. "Brutal" means simply combining multiple training sets together for training. "MTB" means applying a shared backbone network and a multi-branch network with different detection branches for different datasets for training.

[0173] The results show that among these common solutions, the Single and MTB models perform best, while the brutal method performs worst. This demonstrates that the semantic conflict problem in object detection task fusion is fundamentally different from that in single-label image classification task fusion. For object detection, an image may contain many objects of interest, while for single-label image classification, there is usually only one object of interest. Without complete annotation of all objects in the training images, object detection models can easily identify unlabeled objects of interest as background. This leaves significant confusion for object detection models, as predictions may be correct in some images but incorrect in others. Compared to the single model, the proposed solution improves the mean average precision (mAP) by 2% on the PASCAL-VOC2007 test set, demonstrating a significant improvement in object detection with incomplete annotation.

[0174] Simultaneously, an ablation study was conducted using dynamic loss equalization strategies based on iterative sliding windows (MDWAs) and attention feature interaction mechanisms (MIB-Head) to evaluate the effectiveness of each module. The results, shown in Table 2, demonstrate that each module played its corresponding role without hindering the positive effects of other modules.

[0175] Table 2: Comparison of mAP results with other commonly used methods for the divided COCO and VOC datasets, and ablation experimental results of the proposed methods.

[0176]

[0177] The proposed Dynamic Loss Equalization (MDWA) strategy based on iterative sliding windows is compared with the baseline MTB-based DWA method. The results are shown in Table 3. The effectiveness of the proposed method in balancing multiple object detection tasks is demonstrated on different datasets. Compared with DWA, the proposed method has a positive impact and is significantly better than DWA, especially on COCO_A.

[0178] Table 3: Results of the dynamic loss equalization strategy based on iterative sliding window compared with DWA and baseline MTB.

[0179]

[0180] The role of iterative sliding windows in balancing branch losses was investigated using multiple combinations of different window sizes and step sizes, and the results are shown in Table 4. Among all parameter settings for the iterative sliding window, COCO_A shows more improvements compared to COCO_B. Furthermore, it can be observed that even without an iterative sliding window, where both the size and step size are 1, the method in this application is more efficient than DWA, meaning that considering the magnitude of the loss is very useful. COCO performs best when the window size is 10 and the step size is 3.

[0181] Table 4: Results of the dynamic loss equalization strategy based on iterative sliding window under different window sizes and step sizes.

[0182]

[0183] The proposed attention feature interactor is compared with CBAM, Neural Discriminative Dimensionality Reduction (NDDR), and ResAtt modules to evaluate its performance. The results are shown in Table 5. The proposed method achieves significant improvements, increasing mAP on COCO partitions by 1.7% and 1.1% compared to MTB, respectively.

[0184] Table 5: Comparison of the effects of attention-featured interactors with other methods

[0185]

[0186] To illustrate the role of the attention feature interactor in mining semantic information of relevant target objects, the mAP results of some relevant objects detected on different subtasks are shown in Table 6. It can be observed that compared with the single model and MTB, MIB-Head significantly improves performance on each relevant object (e.g., TV, Mouse) or obfuscated object (e.g., Cat, Dog).

[0187] To further observe the impact of the attention feature interactor on relevant or confused objects, the object detection of some examples is visualized. Figure 2bIn the first row, a laptop was mistaken for a mouse, a cat for a dog, and a television for a television. Conversely, the attention-based feature interactor proposed in this application effectively avoids these false positives and false negatives, as shown in the second row. On the heatmap, the areas of the falsely detected laptop and cat have lower heat values, while the falsely detected television has higher heat values.

[0188] Table 6: Research Results of Attention Feature Interactor, Single Model, and MTB Network

[0189]

[0190] To accurately observe how the object detection scheme proposed in this application works, the detection results of target objects from some images in the COCO test set are visualized, such as... Figure 2c As shown. Generally, it is easy to see that a single model can only detect the category of the target object it is interested in, while the target detection model of this application can locate all target objects of interest. Furthermore, it is worth noting that the scheme of this application successfully reduces the number of missed targets or mispredictions.

[0191] As shown above, this application can obtain multiple training sets, each corresponding one-to-one with a branch in a pre-defined object detection model. The labels on the training images in each training set are different, and these labels are used to distinguish different categories of target objects in the training images. When training the pre-defined object detection model using these training images, attention feature interactors are added between different branches to provide cross-branch potential information for each branch, helping to discover potential relationships between target objects from different datasets. Then, a dynamic loss balancing strategy based on an iterative sliding window is proposed. By simultaneously considering the rate of change of loss and the magnitude of loss within a certain number of iterations, more efficient and stable backpropagation performance is achieved on each branch. Finally, the trained object detection network is used to perform object detection on the image to be detected, detecting all objects of interest in the image, thereby improving the performance of object detection.

[0192] To better implement the above methods, this application also provides a target detection device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0193] For example, in this embodiment, the method of this application will be described in detail by taking the target detection device as specifically integrated into an electronic device.

[0194] For example, such as Figure 3As shown, the target detection device may include an acquisition unit 301, a feature extraction unit 302, a branch feature extraction unit 303, an interaction unit 304, and a detection unit 305, as follows:

[0195] (a) Acquisition Unit 301

[0196] The acquisition unit 301 is used to acquire an image to be detected, which includes at least one target object.

[0197] (ii) Feature extraction unit 302

[0198] The feature extraction unit 302 is used to extract features from the image to be detected, and obtain head features of multiple different scales.

[0199] (III) Branch Feature Extraction Unit 303

[0200] The branch feature extraction unit 303 is used to extract branch features from the head features to obtain branch features corresponding to multiple branches. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0201] (4) Interactive Unit 304

[0202] The interaction unit 304 is used to perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0203] (5) Detection unit 305

[0204] The detection unit 305 is used to detect the target object in the image to be detected based on the branch interaction features corresponding to the branch.

[0205] In some embodiments, the interaction unit 304 is specifically used for:

[0206] Attention-gathering interaction networks;

[0207] An attention interaction network is used to process the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to each branch.

[0208] In some embodiments, the attention interaction network includes an attention subnetwork and a feature extraction subnetwork, and the interaction unit 304 is specifically used for:

[0209] The branch features corresponding to multiple branches are fused to obtain the branch fusion features;

[0210] A feature extraction subnetwork is used to extract features from the branch fusion features, resulting in multiple new branch features;

[0211] An attention subnetwork is used to adjust the attention of the new branch features to obtain the adjusted new branch features.

[0212] The adjusted new branch features and the corresponding branch features are subjected to residual processing to obtain the branch interaction features corresponding to the branches.

[0213] In some embodiments, the branch feature includes a first sub-branch feature and a second sub-branch feature, and the branch fusion feature includes a first sub-branch fusion feature and a second sub-branch fusion feature. The interaction unit 304 is specifically used for:

[0214] The first sub-branch features corresponding to multiple branches are fused to obtain the first sub-branch fused features.

[0215] The features of the second sub-branch corresponding to multiple branches are fused to obtain the fused features of the second sub-branch.

[0216] In some embodiments, the target detection device is further configured to:

[0217] Obtain the output networks of the first and second sub-branches corresponding to the branch;

[0218] The branch interaction features corresponding to the branches include the branch interaction features corresponding to the first sub-branch feature and the branch interaction features corresponding to the second sub-branch feature. The detection unit is specifically used for:

[0219] The first sub-branch output network is used to perform convolution operation on the branch interaction features corresponding to the first sub-branch features to obtain the category and centrality of the pixels in the image to be detected.

[0220] The second sub-branch output network is used to perform convolution operation on the branch interaction features corresponding to the second sub-branch features to obtain the distance regression value of the pixel in the image to be detected.

[0221] The target object in the image is detected based on the category, centrality, and distance regression values ​​of the pixels in the image to be detected.

[0222] In some embodiments, the branch feature extraction unit 303 is specifically used for:

[0223] Obtain the branch extraction network, and use the branch extraction network to extract branch features from the head features to obtain branch features corresponding to multiple branches;

[0224] The branches include a first branch and a second branch. The branch extraction network includes a first branch extraction network and a second branch extraction sub-network. The branch extraction network is used to extract branch features from the head features, resulting in branch features corresponding to multiple branches, including:

[0225] The first branch extraction network is used to extract branch features from the head features to obtain the first branch features corresponding to the first branch.

[0226] The second branch extraction network is used to extract branch features from the head features to obtain the second branch features corresponding to the second branch.

[0227] In some embodiments, the feature extraction unit 302 is specifically used for:

[0228] Obtain the backbone network and feature pyramid network;

[0229] A backbone network is used to extract features from the image to be detected, resulting in multiple backbone features;

[0230] Feature pyramid network is used to extract features from multiple backbone features, resulting in head features of different scales.

[0231] In some embodiments, the target detection device is further configured to:

[0232] Multiple training sets and a preset object detection model are obtained. The multiple training sets correspond one-to-one with the multiple branches in the preset object detection model. The labels of the training images in the multiple training sets are different. The labels are used to distinguish different categories of target objects in the training images.

[0233] The target detection model is obtained by training the pre-defined target detection model using training images.

[0234] In some embodiments, each branch in the preset target detection model includes a preset first sub-branch output network and a preset second sub-branch output network, and the target detection device is further configured to:

[0235] A preset target detection model is used to detect target objects in the training images, and the first output result of the preset first sub-branch output network and the second output result of the preset second sub-branch output network are obtained for the branch.

[0236] Based on the first output result, the second output result corresponding to the branch, and the label of the training image, determine the first loss value and the second loss value corresponding to the branch;

[0237] Based on the first and second loss values ​​corresponding to the branch, determine the loss value corresponding to the branch;

[0238] Based on the loss value corresponding to the branch, determine the loss value of the preset target detection model;

[0239] Based on the loss value of the preset target detection model, the parameters of the preset target detection model are updated, and the process returns to the step of using the preset target detection model to detect target objects on the training image. The process is iterated until the loss value of the preset target detection model meets the preset conditions, and the target detection model is obtained.

[0240] In some embodiments, the target detection device is further configured to:

[0241] Determine the rate of change of loss corresponding to each branch;

[0242] Determine the loss weight corresponding to each branch based on the rate of change of loss for that branch.

[0243] Based on the loss weight and loss value corresponding to each branch, the loss value of the preset target detection model is determined.

[0244] In some embodiments, the target detection device is further configured to:

[0245] Determine the historical average loss value corresponding to the current iteration sliding window branch, and the historical average loss value of the target detection model preset in the current iteration sliding window. The current iteration sliding window represents the period from the nth iteration to the mth iteration.

[0246] Determine the historical average loss value corresponding to the previous iteration sliding window branch. The previous iteration sliding window represents the period from the njth iteration to the mjth iteration, where j, n, and m are all positive integers.

[0247] Based on the historical average loss value corresponding to the previous iteration sliding window branch, the historical average loss value corresponding to the current iteration sliding window branch, and the historical average loss value of the current iteration sliding window preset target detection model, the loss change rate corresponding to the branch is determined.

[0248] In some embodiments, the target detection device is further configured to:

[0249] The current iterative sliding window presets the historical average loss value of the target detection model;

[0250] Based on the rate of change of loss corresponding to each branch, determine the loss weight corresponding to that branch, including:

[0251] The loss magnitude corresponding to the branch is determined based on the historical average loss value corresponding to the current iteration sliding window branch and the historical average loss value of the current iteration sliding window preset target detection model.

[0252] The loss weight corresponding to each branch is determined based on the magnitude of the loss and the rate of change of the loss corresponding to each branch.

[0253] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0254] As described above, the target detection device in this embodiment can acquire an image to be detected, which includes at least one target object; perform feature extraction on the image to be detected to obtain multiple header features at different scales; extract branch features from the header features to obtain multiple branch features corresponding to different categories of target objects in the image to be detected; perform feature interaction processing on the branch features corresponding to multiple branches to obtain branch interaction features corresponding to the branches; and detect the target object in the image to be detected based on the branch interaction features corresponding to the branches. After obtaining the header features of the image to be detected, features of different categories of target objects are extracted through different branches, and feature interaction processing is performed on the features of different branches to discover potential information between different categories of target objects, which helps to detect all objects of interest in the image to be detected. This improves the performance of target detection.

[0255] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0256] In some embodiments, the target detection device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the target detection method of this application.

[0257] In this embodiment, the electronic device will be used as an example for detailed description, such as... Figure 4 As shown, it illustrates the structural diagram of the electronic device involved in this application, specifically:

[0258] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0259] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0260] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0261] The electronic device also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0262] The electronic device may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0263] The electronic device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0264] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0265] Acquire an image to be detected, which includes at least one target object;

[0266] Feature extraction is performed on the image to be detected to obtain head features at multiple different scales;

[0267] Branch features are extracted from the head features to obtain branch features corresponding to multiple branches. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0268] Perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0269] The target object in the image to be detected is detected based on the branch interaction features corresponding to the branches.

[0270] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0271] As shown above, the electronic device can acquire an image to be detected, which includes at least one target object; perform feature extraction on the image to be detected to obtain multiple header features at different scales; extract branch features from the header features to obtain branch features corresponding to multiple branches, which are features corresponding to different categories of target objects in the image to be detected; perform feature interaction processing on the branch features corresponding to multiple branches to obtain branch interaction features corresponding to the branches; and detect the target objects in the image to be detected based on the branch interaction features corresponding to the branches. After obtaining the header features of the image to be detected, features of different categories of target objects are extracted through different branches, and feature interaction processing is performed on the features of different branches to discover potential information between different categories of target objects, which helps to detect all objects of interest in the image to be detected. This improves the performance of object detection.

[0272] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0273] Therefore, this application provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the target detection methods provided in this application. For example, the instructions can execute the following steps:

[0274] Acquire an image to be detected, which includes at least one target object;

[0275] Feature extraction is performed on the image to be detected to obtain head features at multiple different scales;

[0276] Branch features are extracted from the head features to obtain branch features corresponding to multiple branches. The branch features are the features corresponding to different categories of target objects in the image to be detected.

[0277] Perform feature interaction processing on the branch features corresponding to multiple branches to obtain the branch interaction features corresponding to the branches.

[0278] The target object in the image to be detected is detected based on the branch interaction features corresponding to the branches.

[0279] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0280] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the computer vision or detection aspects provided in the above embodiments.

[0281] Since the instructions stored in the storage medium can execute the steps of any of the target detection methods provided in this application, the beneficial effects that any of the target detection methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0282] The above provides a detailed description of a target detection method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A target detection method characterized by, The method comprises the following steps: acquiring a to-be-detected image, wherein the to-be-detected image comprises at least one target object; performing feature extraction on the to-be-detected image to obtain a plurality of head features of different scales; performing branch feature extraction on the head features to obtain branch features corresponding to a plurality of branches, wherein the branch features are features corresponding to target objects of different categories in the to-be-detected image; performing feature interaction processing on the branch features corresponding to the plurality of branches to obtain branch interaction features corresponding to the branches, comprising: acquiring an attention interaction network, wherein the attention interaction network comprises an attention subnetwork and a feature extraction subnetwork; performing feature fusion processing on the branch features corresponding to the plurality of branches to obtain branch fusion features; performing feature extraction on the branch fusion features by using the feature extraction subnetwork to obtain a plurality of new branch features; performing attention adjustment on the new branch features by using the attention subnetwork to obtain adjusted new branch features; and performing residual processing on the adjusted new branch features and the corresponding branch features to obtain the branch interaction features corresponding to the branches; detecting the target objects in the to-be-detected image based on the branch interaction features corresponding to the branches.

2. The object detection method of claim 1, wherein, The branch features comprise first sub-branch features and second sub-branch features, the branch fusion features comprise first sub-branch fusion features and second sub-branch fusion features, and the feature fusion processing on the branch features corresponding to the plurality of branches to obtain branch fusion features comprises: performing feature fusion processing on the first sub-branch features corresponding to the plurality of branches to obtain the first sub-branch fusion features; performing feature fusion processing on the second sub-branch features corresponding to the plurality of branches to obtain the second sub-branch fusion features.

3. The object detection method of claim 2, wherein, Further comprising: acquiring first sub-branch output networks and second sub-branch output networks corresponding to the branches; the branch interaction features corresponding to the branches comprise branch interaction features corresponding to the first sub-branch features and branch interaction features corresponding to the second sub-branch features, and the detection of the target objects in the to-be-detected image based on the branch interaction features corresponding to the branches comprises: performing convolution operation on the branch interaction features corresponding to the first sub-branch features by using the first sub-branch output networks to obtain categories and centralities of pixel points in the to-be-detected image; performing convolution operation on the branch interaction features corresponding to the second sub-branch features by using the second sub-branch output networks to obtain distance regression values of pixel points in the to-be-detected image; detecting the target objects in the to-be-detected image based on the categories, centralities and distance regression values of the pixel points in the to-be-detected image.

4. The object detection method of claim 1, wherein, Before the acquisition of the to-be-detected image, further comprising: acquiring a plurality of training sets and a preset target detection model, wherein the plurality of training sets correspond one-to-one to a plurality of branches in the preset target detection model, labels of training images in the plurality of training sets are different from each other, and the labels are used to distinguish target objects of different categories in the training images; training the preset target detection model by using the training images to obtain a target detection model.

5. The object detection method of claim 4, wherein, Each branch of the preset target detection model comprises a preset first sub-branch output network and a preset second sub-branch output network, the preset target detection model is trained by using the training image, and a target detection model is obtained, comprising: The target object in the training image is detected by using the preset target detection model, and the first output result of the preset first sub-branch output network and the second output result of the preset second sub-branch output network corresponding to the branch are obtained; According to the first output result, the second output result and the label of the training image corresponding to the branch, the first loss value and the second loss value corresponding to the branch are determined; Based on the first loss value and the second loss value corresponding to the branch, the loss value corresponding to the branch is determined; Based on the loss value corresponding to the branch, the loss value of the preset target detection model is determined; According to the loss value of the preset target detection model, the parameters of the preset target detection model are updated, and the step of detecting the target object in the training image by using the preset target detection model is returned to be executed, and iteration is performed until the loss value of the preset target detection model meets a preset condition, and a target detection model is obtained.

6. The object detection method of claim 5, wherein, The loss value of the preset target detection model is determined based on the loss value corresponding to the branch, comprising: Determine the historical average loss value corresponding to the branch in the current iteration sliding window, the current iteration sliding window represents from the nth iteration to the mth iteration; Determine the historical average loss value corresponding to the branch in the last iteration sliding window, the last iteration sliding window represents from the nth-j iteration to the mth-j iteration, j, n and m are all positive integers; Based on the historical average loss value corresponding to the branch in the last iteration sliding window and the historical average loss value corresponding to the branch in the current iteration sliding window, the loss change rate corresponding to the branch is determined; According to the loss change rate corresponding to the branch, the loss weight corresponding to the branch is determined; Based on the loss weight corresponding to the branch and the loss value corresponding to the branch, the loss value of the preset target detection model is determined.

7. A target detection apparatus characterized by comprising: Comprise: An acquisition unit is used for acquiring an image to be detected, and the image to be detected comprises at least one target object; A feature extraction unit is used for performing feature extraction on the image to be detected to obtain a plurality of first features of different scales; A branch feature extraction unit is used for performing branch feature extraction on the first features to obtain branch features corresponding to a plurality of branches, and the branch features are features corresponding to target objects of different categories in the image to be detected; The interaction unit is configured to perform feature interaction processing on the branch features corresponding to the plurality of branches to obtain branch interaction features corresponding to the plurality of branches, and includes: obtaining an attention interaction network, the attention interaction network including an attention subnetwork and a feature extraction subnetwork; performing feature fusion processing on the branch features corresponding to the plurality of branches to obtain branch fusion features; performing feature extraction on the branch fusion features by using the feature extraction subnetwork to obtain a plurality of new branch features; performing attention adjustment on the new branch features by using the attention subnetwork to obtain adjusted new branch features; and performing residual processing on the adjusted new branch features and the corresponding branch features to obtain the branch interaction features corresponding to the plurality of branches; The detection unit is configured to detect the target object in the to-be-detected image based on the branch interaction features corresponding to the plurality of branches.

8. The object detection apparatus according to claim 7, wherein The branch features include first sub-branch features and second sub-branch features, and the branch fusion features include first sub-branch fusion features and second sub-branch fusion features. The interaction unit is specifically configured to perform feature fusion processing on the first sub-branch features corresponding to the plurality of branches to obtain the first sub-branch fusion features, and perform feature fusion processing on the second sub-branch features corresponding to the plurality of branches to obtain the second sub-branch fusion features.

9. The object detection apparatus according to claim 8, wherein The target detection device is further configured to: obtain a first sub-branch output network and a second sub-branch output network corresponding to the branch; The branch interaction features corresponding to the plurality of branches include branch interaction features corresponding to the first sub-branch features and branch interaction features corresponding to the second sub-branch features, and the detection unit is specifically configured to: perform convolution operation on the branch interaction features corresponding to the first sub-branch features by using the first sub-branch output network to obtain the category and the center degree of the pixel points in the to-be-detected image; perform convolution operation on the branch interaction features corresponding to the second sub-branch features by using the second sub-branch output network to obtain the distance regression value of the pixel points in the to-be-detected image; detect the target object in the to-be-detected image based on the category, the center degree, and the distance regression value of the pixel points in the to-be-detected image.

10. The object detection apparatus according to claim 7, wherein The branch feature extraction unit is specifically configured to: obtain a branch extraction network, and perform branch feature extraction on the head features by using the branch extraction network to obtain the branch features corresponding to the plurality of branches; The branch includes a first branch and a second branch, the branch extraction network includes a first branch extraction network and a second branch extraction subnetwork, and the branch feature extraction on the head features by using the branch extraction network to obtain the branch features corresponding to the plurality of branches includes: perform branch feature extraction on the head features by using the first branch extraction network to obtain the first branch features corresponding to the first branch; perform branch feature extraction on the head features by using the second branch extraction network to obtain the second branch features corresponding to the second branch.

11. The object detection apparatus according to claim 7, wherein The feature extraction unit is specifically configured to: obtain a backbone network and a feature pyramid network; perform feature extraction on the to-be-detected image by using the backbone network to obtain a plurality of backbone features; perform feature extraction on the plurality of backbone features by using the feature pyramid network to obtain a plurality of head features of different scales.

12. The object detection apparatus according to claim 7, wherein The target detection device is further configured to: Obtain a plurality of training sets and a preset target detection model, the plurality of training sets correspond to a plurality of branches in the preset target detection model one by one, and labels of training images in the plurality of training sets are different, and the labels are used to distinguish different categories of target objects in the training images; The preset target detection model is trained by using the training images to obtain a target detection model.

13. The object detection apparatus according to claim 12, wherein Each branch in the preset target detection model includes a preset first sub-branch output network and a preset second sub-branch output network, and the target detection device is further configured to: The preset target detection model is used to detect target objects in the training images to obtain a first output result of the preset first sub-branch output network and a second output result of the preset second sub-branch output network corresponding to the branch; According to the first output result, the second output result and the label of the training image corresponding to the branch, a first loss value and a second loss value corresponding to the branch are determined; Based on the first loss value and the second loss value corresponding to the branch, a loss value corresponding to the branch is determined; Based on the loss value corresponding to the branch, a loss value of the preset target detection model is determined; According to the loss value of the preset target detection model, the parameters of the preset target detection model are updated, and the step of using the preset target detection model to detect target objects in the training images is returned to be iterated until the loss value of the preset target detection model meets a preset condition to obtain a target detection model.

14. The object detection apparatus according to claim 13, wherein The target detection device is further configured to: Determine a loss change rate corresponding to the branch; According to the loss change rate corresponding to the branch, a loss weight corresponding to the branch is determined; Based on the loss weight corresponding to the branch and the loss value corresponding to the branch, a loss value of the preset target detection model is determined.

15. The object detection apparatus according to claim 13, wherein The target detection device is further configured to: Determine a historical average loss value corresponding to the branch in a current iteration sliding window and a historical average loss value of the preset target detection model in the current iteration sliding window, the current iteration sliding window representing from the nth iteration to the mth iteration; Determine a historical average loss value corresponding to the branch in a previous iteration sliding window, the previous iteration sliding window representing from the nth-j iteration to the mth-j iteration, j, n and m are all positive integers; Based on the historical average loss value corresponding to the branch in the previous iteration sliding window, the historical average loss value corresponding to the branch in the current iteration sliding window and the historical average loss value of the preset target detection model in the current iteration sliding window, a loss change rate corresponding to the branch is determined.

16. The object detection apparatus according to claim 14, wherein The target detection device is further configured to: Determine a historical average loss value of the preset target detection model in a current iteration sliding window; According to the loss change rate corresponding to the branch, a loss weight corresponding to the branch is determined, including: Based on the historical average loss value corresponding to the branch in the current iteration sliding window and the historical average loss value of the preset target detection model in the current iteration sliding window, a loss magnitude corresponding to the branch is determined; According to the loss magnitude corresponding to the branch and the loss change rate corresponding to the branch, a loss weight corresponding to the branch is determined.

17. An electronic device, comprising: A processor and a memory are included, and the memory stores a plurality of instructions; the processor loads the instructions from the memory to execute the steps in the target detection method of any one of claims 1-6.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which are loaded by a processor to execute the steps in the target detection method according to any one of claims 1-6.

19. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the target detection method according to any one of claims 1-6.

Images