Target detection method, device, electronic device and storage medium
By self-attention encoding of the target image and category set, the similarity score is calculated to filter the target category encoding characteristics, the problem of unscalable target detection in the prior art is solved, and the ability to detect open categories and generalize is improved.
Patent Information
- Application Number
- CN202310153095.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-10
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-02-10
AI Technical Summary
The existing object detection algorithm is not scalable in terms of target category labels, and cannot implement open-category object detection, and cannot detect new categories outside predefined.
By self-attention encoding based on the image block features and category features of the target image and category set, the image block encoding features and category encoding features are obtained, the similarity score is calculated, and the preset number of target category encoding features is filtered for object detection.
Accurate detection of target images containing any type is achieved, which improves the generalization ability of the object detection model in categories that have not been seen before, and avoids the limitations of fixed category detection.
Smart Images

Figure CN116310520B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a target detection method, device, electronic equipment and storage medium. Background Art
[0002] Object detection is a fundamental computer vision task that aims to locate objects with tight boundaries in an image. Existing object detection algorithms are not scalable in terms of object category labels. Limited by the training dataset, most object detectors can only detect objects from a predefined set of fixed categories. For example, an object detector trained on a dataset can only detect objects in 80 categories and cannot predict new categories outside the training dataset.
[0003] In existing target detection methods, target category labels are not scalable and cannot detect open category targets. Therefore, how to achieve open category target detection is an important issue that needs to be solved in the industry. Summary of the invention
[0004] The present invention provides a target detection method, device, electronic device and storage medium, which are used to solve the technical problem that the prior art cannot realize open category target detection.
[0005] The present invention provides a target detection method, comprising:
[0006] Based on a target image and a category set of the target image, determining a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0007] Based on an image category unified encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set;
[0008] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, obtaining a preset number of target category coding features from the category coding feature set;
[0009] Target detection is performed based on the target category coding features to obtain a target detection result of the target image.
[0010] According to a target detection method provided by the present invention, based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, a preset number of target category coding features are obtained from the category coding feature set, including:
[0011] Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set;
[0012] The first preset number of image block encoding features with the highest similarity scores are used as the target category encoding features.
[0013] According to a target detection method provided by the present invention, determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set includes:
[0014] Acquire the global features of the target image, and perform self-attention encoding on the global features based on an image category unified encoder to obtain global encoding features of the global features;
[0015] Determining the image global category score of each category coding feature based on the similarity between the global coding feature and each category coding feature in the category coding feature set;
[0016] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set and the image global category score, a similarity score between the image block coding feature and each category coding feature in the category coding feature set is determined.
[0017] According to a target detection method provided by the present invention, based on a target image and a category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set are determined, including:
[0018] Divide the target image into multiple image blocks according to the grid, and obtain the image block features of each image block based on the residual network;
[0019] Based on the text encoder, feature extraction is performed on the categories in the category set to determine a category feature set corresponding to the category set.
[0020] According to a target detection method provided by the present invention, based on an image category unified encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set, including:
[0021] splicing the category features in the category feature set and the image block features to obtain a feature sequence of the target image;
[0022] The feature sequence is input into the image category unified encoder for self-attention encoding to obtain the image block encoding features of the image block features and the category encoding feature set of the category feature set.
[0023] According to a target detection method provided by the present invention, target detection is performed based on the target category coding feature to obtain a target detection result of the target image, including:
[0024] The target category encoding feature is input into a decoder to obtain a target detection result of the target image, and the decoder is trained based on image samples and their corresponding category labels.
[0025] According to a target detection method provided by the present invention, the target detection result includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image.
[0026] The present invention also provides a target detection device, comprising:
[0027] A feature extraction module, used to determine, based on a target image and a category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0028] An encoding module, configured to perform self-attention encoding on the image block features and the category features in the category feature set based on an image category unified encoder, to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set;
[0029] A screening module, configured to obtain a preset number of target category coding features from the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set;
[0030] The target detection module is used to perform target detection based on the target category coding feature to obtain the target detection result of the target image.
[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the target detection methods described above when executing the computer program.
[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above-mentioned target detection methods when executed by a processor.
[0033] The target detection method, device, electronic device and storage medium provided by the present invention, by taking the target image and the category set of the target image as input at the same time before performing target detection on the target image, and extracting coding features for the image block and category in a unified manner based on the image category unified encoder. Based on the extracted coding features, the target category coding features used for target detection are determined, and the target detection result of the target image is obtained. It is achieved that in the process of performing target detection, it is no longer limited to a small number of categories of fixed targets. For target images containing any type of target, accurate detection results can be obtained based on a predefined category set, thereby realizing open category target detection, effectively improving the generalization ability of the target detection model on unseen categories. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 It is a flow chart of the target detection method provided by the present invention;
[0036] Figure 2 It is a schematic diagram of the input and output of the unified encoder for image categories provided by the present invention;
[0037] Figure 3 It is a schematic diagram of target category coding feature screening provided by the present invention;
[0038] Figure 4 It is a flow chart of the target detection method provided by the present invention;
[0039] Figure 5 It is a schematic diagram of the execution flow provided by the present invention;
[0040] Figure 6 It is a structural schematic diagram of the target detection device provided by the present invention;
[0041] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] Object detection is a fundamental computer vision task that aims to locate objects with tight boundaries in an image. With the advent of deep learning, object detection has made significant progress. However, most object detection algorithms are not scalable in terms of object category labels. That is, limited by the training dataset, most object detectors can only detect objects from a pre-defined set of fixed categories. For example, an object detector trained on a dataset can only detect objects in 80 categories and cannot predict new categories outside the training dataset. If you want to detect new categories, a direct approach is to collect the corresponding training images and then retrain or fine-tune the detection model. However, the cost of data collection and model training is often very high and is not a practical solution.
[0044] With the development of large-scale general language models, researchers have tried to use the text features of category labels to assist in open category recognition and detection. Since the language model is learned from a large-scale corpus, it also contains knowledge of unseen category labels, so it has the ability to determine whether the object belongs to any input category. Related methods often learn how to align the image features extracted by the detector with the label text features extracted by the general language model, and have achieved certain results. This method often presupposes that the model can first give a set of definite, category-independent detection box results, and then match these results with text features to determine their corresponding categories. However, these object detection box results are not guided or influenced by the category of the object to be detected, and whether they can meet the correctness under any category input conditions is still questioned. Therefore, how to make full use of the label text that you want to detect to guide the object detector to focus on the object you really want to detect is an urgent problem to be solved.
[0045] In view of the defects of the related methods, the present invention provides a target detection method. Figure 1 One of the flow charts of the target detection method provided by the present invention is shown in FIG. Figure 1 The target detection method provided by the present invention comprises the following steps:
[0046] Step 110: Based on a target image and a category set of the target image, determine a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0047] Step 120: Based on the image category unified encoder, self-attention encoding is performed on the image block feature and the category features in the category feature set to obtain the image block encoding feature of the image block feature and the category encoding feature set of the category feature set;
[0048] Step 130: based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, obtaining a preset number of target category coding features from the category coding feature set;
[0049] Step 140: Perform target detection based on the target category coding feature to obtain a target detection result of the target image.
[0050] The target detection method provided by the present invention can be executed by an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (Network Attached Storage, NAS) or a personal computer (personal computer, PC), etc., which is not specifically limited by the present invention.
[0051] The following takes a computer executing the target detection method provided by the present invention as an example to describe the technical solution of the present invention in detail.
[0052] In step 110, a target image for target detection is obtained, and based on the target image and the category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set are determined.
[0053] The category set of the target image is a set of category texts corresponding to the targets that may appear in the target image when the target image is detected. Based on the target image, the specific categories in the category set can be predefined. For example, if the target image contains two targets, pedestrians and vehicles, the category set of the target image is a set containing multiple pedestrian categories and multiple vehicle categories. Each category in the category set is a category in text form.
[0054] After acquiring the target image, the target image is divided into blocks to obtain multiple image blocks of the target image, and the image block features corresponding to each image block are determined. Based on the category set of the target image, the category features corresponding to each category in the category set are determined, thereby constructing a category feature set.
[0055] Optionally, the image block features of the target image can be divided into a number of image blocks according to a grid, and the features of each image block are extracted based on the residual network ResNet to obtain multiple image block features of the target image. For each text form category in the category set, based on CLIP (Contrastive Language-Image Pre-Training, a pre-trained neural network model for text), feature extraction is performed on each category text to obtain the category features corresponding to each category text, thereby finally obtaining a category feature set containing the features of each category.
[0056] It can be understood that the method based on the target image and the category set of the target image makes it no longer limited to a specific number of categories during the target detection process. For target images containing any type of target, accurate detection results can be obtained based on the pre-defined category set, thereby realizing open category target detection.
[0057] In step 120, after obtaining multiple image block features of the target image and a category feature set corresponding to the category set, the image block features and the category features in the category feature set are input into the image category unified encoder for self-attention encoding to obtain image block coding features of the image block features and a category coding feature set of the category feature set.
[0058] Optionally, the unified image category encoder can be constructed based on a 6-layer multi-scale deformable self-attention encoding layer. After obtaining multiple image block features of the target image and the category feature set corresponding to the category set, the multiple image block features and the category features in the category feature set are concatenated to obtain the feature sequence of the target image.
[0059]
[0060] Among them, [c (1) …c (K) ] is a category feature set consisting of K category features; [a (1) …a (N) ] are N image block features of the target image.
[0061] Feature sequence of target image The image category unified encoder constructed by inputting 6 layers of multi-scale deformable self-attention encoding layers is used for self-attention encoding to obtain the image block encoding features of the image block features and the category encoding feature set of the category feature set.
[0062] Specifically, the image category unified encoder is constructed by 6 layers of multi-scale deformable self-attention encoding layers, and the operation of each multi-scale deformable self-attention encoding layer is:
[0063]
[0064]
[0065] Among them, l is the index of the number of layers in the multi-scale deformable self-attention encoding layer, and its value ranges from 1 to 6; is the first feature of each layer; is the output feature of each layer; LN is the normalization process; MLP is the multi-layer perceptron; selfAttn is the self-attention module.
[0066] like Figure 2 As shown in the input and output schematic diagram of the image category unified encoder provided by the present invention, the feature sequence constructed based on the image block feature and the category feature set is After inputting the image category unified encoder built based on 6 layers of multi-scale deformable self-attention encoding layers, the output is An image block coding feature including an image block feature and a category coding feature set of a category feature set;
[0067]
[0068] in, is the output of the unified encoder for image categories; L is the number of multi-scale deformable self-attention encoding layers, which is 6; Encode features for K image blocks; f I (1) …f I (N) is a category encoding feature set containing N category encoding features.
[0069] In step 130, after obtaining the image block coding features and the category coding feature set, the category coding features in the category coding feature set are screened based on the similarity between the obtained image block coding features and each category coding feature in the category coding feature set, and a preset number of target category coding features are obtained from the category coding feature set.
[0070] After obtaining the image block coding features and the category coding feature set, since there are multiple image block coding features, the similarity between each image block coding feature and each category coding feature in the category coding feature set is calculated respectively, and based on the obtained similarity calculation results, the category coding features in the category coding feature set are screened to determine a preset number of target category coding features.
[0071] Optionally, when performing similarity calculation, the image block coding feature and each category coding feature in the category coding feature set can be obtained to perform similarity scoring. From the obtained multiple similarity scores, a preset number of target category coding features with the highest scores are selected. The preset number can be set according to the needs of target detection.
[0072] In step 140, after acquiring the target category coding features, target detection is performed on the target category coding features to obtain a target detection result of the target image.
[0073] Optionally, after obtaining the target category coding features, target detection can be performed on the target category coding features based on a target detection algorithm to obtain a target detection result of the target image. During the target detection process, the category of each target in the target image, the category detection score, and the rectangular frame coordinates of each target in the target image can be obtained as the target detection result of the target image.
[0074] The target detection method provided by the embodiment of the present invention takes the target image and the category set of the target image as input before performing target detection on the target image, and extracts coding features for the image block and the category in a unified manner based on the image category unified encoder. Based on the extracted coding features, the target category coding features used for target detection are determined, and the target detection result of the target image is obtained. It is achieved that in the process of target detection, it is no longer limited to a small number of categories of fixed targets. For target images containing any type of target, accurate detection results can be obtained based on a predefined category set, thereby realizing open category target detection, effectively improving the generalization ability of the target detection model on unseen categories.
[0075] In one embodiment, based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, a preset number of target category coding features are obtained from the category coding feature set, including: determining the similarity score between the image block coding feature and each category coding feature in the category coding feature set; and using the first preset number of image block coding features with the highest similarity scores as the target category coding features.
[0076] When determining the similarity between the image block coding feature and each category coding feature in the category coding feature set, each category coding feature in the category coding feature set is screened based on calculating the similarity score between the image block coding feature and each category coding feature in the category coding feature set.
[0077] like Figure 3 As shown in the target category coding feature screening schematic diagram provided by the present invention, after calculating the similarity scores between the image block coding features and the category coding features in the category coding feature set, the obtained similarity scores are sorted, and the first preset number of image coding block features with the highest similarity scores are used as the target category coding features for subsequent target detection.
[0078] The target detection method provided by the embodiment of the present invention screens each category coding feature in the category coding feature set based on calculating the similarity score between the image block coding feature and each category coding feature in the category coding feature set, thereby realizing the determination of the target category coding feature used for target detection.
[0079] In one embodiment, determining the similarity score between the image block coding feature and each category coding feature in the category coding feature set includes: obtaining the global feature of the target image, and based on the image category unified encoder, performing self-attention encoding on the global feature to obtain the global coding feature of the global feature; determining the image global category score of each category coding feature based on the similarity between the global coding feature and each category coding feature in the category coding feature set; determining the similarity score between the image block coding feature and each category coding feature in the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set and the image global category score.
[0080] The global features of the target image are obtained. The global features of the target image are global markers of the target image, which can reflect the global characteristics of the target image. The multiple image block features of the target image obtained are multiple local features of the target image, while the global features of the target image are the features of the entire target image relative to the image block features.
[0081] After obtaining the global features of the target image, the global features are self-attention encoded based on the image category unified encoder to obtain the global encoding features of the global features. Based on the similarity between the global encoding features and the encoding features of each category in the category encoding feature set, the image global category score of each category encoding feature is determined. Based on the global category score and the similarity between the image block encoding features and the encoding features of each category in the category encoding feature set, the similarity score between the image block encoding features and the encoding features of each category in the category encoding feature set is determined.
[0082] Optionally, the specific process of determining the similarity score may be:
[0083] The global feature g of the target image is input into the image category unified encoder, and the global feature is self-attention encoded to obtain the global encoding feature f of the global feature. g .
[0084] Based on the global encoding feature f g The similarity with each category encoding feature in the category encoding feature set is used to determine the image global category score of each category encoding feature.
[0085]
[0086] Among them, s (k) is the global category score of the image; f g is the global encoding feature; f C is the category coding feature, i is the index of the category coding feature in the category coding feature set; d is the feature dimension of the unified encoder of the image category.
[0087] Based on the global category score and the similarity between the image block coding feature and each category coding feature in the category coding feature set, the similarity score between the image block coding feature and each category coding feature in the category coding feature set is determined:
[0088]
[0089] in, Similarity scores for class-encoded features; is the image block coding feature, k is the index of the image block coding feature; f I (i) is the category coding feature in the category coding feature set, i is the index of the category coding feature; s (k) is the global category score of the image; d is the feature dimension of the unified encoder of the image category.
[0090] The target detection method provided by the embodiment of the present invention determines the global category score and realizes the determination of the similarity score between the image block coding feature and each category coding feature in the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, as well as the image global category score.
[0091] In one embodiment, based on a target image and a category set of the target image, multiple image block features of the target image and a category feature set corresponding to the category set are determined, including: dividing the target image into multiple image blocks according to a grid, and obtaining image block features of each image block based on a residual network; based on a text encoder, performing feature extraction on categories in the category set to determine a category feature set corresponding to the category set.
[0092] After acquiring the target image, the target image is divided to obtain a plurality of image blocks of the target image. Specifically, the target image is divided into a plurality of image blocks according to a grid, and image block features of each image block are obtained based on a residual network.
[0093] The category set of the target image is a set of category texts corresponding to the targets that may appear in the target image when the target image is detected. For each text category in the category set, based on the text encoder, feature extraction is performed on each category text to obtain the category features corresponding to each category text, thereby finally obtaining a category feature set containing the features of each category.
[0094] The target detection method provided by the embodiment of the present invention divides the target image after acquiring the target image to obtain multiple image blocks of the target image. For each text form category in the category set, based on the text encoder, feature extraction is performed on each category text to obtain the category features corresponding to each category text, which provides a basis for subsequently extracting coding features for image blocks and categories in a unified manner.
[0095] In one embodiment, based on a unified image category encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block coding features of the image block features and a category coding feature set of the category feature set, including: concatenating the category features in the category feature set and the image block features to obtain a feature sequence of the target image; inputting the feature sequence into a unified image category encoder for self-attention encoding to obtain image block coding features of the image block features and a category coding feature set of the category feature set.
[0096] When performing self-attention encoding on the image block features and the category features in the category feature set, the image block features and the category features in the category feature set can be concatenated to obtain the feature sequence of the target image.
[0097]
[0098] Among them, [c (1) … c (K)] is a category feature set consisting of K category features; [a (1) … a (N) ] are N image block features of the target image.
[0099] After obtaining the feature sequence of the target image, the feature sequence is input into the image category unified encoder for self-attention encoding to obtain the image block encoding features of the image block features and the category encoding feature set of the category feature set.
[0100] The target detection method provided by the embodiment of the present invention obtains a feature sequence of the target image by splicing image block features and category features in a category feature set, and inputs the feature sequence into an image category unified encoder for self-attention encoding, thereby realizing the determination of image block coding features of image block features and category coding feature sets of category feature sets.
[0101] In one embodiment, target detection is performed based on the target category coding features to obtain the target detection result of the target image, including: inputting the target category coding features into a decoder to obtain the target detection result of the target image, and the decoder is trained based on image samples and their corresponding category labels.
[0102] After obtaining the target category encoding features for target detection, the target category encoding features are input into the decoder for target detection to obtain the target detection result of the target image.
[0103] The decoder is a supervised object detection model that can realize object detection. The decoder needs to be trained before use, and is trained based on image samples and their corresponding category labels.
[0104] The target detection method provided by the embodiment of the present invention uses a decoder obtained by training image samples and their corresponding category labels to perform target detection on target category encoding features, obtain target detection results for the target image, and realize the target detection process for the target image.
[0105] In one embodiment, the target detection result includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image.
[0106] In the process of performing target detection on the target image, target detection is performed on the target category encoding features based on the target detection algorithm to obtain the target detection result of the target image. The obtained target detection result includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image.
[0107] The target category of each target in the target image is determined by screening based on the target image category set. The target category detection score is the probability of the target category output by the detection model during the target detection process. The coordinates of the rectangular box in the target image are the specific positions of the targets identified in the target image.
[0108] The target detection method provided by the embodiment of the present invention performs target detection on the target category coding feature to obtain the target detection result of the target image. The target detection result obtained includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image, thereby realizing the determination of the detection result of the target image.
[0109] The following is a flow chart of a target detection method provided by the present invention: Figure 4 Take the following as an example to illustrate the technical solution provided by the present invention:
[0110] Step 410, input the target image and specify the target category to be detected. Divide the target image into a number of image blocks according to the grid, extract the features of each image block based on the residual network ResNet, and obtain multiple image block features of the target image. Based on the CLIP text encoder, extract the features of each category of text to obtain the category features corresponding to each category of text, so as to finally obtain a category feature set containing the features of each category.
[0111] Step 420, a unified image category encoder is constructed to process all inputs, which includes 6 layers of multi-scale deformable self-attention encoding layers to process the input feature sequence. The feature sequence is obtained by concatenating the global features, category feature sets, and image block features of the target image.
[0112] The feature sequence obtained by concatenating the global features, category feature set and image block features of the target image for:
[0113]
[0114] Where g is the global feature of the target image; C = [c (1) … c (K) ], is a set of category features consisting of K categories; a (1) …a (N) are N image block features of the target image.
[0115] Specifically, the image category unified encoder is constructed by 6 layers of multi-scale deformable self-attention encoding layers, and the operation of each multi-scale deformable self-attention encoding layer is:
[0116]
[0117]
[0118] Among them, l is the index of the number of layers in the multi-scale deformable self-attention encoding layer, and its value ranges from 1 to 6; is the first feature of each layer; is the output feature of each layer; LN is the normalization process; MLP is the multi-layer perceptron; selfAttn is the self-attention module.
[0119] In the feature sequence After inputting the image category unified encoder built based on 6 layers of multi-scale deformable self-attention encoding layers, the output is The image block encoding features containing the image block features and the category encoding feature set of the category feature set:
[0120]
[0121] in, is the output of the unified encoder for image categories; L is the number of multi-scale deformable self-attention encoding layers, which is 6; Encode features for K image blocks; f I (1) …f I (N) is a category coding feature set containing N category coding features; f g is the global encoding feature of the global feature.
[0122] Based on the global encoding feature f g The similarity with each category encoding feature in the category encoding feature set is used to determine the image global category score of each category encoding feature:
[0123]
[0124] Among them, s (k) is the global category score of the image; f g is the global encoding feature; f C is the category coding feature; i is the index of the category coding feature in the category coding feature set; d is the feature dimension of the unified encoder of the image category.
[0125] Step 430 , based on the global category score and the similarity between the image block coding feature and each category coding feature in the category coding feature set, determine the similarity score between the image block coding feature and each category coding feature in the category coding feature set.
[0126]
[0127] in, Similarity scores for class-encoded features; is the image block encoding feature; k is the index of the image block encoding feature; f I (i) is the category coding feature in the category coding feature set; i is the index of the category coding feature; s (k) is the global category score of the image; d is the feature dimension of the unified encoder of the image category.
[0128] For the obtained similarity scores of the category coding features, a preset number of target category coding features with the highest scores are selected from the obtained multiple similarity scores.
[0129]
[0130] in, Encoding features for a preset number of target categories obtained; is the similarity score of the category encoding feature; k is the index of the image block encoding feature.
[0131] Step 440, input the target category encoding feature into the decoder for target detection, and obtain the target detection result of the target image. The decoder is a supervised target detection model that can realize target detection. The decoder needs to be trained before use, and is trained based on image samples and their corresponding category labels. Among them, the target detection result obtained includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image.
[0132] The specific execution process is as follows: Figure 5 The execution flow diagram provided by the present invention is shown in the figure. The feature sequence obtained after splicing is input into the image category unified encoder to obtain the image global category score, image block coding features and category coding feature set. Based on the image global category score, image block coding features and category coding feature set, the target category coding features are determined and input into the decoder to obtain the target detection result.
[0133] Figure 6 A schematic diagram of the structure of the target detection device provided by the present invention is shown in FIG. Figure 6 As shown, the device comprises:
[0134] A feature extraction module 610 is used to determine, based on a target image and a category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0135] An encoding module 620 is used to perform self-attention encoding on the image block feature and the category features in the category feature set based on an image category unified encoder to obtain an image block encoding feature of the image block feature and a category encoding feature set of the category feature set;
[0136] A screening module 630 is configured to obtain a preset number of target category coding features from the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set;
[0137] The target detection module 640 is used to perform target detection based on the target category coding feature to obtain the target detection result of the target image.
[0138] The target detection device provided by the embodiment of the present invention takes the target image and the category set of the target image as input before performing target detection on the target image, and extracts coding features for the image block and the category in a unified manner based on the image category unified encoder. Based on the extracted coding features, the target category coding features used for target detection are determined, and the target detection result of the target image is obtained. It is achieved that in the process of target detection, it is no longer limited to a small number of categories of fixed targets. For target images containing any type of target, accurate detection results can be obtained based on a predefined category set, thereby realizing open category target detection, effectively improving the generalization ability of the target detection model on unseen categories.
[0139] In one embodiment, the screening module 630 is specifically used to:
[0140] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, a preset number of target category coding features are obtained from the category coding feature set, including:
[0141] Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set;
[0142] The first preset number of image block encoding features with the highest similarity scores are used as the target category encoding features.
[0143] In one embodiment, the screening module 630 is further specifically configured to:
[0144] Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set includes:
[0145] Acquire the global features of the target image, and perform self-attention encoding on the global features based on an image category unified encoder to obtain global encoding features of the global features;
[0146] Determining the image global category score of each category coding feature based on the similarity between the global coding feature and each category coding feature in the category coding feature set;
[0147] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set and the image global category score, a similarity score between the image block coding feature and each category coding feature in the category coding feature set is determined.
[0148] In one embodiment, the feature extraction module 610 is specifically used to: determine, based on a target image and a category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set, including:
[0149] Divide the target image into multiple image blocks according to the grid, and obtain the image block features of each image block based on the residual network;
[0150] Based on the text encoder, feature extraction is performed on the categories in the category set to determine a category feature set corresponding to the category set.
[0151] In one embodiment, the encoding module 620 is specifically used to:
[0152] Based on the image category unified encoder, self-attention encoding is performed on the image block feature and the category feature in the category feature set to obtain the image block encoding feature of the image block feature and the category encoding feature set of the category feature set, including:
[0153] splicing the category features in the category feature set and the image block features to obtain a feature sequence of the target image;
[0154] The feature sequence is input into the image category unified encoder for self-attention encoding to obtain the image block encoding features of the image block features and the category encoding feature set of the category feature set.
[0155] In one embodiment, the target detection module 640 is specifically used to:
[0156] Performing target detection based on the target category coding feature to obtain a target detection result of the target image includes:
[0157] The target category encoding feature is input into a decoder to obtain a target detection result of the target image, and the decoder is trained based on image samples and their corresponding category labels.
[0158] In one embodiment, the target detection module 640 is further specifically configured to:
[0159] The target detection result includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image.
[0160] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the target detection method, which includes:
[0161] Based on a target image and a category set of the target image, determining a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0162] Based on an image category unified encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set;
[0163] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, obtaining a preset number of target category coding features from the category coding feature set;
[0164] Target detection is performed based on the target category coding features to obtain a target detection result of the target image.
[0165] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0166] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can perform the target detection method provided by the above methods, the method comprising:
[0167] Based on a target image and a category set of the target image, determining a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0168] Based on an image category unified encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set;
[0169] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, obtaining a preset number of target category coding features from the category coding feature set;
[0170] Target detection is performed based on the target category coding features to obtain a target detection result of the target image.
[0171] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned target detection method, the method comprising:
[0172] Based on a target image and a category set of the target image, determining a plurality of image block features of the target image and a category feature set corresponding to the category set;
[0173] Based on an image category unified encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set;
[0174] Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, obtaining a preset number of target category coding features from the category coding feature set;
[0175] Target detection is performed based on the target category coding features to obtain a target detection result of the target image.
[0176] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A target detection method, characterized in that: The method comprises: Based on a target image and a category set of the target image, determining a plurality of image block features of the target image and a category feature set corresponding to the category set; Based on an image category unified encoder, self-attention encoding is performed on the image block features and the category features in the category feature set to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set; Based on the similarity between the image block coding feature and each category coding feature in the category coding feature set, obtaining a preset number of target category coding features from the category coding feature set; Performing target detection based on the target category coding feature to obtain a target detection result of the target image; The obtaining a preset number of target category coding features from the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set includes: Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set; Using the encoding features of the first preset number of image blocks with the highest similarity scores as the target category encoding features; The determining of the similarity score between the image block coding feature and each category coding feature in the category coding feature set comprises: Acquire the global features of the target image, and perform self-attention encoding on the global features based on an image category unified encoder to obtain global encoding features of the global features; Determining the image global category score of each category coding feature based on the similarity between the global coding feature and each category coding feature in the category coding feature set; The global category score of the image of each category encoding feature is: Among them, s (k) is the global category score of the image; f g is the global encoding feature; f C is the category coding feature, i is the index of the category coding feature in the category coding feature set; d is the feature dimension of the unified encoder of the image category; Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set and the image global category score; The similarity score of the category encoding feature is: in, Similarity scores for class-encoded features; is the image block coding feature, k is the index of the image block coding feature; f I (i) is the category coding feature in the category coding feature set, i is the index of the category coding feature; s (k) is the global category score of the image; d is the feature dimension of the unified encoder of the image category.
2. The target detection method according to claim 1, characterized in that: The determining, based on the target image and the category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set comprises: Divide the target image into multiple image blocks according to the grid, and obtain the image block features of each image block based on the residual network; Based on the text encoder, feature extraction is performed on the categories in the category set to determine a category feature set corresponding to the category set.
3. The target detection method according to claim 1, characterized in that: The method of performing self-attention encoding on the image block feature and the category feature in the category feature set based on the image category unified encoder to obtain the image block encoding feature of the image block feature and the category encoding feature set of the category feature set includes: splicing the category features in the category feature set and the image block features to obtain a feature sequence of the target image; The feature sequence is input into the image category unified encoder for self-attention encoding to obtain the image block encoding features of the image block features and the category encoding feature set of the category feature set.
4. The target detection method according to claim 1, characterized in that: The performing target detection based on the target category coding feature to obtain the target detection result of the target image includes: The target category encoding feature is input into a decoder to obtain a target detection result of the target image, and the decoder is trained based on image samples and their corresponding category labels.
5. The target detection method according to claim 1, characterized in that: The target detection result includes the target category of each target in the target image, the detection score of the target category, and the rectangular frame coordinates of each target in the target image.
6. A target detection device, characterized in that: include: A feature extraction module, used to determine, based on a target image and a category set of the target image, a plurality of image block features of the target image and a category feature set corresponding to the category set; An encoding module, configured to perform self-attention encoding on the image block features and the category features in the category feature set based on an image category unified encoder, to obtain image block encoding features of the image block features and a category encoding feature set of the category feature set; A screening module, configured to obtain a preset number of target category coding features from the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set; A target detection module, used to perform target detection based on the target category coding feature to obtain a target detection result of the target image; The obtaining a preset number of target category coding features from the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set includes: Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set; Using the encoding features of the first preset number of image blocks with the highest similarity scores as the target category encoding features; The determining of the similarity score between the image block coding feature and each category coding feature in the category coding feature set comprises: Acquire the global features of the target image, and perform self-attention encoding on the global features based on an image category unified encoder to obtain global encoding features of the global features; Determining the image global category score of each category coding feature based on the similarity between the global coding feature and each category coding feature in the category coding feature set; The global category score of the image of each category encoding feature is: Among them, s (k) is the global category score of the image; f g is the global encoding feature; f C is the category coding feature, i is the index of the category coding feature in the category coding feature set; d is the feature dimension of the unified encoder of the image category; Determining a similarity score between the image block coding feature and each category coding feature in the category coding feature set based on the similarity between the image block coding feature and each category coding feature in the category coding feature set and the image global category score; The similarity score of the category encoding feature is: in, Similarity scores for class-encoded features; is the image block coding feature, k is the index of the image block coding feature; f I (i) is the category coding feature in the category coding feature set, i is the index of the category coding feature; s (k) is the global category score of the image; d is the feature dimension of the unified encoder of the image category.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the target detection method according to any one of claims 1 to 5 is implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target detection method according to any one of claims 1 to 5 is implemented.
