Target detection methods, devices, equipment and storage media
By training image encoders and text encoders by learning the alignment relationship between images and text, the problem of additional annotation and training required for new types of object detection in existing technologies is solved, and efficient object detection generalization and accuracy are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2023-09-25
- Publication Date
- 2026-07-17
AI Technical Summary
Existing neural network-based object detection solutions require additional labeled data for new object types and retraining of the neural network model when facing out-of-set object detection needs, which increases manpower and time costs.
By learning the alignment relationship between images and text through pre-configured image encoders and text encoders, and training the image encoders and text encoders using text-image pair training data, image features and semantic features are extracted. The similarity between the features of the sub-image and the semantic features of the candidate category labels is calculated, and the target candidate category label with the highest similarity is taken as the category label of the sub-image to achieve object detection.
It improves the generalization and accuracy of object detection without requiring additional labeled data or model retraining, while reducing manpower and time costs.
Smart Images
Figure CN117237989B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of target detection technology, and more specifically, to a target detection method, apparatus, equipment, and storage medium. Background Technology
[0002] Object detection is the process of identifying objects of interest to the user within an image. Most current object detection solutions are based on neural network models. Their working principle involves training the model on pre-labeled training data to achieve a high response to labeled objects, thus enabling object detection. However, the number of object types that this solution can detect is often determined by the labeling types in the training data. If the training data only labels N types of objects, the model can only support the detection of N types. To detect new object types, it is inevitable to supplement the data with a large number of labels for those new types, and then continue training the detection model accordingly.
[0003] As can be seen from the above, when facing the requirement of detecting out-of-set targets (targets that do not belong to the N types of targets labeled in the training data), the target detection scheme based on neural networks has to carry out two steps: data labeling and model training. The data labeling stage requires manual labeling of out-of-set targets in the image, while the model training step uses labeled data and a computing platform to further train the model, resulting in additional manpower and time costs. Summary of the Invention
[0004] In view of the above problems, this application is proposed to provide a target detection method, apparatus, device, and storage medium to solve the problem that existing neural network-based target detection schemes require additional supplementary labeled data for new target types and retraining of the neural network model when facing out-of-set target detection needs, resulting in increased manpower and time costs. The specific solution is as follows:
[0005] Firstly, a target detection method is provided, including:
[0006] The image to be detected is divided into several sub-images;
[0007] The image features of each sub-image are extracted using a pre-configured image encoder, and the semantic features of each candidate category label in the category set are extracted using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text.
[0008] The image features of each sub-image are compared with the semantic features of each candidate category label to calculate the similarity, and the target candidate category label with the highest similarity is taken as the category label of the sub-image.
[0009] The target detection result of the image to be detected is obtained based on the category labels of each of the sub-images.
[0010] Preferably, it further includes:
[0011] The similarity corresponding to the target candidate category label is used as the confidence level of the category label of the sub-image;
[0012] The target detection result of the image to be detected is obtained based on the category labels of each of the sub-images, including:
[0013] Based on the category label, confidence level, and position of each sub-image in the image to be detected, the target detection result of the image to be detected is determined.
[0014] Preferably, it further includes:
[0015] When it is necessary to update the category set, the candidate category labels in the category set are updated, including: adding, deleting and / or replacing.
[0016] Preferably, the process of dividing the image to be detected into several sub-images includes:
[0017] Obtain the training dataset for the object detection task;
[0018] The category labels of the targets in each training image in the training dataset are uniformly set as foreground labels to obtain the updated training dataset, and the foreground detector is trained using the updated training dataset.
[0019] The foreground detector is used to identify the region containing the foreground in the image to be detected;
[0020] Each foreground region in the image to be detected is segmented to obtain several sub-images.
[0021] Preferably, the process of dividing the image to be detected into several sub-images includes:
[0022] The image to be detected is divided into H×W sub-images, where H is the number of rows and W is the number of columns.
[0023] And / or,
[0024] Each sliding window in the configured set of sliding windows is used to sequentially segment the image to be detected into sub-images according to a set sliding step size, wherein the set of sliding windows includes sliding windows of at least one size.
[0025] Preferably, the training process for the image encoder and the text encoder includes:
[0026] Acquire text-image pair training data, wherein the content of the text in the text-image pair is related to the content of the image;
[0027] The image encoder and the text encoder are trained using a text-image comparison learning method based on the text-image pair training data.
[0028] Preferably, determining the target detection result of the image to be detected based on the category label, confidence level, and position of each sub-image in the image to be detected includes:
[0029] Filter each first sub-image and its category label that has a confidence level exceeding a preset confidence threshold;
[0030] Based on each first sub-image and its category label, and the position of each first sub-image in the image to be detected, the non-maximum suppression (NMS) method is used to filter the second sub-image and its category label from each first sub-image;
[0031] The target detection result of the image to be detected is composed of the category label and confidence score of each second sub-image, and the position of the second sub-image in the image to be detected.
[0032] Secondly, a target detection device is provided, comprising:
[0033] The image segmentation unit is used to segment the image to be detected into several sub-images;
[0034] The feature extraction unit is used to extract image features of each of the sub-images using a pre-configured image encoder, and to extract semantic features of each candidate category label in the category set using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text.
[0035] The similarity calculation unit is used to calculate the similarity between the image features of each sub-image and the semantic features of each candidate category label, and take the target candidate category label with the highest similarity as the category label of the sub-image.
[0036] The detection result determination unit is used to obtain the target detection result of the image to be detected based on the category label of each of the sub-images.
[0037] Thirdly, a target detection device is provided, including: a memory and a processor;
[0038] The memory is used to store programs;
[0039] The processor is used to execute the program to implement the various steps of the target detection method as described above.
[0040] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the target detection method as described above.
[0041] Using the above technical solution, this application pre-trains an image encoder and a text encoder based on text-image training data, with the alignment relationship between the image and text as the training objective. This allows the image features extracted by the trained image encoder to better match the semantic features of the relevant content in high-dimensional space, and the trained text encoder to extract the semantic features of the text and map them to the vicinity of the relevant image features. Based on this, this application first divides the image to be detected into several sub-images, and uses the trained image encoder to extract the image features of each sub-image. The trained text encoder then extracts the semantic features of each candidate category label in the category set. The similarity between the image features of the sub-image and the semantic features of each candidate category label is calculated, and the target candidate category label with the highest similarity is taken as the category label of the sub-image. Finally, the target detection result of the image to be detected is obtained based on the category labels of each sub-image. When using the proposed solution to detect newly added target categories, it is only necessary to add the category label of the newly added target category to the category set. Since the text encoder has been trained as described above, it already has the ability to extract semantic features that are closer to the image features of the text-related images. When faced with newly added category labels, it can still extract the semantic features of the text and map them to the vicinity of the relevant image features. Then, the category labels of the sub-images can be selected through feature similarity. That is, the target detection solution of this application has stronger generalization. While ensuring the accuracy of target detection, there is no need to obtain additional labeled data and retrain the neural network model for newly added target categories, which greatly reduces the manpower and time costs. Attached Figure Description
[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0043] Figure 1 This is a schematic flowchart of the target detection method provided in the embodiments of this application;
[0044] Figure 2 This example illustrates a text-image comparison learning process.
[0045] Figure 3An example of a rule-based image segmentation is provided;
[0046] Figure 4 An example of an image segmentation based on a sliding window is provided;
[0047] Figure 5 An example of a detector-assisted image segmentation is provided;
[0048] Figure 6 This example illustrates a target detection scheme based on rule-based image slicing.
[0049] Figure 7 An example of a target detection scheme based on sliding window tiling is shown in the diagram.
[0050] Figure 8 An example diagram illustrates a target detection scheme based on detector-assisted image slicing.
[0051] Figure 9 A schematic diagram illustrating an optional implementation method for a target detection scheme is provided.
[0052] Figure 10 This is a schematic diagram of a target detection device provided in an embodiment of this application;
[0053] Figure 11 This is a schematic diagram of the target detection device provided in an embodiment of this application. Detailed Implementation
[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0055] This application provides a target detection scheme that can be applied to various scenarios requiring target detection.
[0056] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, server, or cloud platform.
[0057] Next, combined Figure 1 The target detection method of this application may include the following steps:
[0058] Step S100: Divide the image to be detected into several sub-images.
[0059] Specifically, the image to be detected is the image for which object detection needs to be performed. Considering that the image to be detected generally contains the target and other background information, directly performing object detection on the image is easily affected by background information and makes it difficult to locate the target in the image. Therefore, in this step, the image to be detected is first divided into several sub-images, and then object detection is performed on each sub-image in subsequent steps.
[0060] There are several ways to segment the image to be detected into sub-images in this step, which will be described in detail in the following embodiments.
[0061] Step S110: Extract image features of each sub-image using a pre-configured image encoder, and extract semantic features of each candidate category label in the category set using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text.
[0062] Specifically, to avoid the need to acquire additional labeled data and update the training for newly added target categories when using existing neural network models for target detection, this embodiment pre-trains the text encoder and image encoder together. The training process aims to learn the alignment relationship between images and text, and trains the training data based on text-image pairs.
[0063] In this text-image pair, the content of the text and the content of the image are correlated. An example of a text-image pair training data is: [(Text1,Image1),(Text2,Image2),…,(TextN,ImageN)], where Text represents the training text and Image represents the training image. Paired text and images are correlated, while unpaired text and images are not correlated.
[0064] Text-image pair training data can take many forms. For example, news headlines and photos have a certain correlation, so the headline and photo can be used as a pair of training data. Other examples include product manual descriptions and illustrations, images and comments published on media platforms, etc. These can all be sources of text-image pair training data. Such text-image pair training data can be obtained through public databases or other legitimate channels.
[0065] In this embodiment, the text encoder encodes the input text to obtain its semantic features. The image encoder encodes the input image to obtain its image features. To bring the text encoder and image encoder closer in the vector space, so that the image features extracted by the image encoder better match the semantic features of the relevant content in the high-dimensional space, and the text encoder can also extract the semantic features of the text and map them to the vicinity of relevant image features, this embodiment uses learning the alignment relationship between the image and text as the training objective, and trains the text encoder and image encoder based on text-image pair training data.
[0066] After obtaining the trained text encoder and image encoder, the image encoder can be used to extract image features for each sub-image, and the text encoder can be used to extract semantic features of each candidate category label in the category set.
[0067] The category set contains all possible category labels of the target to be detected, i.e., candidate category labels.
[0068] The text encoder can be a neural network model composed of Transformer or other network structures. The image encoder can be a neural network model composed of Convolutional Neural Networks (CNN) and Transformer, or other neural network models with different structures.
[0069] Step S120: Calculate the similarity between the image features of each sub-image and the semantic features of each candidate category label, and take the target candidate category label with the highest similarity as the category label of the sub-image.
[0070] Specifically, for each sub-image, the image features of the sub-image can be compared with the semantic features of each candidate category label, and the target candidate category label with the highest similarity can be selected as the category label of the sub-image.
[0071] Furthermore, the similarity between the target candidate category labels can be used as the confidence level of the category labels of the sub-image, and the confidence level indicates the credibility of the corresponding category labels.
[0072] Step S130: Obtain the target detection result of the image to be detected based on the category labels of each of the sub-images.
[0073] Specifically, based on the above steps, the category labels of each sub-image have been obtained, and the target detection results of the image to be detected can be obtained based on the category labels of each sub-image.
[0074] The target detection method provided in this application pre-trains on the alignment relationship between images and text as the training objective. An image encoder and a text encoder are trained based on text-image training data. This allows the image features extracted by the trained image encoder to better match the semantic features of the relevant content in high-dimensional space, and the trained text encoder to extract the semantic features of the text and map them to the vicinity of relevant image features. Based on this, the method first divides the image to be detected into several sub-images, extracts the image features of each sub-image using the trained image encoder, and extracts the semantic features of each candidate category label in the category set using the trained text encoder. The similarity between the image features of the sub-image and the semantic features of each candidate category label is calculated, and the target candidate category label with the highest similarity is taken as the category label of the sub-image. Finally, the target detection result of the image to be detected is obtained based on the category labels of each sub-image. When using the proposed solution to detect newly added target categories, it is only necessary to add the category label of the newly added target category to the category set. Since the text encoder has been trained as described above, it already has the ability to extract semantic features that are closer to the image features of the text-related images. When faced with newly added category labels, it can still extract the semantic features of the text and map them to the vicinity of the relevant image features. Then, the category labels of the sub-images can be selected through feature similarity. That is, the target detection solution of this application has stronger generalization. While ensuring the accuracy of target detection, there is no need to obtain additional labeled data and retrain the neural network model for newly added target categories, which greatly reduces the manpower and time costs.
[0075] Furthermore, when the category set needs to be updated according to the task requirements of the object detection task, the candidate category labels in the above category set can be updated. This update can include: adding, deleting and / or replacing.
[0076] For example, when an object detection task needs to detect newly added object categories, the category labels corresponding to the newly added object categories can be added to the category set without having to re-acquire the labeled data of the newly added object categories and iteratively update the model.
[0077] In summary, the proposed solution achieves out-of-category target detection in a simpler and more efficient way, avoiding the additional manpower and time costs of traditional detection techniques when facing changes in the target category, and improving the adaptability and generalization of target detection.
[0078] In some embodiments of this application, the training process of the above-described image encoder and text encoder is further described.
[0079] First, training data for text-image pairs is obtained, in which the content of the text and the content of the image are related.
[0080] Furthermore, a text-image contrast learning approach can be adopted, using text-image pair training data to train both the image encoder and the text encoder.
[0081] Of course, in addition to text-image comparison learning, other training methods can be used, such as LiT-tuning, a paradigm of multimodal model training, which trains image encoders and text encoders based on text-image pair training data, so that the two encoders learn the alignment relationship between text and images.
[0082] Taking text-image contrastive learning as an example, common text-image contrastive learning methods include, but are not limited to: CLIP (Contrastive Language-Image Pre-Training), UNITER, SimVLM, ViLBERT, etc.
[0083] The training process is introduced using CLIP as an example, combined with Figure 2 As shown:
[0084] Obtain the text-image pair training data: [(Text1,Image1),(Text2,Image2),…,(TextN,ImageN)].
[0085] The training text is input into the text encoder to obtain semantic features T1, T2, ..., TN. The training images are input into the image encoder to obtain image features I1, I2, ..., IN. Then, the similarity between all image features and semantic features is calculated to obtain a similarity matrix. Since paired text and image content are more correlated and should have higher similarity, the supervision signal can be designed as an identity matrix (with the same size as the similarity matrix). Finally, the cross-entropy loss function is used to improve the response strength of the diagonal of the similarity matrix.
[0086] After the above text-image comparison learning, the image features extracted by the trained image encoder are more closely aligned with the semantic features of the relevant content in the high-dimensional space, and the trained text encoder is also able to extract the semantic features of the text and map them to the vicinity of the relevant image features.
[0087] In some embodiments of this application, several optional implementations of step S100 above, which divides the image to be detected into several sub-images, are further described.
[0088] 1. Rule-based image slicing:
[0089] Combination Figure 3 As shown, the image to be detected can be divided into H×W sub-images, where H is the number of rows and W is the number of columns.
[0090] During segmentation, the coordinates of each sub-image are preserved; specifically, these can be the coordinates of the top-left and bottom-right corners of the sub-image. The preserved coordinates of each sub-image are used as the position of subsequent sub-images within the image to be detected.
[0091] 2. Image slicing based on sliding windows:
[0092] Combination Figure 4 As shown, each sliding window in the configured sliding window set can be used to sequentially segment the sub-images in the sliding window according to the set sliding step size.
[0093] The set of sliding windows includes sliding windows of at least one size. Generally, the set of sliding windows may include sliding windows of various sizes, such as sliding windows of different dimensions and aspect ratios.
[0094] You can start from the top left corner of the image to be detected, and select sub-images sequentially by sliding the window according to the set sliding step size, following the order from left to right and from top to bottom, and retain the coordinates of each sub-image.
[0095] contrast Figure 3 and Figure 4 It is known that rule-based image segmentation may not perfectly match the target image, but the resulting sub-images have no overlapping areas, are relatively few in number, and require less computation, thus being more efficient. Sliding window-based image segmentation offers greater freedom and often produces sub-images that better fit the target, resulting in more accurate target localization.
[0096] Both of the above-described image slicing methods can obtain a large number of sub-images with less computational power and time. However, because the slicing process does not consider the potential region of the target, the edges of most sub-images tend to have poor alignment with the target, resulting in a lot of unnecessary computation for target classification and localization in subsequent steps. To improve the quality of the sliced sub-images, this embodiment further provides a third image slicing method, namely:
[0097] 3. Detector-assisted image slicing:
[0098] This embodiment introduces detector-assisted image slicing, combined with... Figure 5 As shown, this process can include two stages: the training process of the foreground detector (Train) and the prediction process using the foreground detector (Test). The training process of the foreground detector includes the following steps S1-S2:
[0099] S1. Obtain the training dataset for the object detection task.
[0100] Specifically, training datasets for object detection tasks can be collected and organized, such as publicly available training datasets like Coco and Objects365.
[0101] S2. Set the category label of the target in each training image in the training dataset to the foreground label to obtain the updated training dataset, and use the updated training dataset to train the foreground detector.
[0102] Specifically, the updated training dataset is obtained by uniformly assigning the category of the target in each training image to the foreground label. Further, a target detection model, such as Faster-RCNN or YOLO, can be selected and trained on the updated training dataset to obtain the trained foreground detector. The trained foreground detector can distinguish between foreground and background in an image, and the foreground is often the region where the target is located.
[0103] The prediction process using the foreground detector includes the following steps S3-S4:
[0104] S3. Use the foreground detector to identify the region where the foreground is located in the image to be detected.
[0105] Specifically, the image to be detected can be input into the foreground detector trained in the previous step to obtain the foreground region in the image output by the foreground detector.
[0106] S4. Divide each of the foreground regions in the image to be detected to obtain several sub-images.
[0107] Specifically, during the training process of the reference foreground detector, the target is often the foreground region. Therefore, in this step, the foreground detector predicts the foreground region in the image to be detected, which is the potential region of the target. This foreground region is then further segmented into sub-images. In this embodiment, by introducing a foreground detector, sub-image segmentation can be performed in conjunction with the potential region of the target. The foreground detector helps to remove background sub-images from the image to be detected, saving significant computation for subsequent tasks. Furthermore, the edges of the segmented sub-images in this embodiment more accurately match the target, avoiding interference from the background and other targets, and reducing the difficulty of target classification.
[0108] This application provides the above three optional image slicing methods. In step S100, when performing sub-image slicing on the image to be detected, any one or more combinations of the above three image slicing methods can be selected.
[0109] Further, step S130 above, the process of obtaining the target detection result of the image to be detected based on the category labels of each of the sub-images, may include:
[0110] Based on the category label, confidence level, and position of each sub-image in the image to be detected, the target detection result of the image to be detected is determined.
[0111] After the aforementioned steps, a large number of sub-images can be segmented. After determining the category label of each sub-image, since some sub-images may not contain any target, the confidence of the determined category label of a sub-image belonging to a certain target category is low, so the sub-image can be removed.
[0112] For example, each first sub-image with a confidence level exceeding a preset confidence level threshold can be filtered out, and the target detection result of the target image (including the target's category, confidence level, and location) can be obtained based on the category label, confidence level, and location of the first sub-image and the location of the first sub-image (in the image to be detected).
[0113] Furthermore, considering that when the image to be detected is segmented into sub-images, there may be overlapping areas between the segmented sub-images (for example, the sub-images obtained by using the sliding window slicing method), in order to reduce the number of sub-images with high overlap, the non-maximum suppression (NMS) method can be used for secondary screening of each of the first sub-images after the above screening to retain the optimal result.
[0114] Specifically, for each of the first sub-images selected above, a second sub-image and its category label can be further selected from each first sub-image using the NMS method, based on each first sub-image, its category label, and its position in the image to be detected. The target detection result of the image to be detected is composed of the category label and confidence score of each second sub-image and its position in the image to be detected.
[0115] Next, refer to Figures 6-8 The diagrams illustrate target detection processes using rule-based image slicing, sliding window-based image slicing, and detector-assisted image slicing, respectively.
[0116] When using rule-based image slicing, the resulting sub-images contain a large amount of background and have low similarity (confidence) to the semantic features of the candidate category labels, such as... Figure 6 As shown.
[0117] When using a sliding window-based image slicing method, the edges of the sub-images fit the target better. For sub-images containing the target, the similarity (confidence) with the semantic features of the candidate category labels is higher, thus achieving more accurate and reliable detection results. Figure 7 As shown.
[0118] When using a detector-assisted image slicing method, because sub-image slicing references the potential region of the target, the sub-image containing the target in the image to be detected can be located more accurately. This leads to a more definitive similarity matrix, thereby further improving the confidence of target classification. Figure 8 As shown.
[0119] Reference Figure 9 This embodiment introduces an optional implementation scheme for the target detection method, which may specifically include:
[0120] The encoder pre-training process and the object detection process based on the trained encoder.
[0121] The encoder pre-training process may specifically include:
[0122] A large amount of paired text and image data is collected to form the training data. Then, a text encoder and image encoder can be trained using a text-image contrast learning approach. The trained text encoder and image encoder have learned the alignment relationship between the text and image data.
[0123] The object detection process can specifically include:
[0124] The image to be detected is segmented into several sub-images. Each sub-image is then input into the trained image encoder to obtain its image features.
[0125] For each candidate category label in the category set, it is fed into a text encoder for encoding to obtain the semantic features corresponding to each candidate category label.
[0126] Calculate the similarity between the image features of each sub-image and the semantic features of each candidate category label to obtain a similarity matrix.
[0127] For each sub-image, the category with the highest similarity is selected as the category label corresponding to the sub-image, and the corresponding similarity is used as the confidence score of the class label.
[0128] Based on the confidence threshold and the NMS algorithm, each sub-image is further filtered to obtain the final filtered sub-image. Each final filtered sub-image corresponds to a target detection result (including: the sub-image's category label, confidence score, and the sub-image's position in the image to be detected).
[0129] Furthermore, the system can determine whether the categories to be detected need to be adjusted based on the object detection scheme. If so, the category set can be updated by adding, reducing, or replacing candidate category labels. When performing object detection again using the updated category set, each candidate category label in the updated category set is fed into the text encoder. The rest of the process is the same as in the above embodiment, without any additional manual input or training process, saving time and effort.
[0130] The target detection device provided in the embodiments of this application is described below. The target detection device described below and the target detection method described above can be referred to in correspondence.
[0131] See Figure 10 , Figure 10 This is a schematic diagram of the structure of a target detection device disclosed in an embodiment of this application.
[0132] like Figure 10 As shown, the device may include:
[0133] Image segmentation unit 11 is used to segment the image to be detected into several sub-images;
[0134] The feature extraction unit 12 is used to extract image features of each of the sub-images using a pre-configured image encoder, and to extract semantic features of each candidate category label in the category set using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text.
[0135] The similarity calculation unit 13 is used to calculate the similarity between the image features of each sub-image and the semantic features of each candidate category label, and take the target candidate category label with the highest similarity as the category label of the sub-image.
[0136] The detection result determination unit 14 is used to obtain the target detection result of the image to be detected based on the category label of each of the sub-images.
[0137] Optionally, the apparatus of this application may further include:
[0138] The confidence determination unit is used to take the similarity corresponding to the target candidate category label as the confidence of the category label of the sub-image. The process by which the detection result determination unit obtains the target detection result of the image to be detected based on the category labels of each sub-image includes:
[0139] Based on the category label, confidence level, and position of each sub-image in the image to be detected, the target detection result of the image to be detected is determined.
[0140] Optionally, the process by which the detection result determination unit determines the target detection result of the image to be detected based on the category label, confidence level, and position of each sub-image in the image to be detected includes:
[0141] Filter each first sub-image and its category label that has a confidence level exceeding a preset confidence threshold;
[0142] Based on each first sub-image and its category label, and the position of each first sub-image in the image to be detected, the non-maximum suppression (NMS) method is used to filter the second sub-image and its category label from each first sub-image;
[0143] The target detection result of the image to be detected is composed of the category label and confidence score of each second sub-image, and the position of the second sub-image in the image to be detected.
[0144] Optionally, the apparatus of this application may further include:
[0145] The category set update unit is used to update the candidate category labels in the category set when it is necessary to update the category set. The update includes adding, deleting and / or replacing.
[0146] Optionally, the process of the image segmentation unit dividing the image to be detected into several sub-images can be implemented in various ways, for example:
[0147] The first method:
[0148] Obtain the training dataset for the object detection task;
[0149] The category labels of the targets in each training image in the training dataset are uniformly set as foreground labels to obtain the updated training dataset, and the foreground detector is trained using the updated training dataset.
[0150] The foreground detector is used to identify the region containing the foreground in the image to be detected;
[0151] Each foreground region in the image to be detected is segmented to obtain several sub-images.
[0152] The second method:
[0153] The image to be detected is divided into H×W sub-images, where H is the number of rows and W is the number of columns.
[0154] The third method:
[0155] Each sliding window in the configured set of sliding windows is used to sequentially segment the image to be detected into sub-images according to a set sliding step size, wherein the set of sliding windows includes sliding windows of at least one size.
[0156] Optionally, the apparatus of this application may further include:
[0157] The encoder training unit, used to train the text encoder and the image encoder, may include the following training process:
[0158] Acquire text-image pair training data, wherein the content of the text in the text-image pair is related to the content of the image;
[0159] The image encoder and the text encoder are trained using a text-image comparison learning method based on the text-image pair training data.
[0160] The target detection device provided in this application embodiment can be applied to target detection equipment, such as mobile phones, computers, servers, cloud computing, etc. Optionally, Figure 11 The hardware structure block diagram of the target detection device is shown. (Refer to...) Figure 11 The hardware structure of the target detection device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;
[0161] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;
[0162] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0163] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0164] The memory stores a program, which the processor can call. The program is used for:
[0165] The image to be detected is divided into several sub-images;
[0166] The image features of each sub-image are extracted using a pre-configured image encoder, and the semantic features of each candidate category label in the category set are extracted using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text.
[0167] The image features of each sub-image are compared with the semantic features of each candidate category label to calculate the similarity, and the target candidate category label with the highest similarity is taken as the category label of the sub-image.
[0168] The target detection result of the image to be detected is obtained based on the category labels of each of the sub-images.
[0169] Optionally, the refined and extended functions of the program can be found in the description above.
[0170] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:
[0171] The image to be detected is divided into several sub-images;
[0172] The image features of each sub-image are extracted using a pre-configured image encoder, and the semantic features of each candidate category label in the category set are extracted using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text.
[0173] The image features of each sub-image are compared with the semantic features of each candidate category label to calculate the similarity, and the target candidate category label with the highest similarity is taken as the category label of the sub-image.
[0174] The target detection result of the image to be detected is obtained based on the category labels of each of the sub-images.
[0175] Optionally, the refined and extended functions of the program can be found in the description above.
[0176] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0177] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.
[0178] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A target detection method, characterized in that, include: The image to be detected is divided into several sub-images; wherein, the image slicing method can be at least one of the following: rule-based image slicing, sliding window-based image slicing, and detector-assisted image slicing; The image features of each sub-image are extracted using a pre-configured image encoder, and the semantic features of each candidate category label in the category set are extracted using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text. The image features of each sub-image are compared with the semantic features of each candidate category label to calculate the similarity, and the target candidate category label with the highest similarity is taken as the category label of the sub-image. The target detection result of the image to be detected is obtained based on the category labels of each of the sub-images.
2. The method according to claim 1, characterized in that, Also includes: The similarity corresponding to the target candidate category label is used as the confidence level of the category label of the sub-image; The target detection result of the image to be detected is obtained based on the category labels of each of the sub-images, including: Based on the category label, confidence level, and position of each sub-image in the image to be detected, the target detection result of the image to be detected is determined.
3. The method according to claim 1, characterized in that, Also includes: When it is necessary to update the category set, the candidate category labels in the category set are updated, including: adding, deleting and / or replacing.
4. The method according to claim 1, characterized in that, The process of dividing the image to be detected into several sub-images includes: Obtain the training dataset for the object detection task; The category labels of the targets in each training image in the training dataset are uniformly set as foreground labels to obtain the updated training dataset, and the foreground detector is trained using the updated training dataset. The foreground detector is used to identify the region containing the foreground in the image to be detected; Each foreground region in the image to be detected is segmented to obtain several sub-images.
5. The method according to claim 1, characterized in that, The process of dividing the image to be detected into several sub-images includes: The image to be detected is divided into H×W sub-images, where H is the number of rows and W is the number of columns. And / or, Each sliding window in the configured set of sliding windows is used to sequentially segment the image to be detected into sub-images according to a set sliding step size, wherein the set of sliding windows includes sliding windows of at least one size.
6. The method according to claim 1, characterized in that, The training process for the image encoder and the text encoder includes: Acquire text-image pair training data, wherein the content of the text in the text-image pair is related to the content of the image; The image encoder and the text encoder are trained using a text-image comparison learning method based on the text-image pair training data.
7. The method according to claim 2, characterized in that, The determination of the target detection result of the image to be detected based on the category label, confidence score, and position of each sub-image in the image to be detected includes: Filter each first sub-image and its category label that has a confidence level exceeding a preset confidence threshold; Based on each first sub-image and its category label, and the position of each first sub-image in the image to be detected, the non-maximum suppression (NMS) method is used to filter the second sub-image and its category label from each first sub-image; The target detection result of the image to be detected is composed of the category label and confidence score of each second sub-image, and the position of the second sub-image in the image to be detected.
8. A target detection device, characterized in that, include: The image segmentation unit is used to segment the image to be detected into several sub-images; wherein, the segmentation method can be at least one of the following: rule-based segmentation, sliding window-based segmentation, and detector-assisted segmentation; The feature extraction unit is used to extract image features of each of the sub-images using a pre-configured image encoder, and to extract semantic features of each candidate category label in the category set using a pre-configured text encoder. The image encoder and the text encoder are trained based on text-image pair training data with the training objective of learning the alignment relationship between images and text. The similarity calculation unit is used to calculate the similarity between the image features of each sub-image and the semantic features of each candidate category label, and take the target candidate category label with the highest similarity as the category label of the sub-image. The detection result determination unit is used to obtain the target detection result of the image to be detected based on the category label of each of the sub-images.
9. A target detection device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the target detection method as described in any one of claims 1 to 7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the target detection method as described in any one of claims 1 to 7.