Sample generation methods, devices, image detection methods, and model training methods

Abstract

This disclosure provides a sample generation method applied in the field of machine learning, comprising: acquiring a sample image set, the sample image set including multiple first sample images and attribute labels of a first object in each first sample image; determining a first text description feature corresponding to each attribute label; determining target attribute labels from multiple initial attribute labels based on each first text description feature, thereby obtaining multiple target attribute labels; determining a detection result indicating whether an attribute representing a second object in the second sample image exists in the multiple attribute labels in the sample image set based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels; and generating training samples based on the detection results corresponding to the second sample image and the second sample image.

Description

Technical Field

[0001] This invention relates to the field of human machine learning, and more specifically, to sample generation methods, apparatus, image detection methods, and model training methods. Background Technology

[0002] Out-of-distribution detection is a key technique in machine learning, aiming to identify outliers in test data whose distribution differs significantly from that of the model training data. Its core objective is to enhance the robustness and safety of models in real-world scenarios, preventing them from making high-confidence erroneous predictions about unknown data. Related techniques utilize auxiliary outlier samples, or outlier samples constructed based on in-distribution data mining, to aid model training.

[0003] In the process of realizing the concept of this disclosure, it was found that the related technology has at least the following problems: the number of outlier samples used in the training process is small, and the features of the outlier samples used for training are different from the features of the samples used for out-of-distribution detection. Each outlier sample used for training cannot provide accurate and effective semantic information supervision, resulting in poor performance of the model in out-of-distribution detection. Summary of the Invention

[0004] In view of this, the present disclosure provides a sample generation method, comprising: acquiring a sample image set, the sample image set including multiple first sample images and attribute labels of a first object in each first sample image; determining a first text description feature corresponding to each attribute label; determining target attribute labels from multiple initial attribute labels in a knowledge base based on the first text description feature corresponding to each attribute label, thereby obtaining multiple target attribute labels, wherein the multiple attribute labels and the multiple target attribute labels are all different; determining a detection result for whether an attribute representing a second object in the second sample image exists in the multiple attribute labels in the sample image set based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels; and generating training samples based on the detection result corresponding to the second sample image and the second sample image.

[0005] According to embodiments of the present disclosure, the attribute labels in the sample image set all belong to the same attribute category; determining a target attribute label from a plurality of initial attribute labels in a knowledge base based on a first text description feature corresponding to each attribute label, to obtain a plurality of target attribute labels comprises: performing feature extraction on each of the plurality of initial attribute labels in the knowledge base respectively, to obtain a plurality of word embedding features; clustering the plurality of word embedding features to obtain a plurality of target clusters; for each first text description feature, screening a first target word embedding feature from the target clusters based on the distance between the cluster center word embedding feature of each of the plurality of target clusters and the first text description feature, to obtain a plurality of first target word embedding features; determining the initial attribute label corresponding to each of the plurality of first target word embedding features as a target attribute label, to obtain a plurality of target attribute labels.

[0006] According to embodiments of the present disclosure, determining a target attribute label from a plurality of initial attribute labels in a knowledge base based on a first text description feature corresponding to each attribute label, to obtain a plurality of target attribute labels comprises: for each first sample image: cropping the first sample image to obtain M image fragments respectively, wherein M is an integer greater than or equal to 2; based on the similarity between a sub-visual feature corresponding to each image fragment in the M image fragments and the first text description feature, screening L first target sub-visual features from the M sub-visual features as first predicted text description features, and screening L second target sub-visual features as second predicted text description features, wherein 0<L<M / 2, L is an integer, and the first target sub-visual features are different from the second target sub-visual features; obtaining a first similarity value, a second similarity value and a third similarity value corresponding to each word embedding feature respectively based on the similarity between each word embedding feature in the plurality of word embedding features and the first predicted text description feature, the similarity between each word embedding feature and the second predicted text description feature, and the similarity between each word embedding feature and the first text description feature, wherein the plurality of word embedding features are obtained by performing feature extraction on each of the plurality of initial attribute labels in the knowledge base respectively; screening a second target word embedding feature from the plurality of word embedding features based on the first similarity value, the second similarity value and the third similarity value corresponding to each word embedding feature; determining the initial attribute label corresponding to the second target word embedding feature obtained corresponding to each first sample image as a target attribute label, to obtain a plurality of target attribute labels.

[0007] According to embodiments of this disclosure, selecting a second target word embedding feature from multiple word embedding features based on a first similarity value, a second similarity value, and a third similarity value corresponding to each word embedding feature includes: weighting and summing the first similarity value, the second similarity value, and the third similarity value based on the weight coefficients corresponding to each of the first similarity value, the second similarity value, and the third similarity value corresponding to each word embedding feature to obtain a joint similarity value corresponding to each word embedding feature; and selecting a second target word embedding feature from multiple word embedding features based on the joint similarity value corresponding to each word embedding feature; wherein the weight coefficient corresponding to the first similarity value is greater than the weight coefficient corresponding to the second similarity value and the weight coefficient corresponding to the third similarity value.

[0008] According to embodiments of this disclosure, based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels, a detection result is determined to characterize whether the attribute of the second object in the second sample image exists in the multiple attribute labels of the sample image set. This includes: grouping the multiple target attribute labels to obtain N sets of target attribute labels, where N is an integer greater than or equal to 2; for each set of target attribute labels, based on the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels in the target attribute label set... The similarity between features is used to determine the similarity value corresponding to each of the multiple target attribute labels in the target attribute label set; based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, multiple fourth similarity values ​​are obtained; for each set of target attribute labels, an initial detection probability value is determined based on the similarity value corresponding to each of the multiple target attribute labels in the target attribute label set and the multiple fourth similarity values; based on the initial detection probability value corresponding to each set of target attribute labels, a target detection probability value is determined; based on the target detection probability value, a detection result is determined to determine whether the attribute used to characterize the second object in the second sample image exists in the multiple attribute labels in the sample image set.

[0009] According to another aspect of this disclosure, an image detection model training method is provided, comprising: inputting training samples including at least a second sample image into an image detection model, and outputting training detection results for characterizing whether an attribute of a second object in the second sample image exists in multiple attribute labels in a sample image set; training the image detection model based on the training detection results corresponding to the second sample image and the sample labels of the second sample image; wherein the sample labels of the second sample image are determined based on the detection results obtained by the sample generation method described above.

[0010] According to another aspect of this disclosure, an image detection method is provided, comprising: inputting an image to be detected into a pre-trained image detection model, and outputting a target detection result for characterizing whether an attribute of an object in the image to be detected exists in multiple attribute labels in a sample image set; wherein the pre-trained image detection model is obtained based on an image detection model training method.

[0011] According to embodiments of this disclosure, the image detection method further includes: when it is determined that the attribute representing the object in the image to be detected does not exist in the multiple attribute labels in the sample image set, a third target word embedding feature is selected from the multiple word embedding features based on the similarity between the visual features of the image to be detected and each of the multiple word embedding features, wherein the multiple word embedding features are obtained by feature extraction of each of the multiple initial attribute labels in the knowledge base; and the attribute label corresponding to the third target word embedding feature is added to the multiple target attribute labels.

[0012] According to embodiments of this disclosure, the image detection method further includes: when it is determined that the attributes of the image to be detected, which are characterized by the detection result, exist in multiple attribute labels in a sample image set, determining the attributes of the object in the image to be detected based on the similarity between the visual features of the image to be detected and each first text description feature.

[0013] According to another aspect of this disclosure, a sample generation apparatus is provided, characterized in that the apparatus comprises: an acquisition module for acquiring a sample image set, the sample image set including a plurality of first sample images and attribute labels of a first object in each first sample image; a first determination module for determining a first text description feature corresponding to each attribute label; a second determination module for determining target attribute labels from a plurality of initial attribute labels in a knowledge base based on the first text description feature corresponding to each attribute label, thereby obtaining a plurality of target attribute labels, wherein the plurality of attribute labels and the plurality of target attribute labels are all different; a third determination module for determining a detection result for characterizing whether an attribute of a second object in the second sample image exists in the plurality of attribute labels in the sample image set based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the plurality of attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the plurality of target attribute labels; and a fourth determination module for generating training samples based on the detection result corresponding to the second sample image and the second sample image.

[0014] According to embodiments of this disclosure, by acquiring first textual description features corresponding to multiple attribute labels in a sample image set, the dimension of the image semantic representation of the first sample image can be expanded. Based on the first textual description features corresponding to each attribute label after expanding the semantic representation dimension, target attribute labels different from the multiple attribute labels are determined from multiple initial attribute labels in the knowledge base, resulting in multiple target attribute labels, thereby enhancing the semantic difference between the multiple target attribute labels and the multiple attribute labels. Based on the similarity between the visual features of the second sample image and the in-distribution cue features, and the similarity between the visual features of the second sample image and the out-of-distribution cue features, the detection result for representing whether the attribute of the second object in the second sample image belongs to the multiple attribute labels is determined. Training samples are generated based on the detection result and the second sample image, thereby providing multiple training samples with higher quality semantic supervision and greater adversarial nature for model training, which improves the performance of the image detection model in the out-of-distribution detection process. Attached Figure Description

[0015] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0016] Figure 1 The illustration shows an application scenario of the sample generation method and apparatus according to embodiments of the present disclosure.

[0017] Figure 2 A flowchart of a sample generation method according to an embodiment of the present disclosure is shown.

[0018] Figure 3 A schematic diagram illustrating the determination of multiple target attribute labels according to an embodiment of the present disclosure is shown.

[0019] Figure 4 A schematic diagram of image fragments corresponding to a first predicted text description feature and a second predicted text description feature, according to an embodiment of the present disclosure, is shown.

[0020] Figure 5 A schematic diagram illustrating the determination of a plurality of target attribute labels according to another embodiment of the present disclosure is shown.

[0021] Figure 6 A schematic diagram of an image detection method according to an embodiment of the present disclosure is shown.

[0022] Figure 7 A schematic block diagram of a sample generation apparatus according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In realizing the concept of this disclosure, out-of-distribution detection is used to enable the model to accurately identify samples that differ from the training distribution during deployment, thereby improving the model's performance in identifying out-of-distribution samples. This is crucial for safety-critical applications such as autonomous driving and smart healthcare, as it can minimize unreliable decision-making outcomes.

[0028] Traditional out-of-distribution detection algorithms are typically trained or fine-tuned based on all training data, including post-inference-based out-of-distribution detection algorithms and training-phase regularization-based algorithms. These traditional algorithms are often driven by only a single modality and cannot utilize information from multimodal data representations, resulting in poor generalization ability.

[0029] With visual language models pre-trained on large-scale datasets demonstrating strong zero-shot generalization capabilities across multiple downstream tasks, some researchers have explored leveraging the powerful multimodal representation capabilities of visual language models for out-of-distribution detection tasks. These methods, even with few or zero-shot settings, seek supervision for model cue learning by either mining high-value outlier representations from training images within a few-shot distribution, utilizing pre-defined in-distribution cue representations, or retrieving relevant text from external corpora. Their performance even surpasses traditional algorithms using the entire training set.

[0030] Because the supervisory information utilized by the aforementioned methods largely originates from a small amount of in-distribution data (few-sample setting), or even solely from known in-distribution text labels (zero-sample setting), this supervisory information lacks diversity in its overall distribution at a macro level, resulting in limitations in the semantic space of the learned prompts. Specifically, the distribution of supervisory information constructed from a small number of in-distribution samples is insufficient to fully cover the complex and diverse semantic characteristics of out-of-distribution samples that may appear during the testing phase. Furthermore, prompts learned based on in-distribution text labels, due to their over-reliance on known semantics, may deviate significantly from the semantics of actual out-of-distribution samples, making it difficult to capture the diverse out-of-distribution representations in the real test dataset. Such limitations affect the model's ability to detect out-of-distribution samples, ultimately leading to poor generalization ability of the learned prompts.

[0031] Furthermore, conventional methods of constructing supervised information rely solely on image or text representations from the training phase, failing to efficiently utilize information from potential out-of-distribution samples during the testing phase. The discrepancy between supervised information and real out-of-distribution test samples results in insufficient accuracy of the knowledge distribution contained in the supervised information. These two issues make it difficult to further improve the performance of detecting out-of-distribution samples when the model is deployed in open scenarios.

[0032] Based on this, embodiments of this disclosure provide a sample generation method. The method includes: acquiring a sample image set, the sample image set including multiple first sample images and attribute labels of a first object in each first sample image; determining a first textual description feature corresponding to each attribute label; based on the first textual description feature corresponding to each attribute label, determining target attribute labels from multiple initial attribute labels in a knowledge base, obtaining multiple target attribute labels, wherein the multiple attribute labels and the multiple target attribute labels are all different; based on the similarity between the visual features of a second sample image and the first textual description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target textual description features corresponding to each of the multiple target attribute labels, determining a detection result for whether an attribute representing a second object in the second sample image exists in the multiple attribute labels in the sample image set; and generating training samples based on the detection result corresponding to the second sample image and the second sample image.

[0033] Figure 1 The illustration shows an application scenario of the sample generation method and apparatus according to embodiments of the present disclosure.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the sample generation method provided in this disclosure embodiment can generally be executed by server 105. The sample generation device can generally be configured on server 105. The sample generation method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the sample generation device can generally be configured in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be through Figures 2-5 The sample generation method of the present disclosure will be described in detail.

[0041] Figure 2 A flowchart of a sample generation method according to an embodiment of the present disclosure is shown.

[0042] like Figure 2 As shown, the large model deployment method 200 includes operations S210~S250.

[0043] In operation S210, a sample image set is obtained, which includes multiple first sample images and attribute labels of first objects in each first sample image.

[0044] In operation S220, the first text description feature corresponding to each attribute label is determined.

[0045] In operation S230, based on the first text description feature corresponding to each attribute tag, the target attribute tag is determined from multiple initial attribute tags in the knowledge base, resulting in multiple target attribute tags, all of which are different.

[0046] In operation S240, based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels, a detection result is determined as to whether the attribute representing the second object in the second sample image exists in the multiple attribute labels in the sample image set.

[0047] In operation S250, training samples are generated based on the detection results corresponding to the second sample image and the second sample image.

[0048] According to embodiments of this disclosure, the sample image set can be obtained from multiple open-source training datasets. The sample image set is used as training samples to train an image detection model to perform target tasks such as classification, regression, and detection. A first object in the first sample image represents the object to be identified.

[0049] For example, for a captured image with a water glass in focus, the first sample image is that image, and the first object is the water glass in the image. The attribute label for the first object can be the text "water glass" used to identify the category of the first object.

[0050] According to embodiments of this disclosure, the first text description feature represents an in-distribution cue feature associated with the attribute label. The first text description feature may be a feature generated based on the text describing the first sample image.

[0051] For example, the descriptive text for the attribute tag "vehicle" could be "This is a red vehicle, taken outdoors"; the text description for the attribute tag "dog" could be "This is a photo of an old yellow dog".

[0052] According to embodiments of this disclosure, a first text description feature can be determined from a mapping table that includes multiple attribute tags and multiple first text description features based on attribute tags.

[0053] According to embodiments of this disclosure, the first text description feature can also be determined by the following method: A first sample image and a prompt word describing the first sample image are input into a large model; the large model recognizes the first sample image and outputs text describing the first sample image. The text output by the large model is then input into a text encoder to obtain the first text description feature.

[0054] According to embodiments of this disclosure, the knowledge base can be an external corpus comprising a large amount of text. Examples include "red," "blue," "steel," "plastic," "triangle," "circle," and "ear." Target attribute labels represent out-of-distribution cue information that has low relevance to multiple attribute labels.

[0055] According to embodiments of this disclosure, multiple initial attribute tags of the knowledge base are encoded by preset text encoders to obtain features corresponding to each initial attribute tag. The similarity between each initial attribute tag and a first text description feature is then determined. Initial attribute tags with lower similarity are selected from the multiple initial attribute features as target attribute tags, resulting in multiple target attribute tags. All the multiple attribute tags and the multiple target attribute tags are different.

[0056] For example, if the knowledge base has multiple initial attribute labels such as "vehicle", "triangle", and "ear", for the first text description feature corresponding to "This is a red vehicle, taken in an outdoor environment", the similarity between "vehicle", "triangle", and "ear" and the first text description feature can be calculated respectively, thereby determining the target attribute labels "triangle" and "ear" with low similarity to the first text description feature.

[0057] According to embodiments of this disclosure, the second sample image is an image whose data distribution differs significantly from that used during the training of the image detection model. For example, in the case where the image detection model is used to classify cats and dogs, the samples used during training are images of cats or dogs. The visual features of the second sample image can be obtained by inputting the second sample image into a visual encoder.

[0058] According to embodiments of this disclosure, the similarity between the visual features of the second sample image and the first textual description features corresponding to each of the multiple attribute labels is used to characterize the probability that the second sample image belongs to an in-distribution sample. An in-distribution sample indicates that the attribute of the second object corresponding to the second sample image belongs to multiple attribute labels. For example, if the multiple attribute labels are "cat" and "dog", and the attribute of the second object is "cat" or "dog", then the second sample image belongs to an in-distribution sample.

[0059] According to embodiments of this disclosure, the target text description features represent the feature encoding matrix of the target attribute labels, and the target text description features can be obtained by inputting the target attribute labels into a text encoder. The similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels is used to characterize the probability that the second sample image belongs to an out-of-distribution sample. An in-distribution sample indicates that the attribute of the second object corresponding to the second sample image does not belong to multiple attribute labels. For example, if the multiple attribute labels are "cat" and "dog", and the attribute of the second object is "bird", the second sample image belongs to an out-of-distribution sample.

[0060] According to embodiments of this disclosure, the similarity between the visual features of the second sample image and the first text description features of each attribute label, as well as the similarity between the visual features of the second sample image and the first text description features of each attribute label, can be determined by calculating the cosine similarity between the features.

[0061] According to embodiments of this disclosure, based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute tags, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute tags, the tag with the highest visual feature similarity to the second sample image is determined, so as to determine the detection result based on whether the tag is an attribute tag.

[0062] According to embodiments of this disclosure, when the attributes of the second object in the second sample image, as represented by the detection result, exist in multiple attribute labels, the attributes of the second sample image are used as sample labels to generate training samples, i.e., training samples within the distribution, for training the image detection model to complete target tasks such as classification, regression, and detection.

[0063] According to embodiments of this disclosure, when the non-attributes of the second object in the second sample image represented by the detection result exist in multiple attribute labels, the attributes of the second sample image are used as sample labels to generate training samples, i.e., out-of-distribution training samples or outlier samples, for improving the robustness, security and generalization ability of the image detection model.

[0064] According to embodiments of this disclosure, when the detection result characterizes the second object of the second sample image as having non-attributes that exist in multiple attribute labels, "unknown", "other", etc. can also be set as sample labels for the second sample image to generate out-of-distribution training samples.

[0065] According to embodiments of this disclosure, in order to improve the quality of multiple training samples, multiple out-of-distribution training samples can be determined based on a second sample image.

[0066] According to embodiments of this disclosure, by acquiring first textual description features corresponding to multiple attribute labels in a sample image set, the dimension of the image semantic representation of the first sample image can be expanded. Based on the first textual description features corresponding to each attribute label after expanding the semantic representation dimension, target attribute labels different from the multiple attribute labels are determined from multiple initial attribute labels in the knowledge base, resulting in multiple target attribute labels, thereby enhancing the semantic difference between the multiple target attribute labels and the multiple attribute labels. Based on the similarity between the visual features of the second sample image and the in-distribution cue features, and the similarity between the visual features of the second sample image and the out-of-distribution cue features, the detection result for representing whether the attribute of the second object in the second sample image belongs to the multiple attribute labels is determined. Training samples are generated based on the detection result and the second sample image, thereby providing multiple training samples with higher quality semantic supervision and greater adversarial nature for model training, which improves the performance of the image detection model in the out-of-distribution detection process.

[0067] According to embodiments of this disclosure, attribute labels in the sample image set belong to the same attribute category; based on the first text description feature corresponding to each attribute label, target attribute labels are determined from multiple initial attribute labels in the knowledge base to obtain multiple target attribute labels, including: extracting features from each initial attribute label in the multiple initial attribute labels in the knowledge base to obtain multiple word embedding features; clustering the multiple word embedding features to obtain multiple target clusters; for each first text description feature, based on the distance between the cluster center word embedding feature of each of the multiple target clusters and the first text description feature, selecting first target word embedding features from the target clusters to obtain multiple first target word embedding features; and determining the initial attribute label corresponding to each of the multiple first target word embedding features as the target attribute label to obtain multiple target attribute labels.

[0068] According to embodiments of this disclosure, the attribute labels in the sample image set belong to the same attribute category. For example, if the sample image set is used to train the color classification ability of an image detection model, then multiple attribute labels in the sample image set belong to the color category, and multiple attribute labels of the color category may include: "red", "blue", etc.

[0069] For example, if a sample image set is used to train an image detection model to recognize objects in images, then multiple attribute labels in the sample image set can belong to object recognition categories. Multiple attribute labels for object recognition categories can include labels such as "horse", "car", and "person".

[0070] According to embodiments of this disclosure, a CLIP (Contrastive Language-Image Pre-training) text encoder is used to extract the text representation of each initial attribute label from multiple initial attribute labels in the knowledge base, and then standardizes it to obtain multiple word embedding features; the word embedding features represent the matrix representation of the initial attribute labels.

[0071] According to embodiments of this disclosure, since the external corpus contains a massive amount of text, directly clustering it may affect the efficiency of knowledge distribution construction, and thus affect the efficiency of prompt enhancement during the training phase. The mini-batch K-means algorithm can be used to cluster multiple word embedding features in batches to obtain multiple target clusters, thereby improving clustering efficiency and thus improving the efficiency of training sample generation.

[0072] According to embodiments of this disclosure, initial attribute labels with low similarity are determined from multiple initial attribute labels in the knowledge base based solely on the similarity between initial attribute labels in the knowledge base, including external knowledge, and the first text description features. This approach only considers the accuracy of determining multiple target attribute labels and lacks semantic diversity to a certain extent. It fails to comprehensively cover the various semantic information in the representation space with multiple target attribute labels. This deficiency is more pronounced when the number of attribute labels is small. Among the determined multiple target attribute labels, there will be too many texts with accuracy but similar semantics, making it impossible to obtain multiple target attribute labels that satisfy semantic type diversity. This prevents the model from generating a comprehensive and compact decision boundary, resulting in poor performance of the model in out-of-distribution detection.

[0073] Based on this, for each first text description feature, the similarity between the cluster center word embedding features of multiple target clusters and the first text description feature is calculated. This is to identify target clusters with lower similarity to the first text description feature from among the multiple target clusters, resulting in multiple target clusters with lower similarity. P word embedding features with lower similarity to the first text description feature can be selected from each of the multiple target clusters with lower similarity, resulting in multiple first target word embedding features. This reduces semantically similar text, where the value of P can be preset based on expert experience.

[0074] According to embodiments of this disclosure, multiple first target word embedding features are used to characterize diverse out-of-distribution cue features. From the i-th low-similarity target cluster... P embedding features with low similarity are identified and used as the i-th low-similarity target cluster. First target word embedding features The method can be expressed by formula (1):

[0075] (1);

[0076] in, Represents the i-th target cluster Similarity between word embedding features and first text description features.

[0077] According to embodiments of this disclosure, initial attribute labels corresponding to each of the multiple first target word embedding features are determined as target attribute labels, thereby obtaining multiple target attribute labels.

[0078] According to embodiments of this disclosure, if the knowledge distribution is constructed solely based on the initial attribute tags and the similarity between them, the knowledge distribution generated by this construction strategy, which only considers accuracy, lacks semantic diversity and cannot fully cover the various semantic information in the representation space. This deficiency is more pronounced when the number of attribute tags is small.

[0079] According to embodiments of this disclosure, features are extracted from each of the multiple initial attribute labels in the knowledge base to obtain multiple word embedding features. These multiple word embedding features are then clustered to obtain multiple target clusters. For each first text description feature, first target word embedding features are selected from the target clusters based on the distance between the cluster center word embedding features and the first text description features, resulting in multiple first target word embedding features. This reduces the computational cost of distance calculation and increases the semantic diversity of the multiple first target word embedding features. The initial attribute labels corresponding to each of the multiple first target word embedding features are determined as target attribute labels, resulting in multiple target attribute labels. Multiple target attribute labels are determined based on the principles of accuracy and diversity, further increasing the semantic diversity of the multiple target attribute labels.

[0080] Figure 3 A schematic diagram illustrating the determination of multiple target attribute labels according to an embodiment of the present disclosure is shown.

[0081] like Figure 3 As shown, multiple initial attribute labels in the knowledge base are clustered to obtain target cluster 1, target cluster 2, target cluster 3, and target cluster 4. For the first text description features of each attribute label in the sample image set, for example, the first text description feature of "cow", the distance between the cluster center word embedding feature and the first text description feature of each target cluster is determined. First target word embedding features are then selected from the target clusters to obtain multiple first target word embedding features. Specifically, the initial attribute label corresponding to the cluster center word embedding feature of target cluster 1 is "airplane", the initial attribute label corresponding to the cluster center word embedding feature of target cluster 2 is "jeans", the initial attribute label corresponding to the cluster center word embedding feature of target cluster 3 is "locust tree", and the initial attribute label corresponding to the cluster center word embedding feature of target cluster 4 is "wood".

[0082] For target cluster 4, the cosine similarity between the first text description feature of "cat" in the target attribute label and the cluster center embedding word feature corresponding to the initial attribute label "wood" is determined. If the cosine similarity is greater than a preset threshold, the cluster center embedding word feature of target cluster 4 is used as the first target word embedding feature.

[0083] Using the same method, first target word embedding features are selected from target cluster 1, target cluster 2, and target cluster 3 to obtain multiple first target word embedding features.

[0084] The initial attribute labels corresponding to the embedding features of multiple first target words are determined as target attribute labels, resulting in multiple target attribute labels.

[0085] According to embodiments of the present disclosure, based on the first text description features corresponding to each attribute label, a target attribute label is determined from a plurality of initial attribute labels in a knowledge base respectively, so as to obtain a plurality of target attribute labels, the method includes: for each first sample image: cropping the first sample image to obtain M image fragments respectively, wherein M is an integer greater than or equal to 2; based on the similarity between the sub-visual feature corresponding to each image fragment in the M image fragments and the first text description feature, screening L first target sub-visual features from the M sub-visual features as first predicted text description features, and screening L second target sub-visual features as second predicted text description features, wherein 0<L<M / 2, L is an integer, and the first target sub-visual features are different from the second target sub-visual features; according to the similarity between each word embedding feature in a plurality of word embedding features and the first predicted text description feature, the similarity between each word embedding feature and the second predicted text description feature, and the similarity between each word embedding feature and the first text description feature, obtaining a first similarity value, a second similarity value and a third similarity value corresponding to each word embedding feature respectively, wherein the plurality of word embedding features are obtained by performing feature extraction on each initial attribute label in the plurality of initial attribute labels of the knowledge base respectively; based on the first similarity value, the second similarity value and the third similarity value corresponding to each word embedding feature, screening a second target word embedding feature from the plurality of word embedding features; determining the initial attribute label corresponding to the second target word embedding feature obtained corresponding to each first sample image as a target attribute label, so as to obtain a plurality of target attribute labels.

[0086] According to embodiments of the present disclosure, a first sample image is cropped to obtain M image fragments respectively, where M is an integer greater than or equal to 2. A pre-trained visual encoder of the CLIP model can be used to determine the sub-visual feature corresponding to each image fragment in the M image fragments. Cropping an image into M fragments enables the visual encoder to capture features from both global and local perspectives: details that may be ignored by global features, such as the shape, texture and color of an object, can be explicitly retained through the sub-visual features.

[0087] According to embodiments of the present disclosure, the cosine similarity between the sub-visual feature corresponding to each of the M image fragments and the first text description feature is calculated, L sub-visual features with lower cosine similarity are screened from the M sub-visual features to obtain L first target sub-visual features, and the L first target sub-visual features are used as the first predicted text description features.

[0088] According to embodiments of the present disclosure, L sub-visual features with higher cosine similarity are screened from the M sub-visual features to obtain L second target sub-visual features, and the L second target sub-visual features are used as second predicted text description features.

[0089] According to embodiments of this disclosure, the first predicted text description feature is similar to the first text description feature and often appears in the feature neighborhood of the first object, but is not a feature of the first object. For example, if the first object in the first sample image is a dog, after cropping the first sample image, M image segments are obtained. The sub-visual features of some image segments represent background-related content, but are not features of the dog. Background-related content, such as lawns and frisbees, often appears alongside the first object, the dog.

[0090] According to embodiments of this disclosure, the second target sub-visual feature represents features that are highly correlated with the features representing the first object. For example, if the first object in the first sample image is a goldfish, and for the M image segments obtained after cropping, the second target sub-visual feature can be a feature used to represent the goldfish's fins, scales, eyes, etc.

[0091] Figure 4 A schematic diagram of image fragments corresponding to a first predicted text description feature and a second predicted text description feature, according to an embodiment of the present disclosure, is shown.

[0092] like Figure 4 As shown, the first sample image 400 is cropped to obtain multiple image segments. Among them, the sub-visual features corresponding to image segment 401 have fewer features related to the first object "cat" and are mainly used to characterize the features of the sofa where the cat is located. The similarity between the sub-visual features corresponding to image segment 401 and the first text description features will be low. The sub-visual features corresponding to image segment 401 can be the first predicted text description features.

[0093] Image fragment 402 includes features that are more related to the first object "cat". The sub-visual features corresponding to image fragment 402 and the first text description features have a high similarity. The sub-visual features corresponding to image fragment 402 can be used as the second predicted text description features.

[0094] According to embodiments of this disclosure, a text encoder using CLIP is used to extract the text representation of each initial attribute label from multiple initial attribute labels in a knowledge base, and the representation is standardized to obtain multiple word embedding features; the word embedding features are matrix representations of the initial attribute labels.

[0095] According to embodiments of this disclosure, the first similarity value, the second similarity value, and the third similarity value can be determined by calculating the cosine similarity between features.

[0096] According to embodiments of this disclosure, a first similarity value can be used to characterize the degree of association between an initial attribute label and an image segment used to characterize features of a non-first object. For example, "sofa" and... Figure 4The relevance of image fragment 401 to "dog" and Figure 4 The relevance of image fragment 401 in the middle.

[0097] According to embodiments of this disclosure, the second similarity can be used to characterize the degree of association between the initial attribute label and the image segment used to characterize the features of the first object. For example, "cat" and... Figure 4 The relevance of image fragment 402 to "fish" and Figure 4 The relevance of image fragment 402 in the middle.

[0098] According to embodiments of this disclosure, a third similarity can be used to characterize the degree of association between the initial attribute label and the text content used to characterize the features of the first object. For example, the similarity between "cat" and "a yellow cat", and the similarity between "fish" and "a yellow cat".

[0099] According to embodiments of this disclosure, based on a first similarity value, a second similarity value, and a third similarity value corresponding to the word embedding feature, a word embedding feature with a higher first similarity value and lower second and third similarity values ​​is selected from multiple word embedding features as the second target word embedding feature.

[0100] According to embodiments of this disclosure, the initial attribute label corresponding to the second target word embedding feature obtained for each first sample image is determined as the target attribute label, resulting in multiple target attribute labels.

[0101] According to embodiments of this disclosure, a first sample image is cropped to obtain M image segments. Sub-visual features corresponding to the M image segments are used to retain fine-grained image information in the first sample image, improving the feature representation capability of the sub-visual features in minute details. Based on the similarity between the sub-visual features corresponding to each image segment and the first text description feature, a first predicted text description feature with high similarity to the first text description feature and a second predicted text description feature with low similarity to the first text description feature are obtained. Based on the similarity values ​​(first similarity value, second similarity value, and third similarity value) between the word embedding feature and the first predicted text description feature, the second predicted text description feature, and the first text description feature, a second target word embedding feature is selected from multiple word embedding features to reduce the overlap between multiple initial attribute labels and the first description, improving the accuracy and effectiveness of the target attribute labels, thereby providing training samples carrying high-value supervision signals for the training process.

[0102] Figure 5 A schematic diagram illustrating the determination of a plurality of target attribute labels according to another embodiment of the present disclosure is shown.

[0103] like Figure 5As shown, the first sample image includes outlier representations, which are generally background features unrelated to the first object, and also includes in-distribution representations, which are generally features representing the first object. The first sample image is cropped to obtain M image segments to separate the in-distribution representations and outlier representations.

[0104] The initial attribute tags in the knowledge base are divided into groups: Group 1: triangle, square, circle, bar; Group 2: silk, steel, plastic, wood; Group 3: arm, ear, leg, tail; Group 4: red, blue, gray. The cluster center feature of Group 1 can be bar; the cluster center feature of Group 2 can be tail; the cluster center feature of Group 3 can be gray; and the cluster center feature of Group 4 can be wood.

[0105] Based on the similarity between the sub-visual features corresponding to each of the M image segments and the first text description features, L first target sub-visual features are selected from the M sub-visual features as the first predicted text description features, i.e. outlier features, and L second target sub-visual features are selected as the second predicted text description features, i.e. in-distribution features.

[0106] Based on the similarity between each word embedding feature and the first predicted text description feature, the similarity between each word embedding feature and the second predicted text description feature, and the similarity between each word embedding feature and the first text description feature, the first similarity value, the second similarity value, and the third similarity value corresponding to each word embedding feature are obtained respectively. Among them, the multiple word embedding features are obtained by extracting features from each initial attribute label in the multiple initial attribute labels of the knowledge base.

[0107] Based on the first, second, and third similarity values ​​corresponding to each word embedding feature, a second target word embedding feature is selected from multiple word embedding features to obtain a second target word embedding feature that maximizes the similarity with outlier features and minimizes the similarity with in-distribution representations and multiple attribute labels. Thus, the initial attribute label corresponding to the second target word embedding feature obtained for each first sample image is determined as the target attribute label, resulting in multiple target attribute labels.

[0108] for Figure 5 Based on the first sample image "cat" in the dataset, and using the first, second, and third similarity values, a second target word embedding feature is determined that maximizes the similarity with the background and minimizes the similarity with the first object and the attribute label "cat".

[0109] According to embodiments of this disclosure, selecting a second target word embedding feature from multiple word embedding features based on a first similarity value, a second similarity value, and a third similarity value corresponding to each word embedding feature includes: weighting and summing the first similarity value, the second similarity value, and the third similarity value based on the weight coefficients corresponding to each of the first similarity value, the second similarity value, and the third similarity value corresponding to each word embedding feature to obtain a joint similarity value corresponding to each word embedding feature; and selecting a second target word embedding feature from multiple word embedding features based on the joint similarity value corresponding to each word embedding feature; wherein the weight coefficient corresponding to the first similarity value is greater than the weight coefficient corresponding to the second similarity value and the weight coefficient corresponding to the third similarity value.

[0110] According to embodiments of this disclosure, the first similarity value, the second similarity value, and the third similarity value corresponding to each word embedding feature are weighted and summed to obtain the total similarity value of each word embedding feature. The top X word embedding features with high joint similarity values ​​are selected from multiple word embedding features as the second target word embedding features.

[0111] According to embodiments of this disclosure, the weight coefficient corresponding to the second similarity value is greater than 0, and the weight coefficient corresponding to the first similarity value and the weight coefficient corresponding to the third similarity value are less than 0.

[0112] According to embodiments of this disclosure, the weight coefficient corresponding to the first similarity value can be set to be much larger than the weight coefficients corresponding to the second and third similarity values. For example, the weight coefficient corresponding to the first similarity value is 0.99, the weight coefficient corresponding to the second similarity value is 0.005, and the weight coefficient corresponding to the third similarity value is 0.005.

[0113] According to embodiments of this disclosure, the weight coefficient corresponding to the first similarity value for determining joint similarity is set to be greater than the weight coefficient corresponding to the second similarity value and the weight coefficient corresponding to the third similarity value. By increasing the similarity ratio between each word embedding feature in the joint similarity value and the first predicted text description feature, the overlap between multiple initial attribute labels and the first description is reduced, thereby improving the accuracy and effectiveness of the target attribute label.

[0114] According to embodiments of this disclosure, multiple target attribute tags are determined. The formula can be expressed by formula (2):

[0115] (2);

[0116] in, Represents multiple word embedding features, The joint similarity value representing word embedding features. Indicates the number of target attribute tags.

[0117] According to embodiments of this disclosure, based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels, a detection result is determined to characterize whether the attribute of the second object in the second sample image exists in the multiple attribute labels of the sample image set. This includes: grouping the multiple target attribute labels to obtain N sets of target attribute labels, where N is an integer greater than or equal to 2; for each set of target attribute labels, based on the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels in the target attribute label set... The similarity between features is used to determine the similarity value of each target attribute label in the target attribute label set; based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, multiple fourth similarity values ​​are obtained; for each set of target attribute labels, an initial detection probability value is determined based on the similarity value and the multiple fourth similarity values ​​corresponding to each of the multiple target attribute labels in the target attribute label set; based on the initial detection probability value corresponding to each set of target attribute labels, a target detection probability value is determined; based on the target detection probability value, a detection result is determined to determine whether the attribute used to characterize the second object in the second sample image exists in the multiple attribute labels in the sample image set.

[0118] According to embodiments of this disclosure, multiple target attribute labels can be grouped to obtain N sets of target attribute labels, where N is an integer greater than or equal to 2, following the prompt word grouping integration strategy used in NegLabel.

[0119] For example, multiple predefined labels, such as "red" and "stripes", are divided into N groups based on the correlation between the semantic target attribute labels and the predefined labels. For example, group 1 is color: {red, blue, green}; group 2 is texture: {stripes, checks, solid color}.

[0120] According to embodiments of this disclosure, for each set of target attribute labels, the cosine similarity between the visual features of the second sample image and the target text description features corresponding to each target attribute label is calculated, and the similarity value corresponding to each of the multiple target attribute labels in the target attribute label set is determined.

[0121] According to embodiments of this disclosure, the mean similarity is used to characterize the correlation between the attributes of the second sample image and the target attribute label set. For example, the correlation between "yellow" and group 1 colors: {red, blue, green}. For example, the correlation between "yellow" and group 2 textures: {stripes, checks, solid colors}.

[0122] According to embodiments of this disclosure, for each attribute label, the cosine similarity between the visual features of the second sample image and the first text description features corresponding to the attribute label is calculated to determine a fourth similarity value corresponding to each attribute label.

[0123] According to embodiments of this disclosure, the fourth similarity value characterizes the correlation between the attributes of the second sample image and the attribute label, for example, the correlation between "dog" and "yellow cat"; for example, the correlation between "cat" and "yellow cat".

[0124] According to embodiments of this disclosure, for each target attribute label set, the similarity values ​​corresponding to each of the multiple target attribute labels in the target attribute label set are summed to obtain the total similarity to the target attribute label set. The sum of the total similarity to the target attribute label set and the sum of the multiple fourth similarity values ​​is used as the denominator, and the sum of the multiple fourth similarity values ​​is used as the numerator to calculate the initial detection probability value of the second sample image relative to the target attribute label set, thereby determining the initial detection probability values ​​corresponding to the second sample image and the multiple target attribute label sets respectively.

[0125] According to embodiments of this disclosure, the average value of the initial detection probability values ​​corresponding to the second sample image and the multiple target attribute label sets is calculated, and the average value of the N initial detection probability values ​​is determined as the target detection probability value of the second sample image, wherein the target detection probability value represents the probability that the attribute of the second object in the second sample image belongs to the multiple attribute labels.

[0126] According to embodiments of this disclosure, when the target detection probability value is greater than a preset value, it is determined that the attribute used to characterize the second object in the second sample image exists among multiple attribute labels in the sample image set; when the detection probability value is less than the preset value, it is determined that the attribute used to characterize the second object in the second sample image does not exist among multiple attribute labels in the sample image set. The preset value can be, for example, 0.5 or 0.6.

[0127] According to embodiments of this disclosure, multiple target attribute labels are grouped to obtain N sets of target attribute labels. Based on the initial detection probability values ​​corresponding to the second sample image and the multiple target attribute label sets, the probability that the attribute of the second object in the second sample image belongs to multiple attribute labels is calculated, i.e., the target detection probability value. This determines the detection result used to characterize whether the attribute of the second object in the second sample image exists in the multiple attribute labels in the sample image set. This method of calculating the target detection probability value on a per-set basis to determine the detection result reduces the influence of a single target attribute label on the determination of the detection result, thereby reducing the probability of inaccurate detection results due to semantic overlap between target attribute labels and attribute labels, and improving the accuracy of the detection result.

[0128] According to another aspect of this disclosure, an image detection model training method is provided, comprising: inputting training samples including at least a second sample image into an image detection model, and outputting training detection results for characterizing whether an attribute of a second object in the second sample image exists in a plurality of attribute labels in a sample image set; training the image detection model based on the training detection results corresponding to the second sample image and the sample labels of the second sample image; wherein the sample labels of the second sample image are determined by the detection results obtained by the sample generation method of any of the above embodiments.

[0129] According to embodiments of this disclosure, the image detection model can be trained based on a sample image set.

[0130] For example, the sample image set includes multiple first sample images with attribute labels of "cat" and "dog". The second sample image with the attribute "cat" is input into the image detection model, and the image detection model outputs the training detection result "yes".

[0131] For example, the sample image set includes multiple first sample images with attribute labels of "cat" and "dog". The second sample image with the attribute of "car" is input into the image detection model, and the image detection model outputs the training detection result "no".

[0132] According to embodiments of this disclosure, when the attributes of the second object exist in multiple attribute labels in a sample image set, the sample label of the second sample image can be the attribute of the second object.

[0133] According to embodiments of this disclosure, when the attributes of the second object do not exist in the multiple attribute labels of the sample image set, the sample label of the second sample image can be the attribute of the second object.

[0134] According to embodiments of this disclosure, if the attribute of the second object does not exist in the multiple attribute labels of the sample image set, the sample label of the second sample image may also be "other".

[0135] According to embodiments of this disclosure, a second sample image is input into an image detection model. The image detection model performs feature extraction and target recognition on the image based on its own network structure and parameters, and outputs preliminary recognition results. Subsequently, the preliminary recognition results are compared with the sample labels in detail. By calculating the errors between the two, such as classification errors and localization errors, the error signals are propagated back to each network layer of the model layer by layer using a backpropagation algorithm, thereby making targeted adjustments to the parameters in the image detection model.

[0136] According to embodiments of this disclosure, an image detection model is trained using a second sample image and sample labels obtained by the sample generation method of any of the above embodiments. This provides the image detection model with multiple training samples that have higher quality semantic supervision and are more adversarial, thereby improving the performance of the image detection model in the out-of-distribution detection process.

[0137] According to another aspect of this disclosure, an image detection method is provided, comprising: inputting an image to be detected into a pre-trained image detection model, and outputting a target detection result for characterizing whether an attribute of an object in the image to be detected exists in multiple attribute labels in a sample image set; wherein the pre-trained image detection model is obtained by the above-described image detection model training method.

[0138] According to embodiments of this disclosure, the image to be detected can be an image used for out-of-distribution detection by an image detection model. The target detection result is determined using a pre-trained image detection model.

[0139] For example, multiple attribute labels include "cat" and "dog". The image to be detected with the attribute "cat" is input into the image detection model, and the pre-trained image detection model outputs the target detection result, which represents that the attribute of the object in the image to be detected exists in multiple attribute labels in the sample image set.

[0140] For example, multiple attribute labels include "cat" and "dog". The image to be detected is input into the image detection model with the attribute of "tiger". The pre-trained image detection model outputs the target detection result, which indicates that the attribute of the object in the image to be detected does not exist in multiple attribute labels in the sample image set.

[0141] According to embodiments of this disclosure, out-of-distribution detection is performed on a pre-trained image detection model using the image to be detected to determine the target detection result, thereby determining the robustness of the image detection model.

[0142] According to embodiments of this disclosure, the image detection method further includes: when it is determined that the attribute representing the object in the image to be detected does not exist in the multiple attribute labels in the sample image set, a third target word embedding feature is selected from the multiple word embedding features based on the similarity between the visual features of the image to be detected and each of the multiple word embedding features, wherein the multiple word embedding features are obtained by feature extraction of each of the multiple initial attribute labels in the knowledge base; and the attribute label corresponding to the third target word embedding feature is added to the multiple target attribute labels.

[0143] According to embodiments of this disclosure, the visual features of the image to be detected can be obtained by inputting the image to be detected into a pre-trained visual encoder.

[0144] According to embodiments of this disclosure, there is a lot of redundancy in the multiple initial attribute labels of the knowledge base. Based on the visual features of the image to be detected and the cosine similarity between each word embedding feature in the multiple word embedding features, word embedding features with higher cosine similarity are selected from the multiple word embedding features as the third target word embedding features.

[0145] According to embodiments of this disclosure, attribute labels corresponding to the embedding features of the third target word are added to multiple target attribute labels to expand the number of target attribute labels. The images to be detected corresponding to the attribute labels can be used as training samples to improve the robustness of the model.

[0146] Figure 6 A schematic diagram of an image detection method according to an embodiment of the present disclosure is shown.

[0147] like Figure 6 As shown, multiple word embedding features are obtained by extracting features from each of the multiple initial attribute labels in the knowledge base. When it is determined that the attribute representing the object in the image to be detected 1 does not exist in the multiple attribute labels in the sample image set, a third target word embedding feature is selected from the multiple word embedding features based on the similarity between the visual features of the image to be detected 1 and each word embedding feature in the multiple word embedding features. The attribute labels "drooping tail", "wolf" and "pointed ears" corresponding to the third target word embedding feature are added to the multiple target attribute labels.

[0148] For the image n to be detected, if it is determined that the attribute representing the object in the image n does not exist in the multiple attribute labels in the sample image set, a third target word embedding feature is selected from multiple word embedding features based on the same method; the attribute labels "car" and "steel" corresponding to the third target word embedding feature are added to the multiple target attribute labels to expand the multiple target attribute labels.

[0149] In cases where the attributes of objects in other images to be detected, including image 2, do not exist in the multiple attribute labels in the sample image set, multiple target attribute labels can be expanded using the above method.

[0150] According to embodiments of this disclosure, the image detection method further includes: when it is determined that the attributes of the image to be detected, which are characterized by the detection result, exist in multiple attribute labels in a sample image set, determining the attributes of the object in the image to be detected based on the similarity between the visual features of the image to be detected and each first text description feature.

[0151] According to embodiments of this disclosure, when it is determined that the attribute of the image to be detected, as represented by the detection result, exists in multiple attribute labels in the sample image set, the similarity between the visual features of the image to be detected and each first text description feature is calculated, the first text description feature with the highest similarity to the visual features of the image to be detected is determined, and the attribute label corresponding to the first text description feature is determined as the attribute of the object in the image to be detected.

[0152] According to embodiments of this disclosure, by directly comparing visual features with textual descriptive features, image content and semantic information can be aligned in a unified high-dimensional space. This alignment fully utilizes the generalization ability of pre-trained image detection models to capture deep semantic relationships between images and text, thereby improving the accuracy of fine-grained attribute recognition.

[0153] Based on the above sample generation method, this disclosure also provides a sample generation apparatus. The following will be combined with... Figure 7 The device is described in detail.

[0154] Figure 7 A schematic block diagram of a sample generation apparatus according to an embodiment of the present disclosure is shown.

[0155] like Figure 7 As shown, the sample generation device 700 of this embodiment includes an acquisition module 710, a first determination module 720, a second determination module 730, a third determination module 740, and a fourth determination module 750.

[0156] The acquisition module is used to acquire a sample image set, which includes multiple first sample images and attribute labels of a first object in each first sample image;

[0157] The first determining module is used to determine the first text description feature corresponding to each attribute label;

[0158] The second determining module is used to determine the target attribute label from multiple initial attribute labels in the knowledge base based on the first text description feature corresponding to each attribute label, thereby obtaining multiple target attribute labels, and the multiple attribute labels and multiple target attribute labels are all different.

[0159] The third determining module is used to determine the detection result of whether the attribute of the second object in the second sample image exists in the multiple attribute labels in the sample image set based on the similarity between the visual features of the second sample image and the first text description features corresponding to the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to the multiple target attribute labels.

[0160] A fourth determining module, configured to generate training samples based on a detection result corresponding to a second sample image and the second sample image.

[0161] According to an embodiment of the present disclosure, attribute labels in a sample image set all belong to the same attribute category; the second determining module includes: a feature determining unit, a cluster determining unit, a first screening unit, and a first determining unit.

[0162] The feature determining unit is configured to perform feature extraction on each of a plurality of initial attribute labels in a knowledge base respectively to obtain a plurality of word embedding features.

[0163] The cluster determining unit is configured to cluster the plurality of word embedding features to obtain a plurality of target clusters.

[0164] The first screening unit is configured to, for each first text description feature, screen a first target word embedding feature from the target clusters based on distances between cluster center word embedding features of each of the plurality of target clusters and the first text description feature, to obtain a plurality of first target word embedding features.

[0165] The first determining unit is configured to determine initial attribute labels respectively corresponding to the plurality of first target word embedding features as target attribute labels, to obtain a plurality of target attribute labels.

[0166] According to an embodiment of the present disclosure, the second determining module includes, for each first sample image: an image cropping unit, a second screening unit, a similarity determining unit, a third screening unit, and a second determining unit.

[0167] The image cropping unit is configured to crop the first sample image to obtain M image fragments respectively, wherein M is an integer greater than or equal to 2;

[0168] The second screening unit is configured to screen L first target sub-visual features from M sub-visual features as first predicted text description features, and screen L second target sub-visual features as second predicted text description features based on similarity between a sub-visual feature corresponding to each of the M image fragments and a first text description feature, wherein 0 < L < M / 2 and L is an integer, and the first target sub-visual features are different from the second target sub-visual features;

[0169] The similarity determining unit is configured to obtain a first similarity value, a second similarity value and a third similarity value corresponding to each word embedding feature respectively based on similarity between each word embedding feature in the plurality of word embedding features and the first predicted text description feature, similarity between each word embedding feature and the second predicted text description feature, and similarity between each word embedding feature and the first text description feature, wherein the plurality of word embedding features are obtained by performing feature extraction on each of the plurality of initial attribute labels in the knowledge base respectively.

[0170] The third filtering unit is used to filter the second target word embedding feature from multiple word embedding features based on the first similarity value, the second similarity value and the third similarity value corresponding to each word embedding feature.

[0171] The second determining unit is used to determine the initial attribute label corresponding to the second target word embedding feature obtained for each first sample image as the target attribute label, thereby obtaining multiple target attribute labels.

[0172] According to embodiments of this disclosure, the third screening unit includes: a first determining subunit and a second determining subunit.

[0173] The first determining subunit is used to perform a weighted summation of the first similarity value, the second similarity value, and the third similarity value based on the weight coefficients corresponding to the first similarity value, the second similarity value, and the third similarity value corresponding to each word embedding feature, to obtain the joint similarity value corresponding to each word embedding feature.

[0174] The second determining subunit is used to select the second target word embedding feature from multiple word embedding features based on the joint similarity value corresponding to each word embedding feature. The weight coefficient corresponding to the first similarity value is greater than the weight coefficient corresponding to the second similarity value and the weight coefficient corresponding to the third similarity value.

[0175] According to embodiments of this disclosure, the third determining module includes: a tag grouping unit, a third determining unit, a fourth determining unit, a fifth determining unit, a sixth determining unit, and a seventh determining unit.

[0176] The tag grouping unit is used to group multiple target attribute tags to obtain N sets of target attribute tags, where N is an integer greater than or equal to 2.

[0177] The third determining unit is used to determine the similarity value of each target attribute label in the target attribute label set based on the similarity between the visual features of the second sample image and the target text description features corresponding to each target attribute label in the target attribute label set.

[0178] The fourth determining unit is used to obtain multiple fourth similarity values ​​based on the similarity between the visual features of the second sample image and the first text description features corresponding to the multiple attribute labels.

[0179] The fifth determining unit is used to determine the initial detection probability value for each target attribute label set based on the similarity value corresponding to each of the multiple target attribute labels in the target attribute label set and multiple fourth similarity values.

[0180] The sixth determining unit is used to determine the target detection probability value based on the initial detection probability value corresponding to each set of target attribute labels;

[0181] The seventh determining unit is used to determine, based on the target detection probability value, the detection result of whether the attribute representing the second object in the second sample image exists in multiple attribute labels in the sample image set.

[0182] According to embodiments of this disclosure, any multiple modules of the acquisition module 710, the first determining module 720, the second determining module 730, the third determining module 740, and the fourth determining module 750 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 710, the first determining module 720, the second determining module 730, the third determining module 740, and the fourth determining module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the following modules can be implemented, at least partially, as a computer program module that performs a corresponding function when the computer program module is run.

[0183] It should be noted that the large model deployment device part in the embodiments of this disclosure corresponds to the large model deployment method part in the embodiments of this disclosure. For a detailed description of the large model deployment device part, please refer to the large model deployment method part, which will not be repeated here.

[0184] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0185] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A sample generation method, characterized in that, The method includes: Obtain a sample image set, the sample image set including a plurality of first sample images and attribute labels of a first object in each first sample image; Determine the first text description feature corresponding to each of the attribute tags; Based on the first text description feature corresponding to each attribute tag, initial attribute tags with low similarity are selected from the features corresponding to each of the multiple initial attribute tags in the knowledge base as target attribute tags, resulting in multiple target attribute tags. The multiple attribute tags and the multiple target attribute tags are all different. Based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels, the label with the highest visual feature similarity to the second sample image is determined, so as to obtain the detection result of whether the attribute of the second object in the second sample image exists in the multiple attribute labels in the sample image set, depending on whether the label is an attribute label. Training samples are generated based on the detection results corresponding to the second sample image and the second sample image.

2. The method according to claim 1, characterized in that, The attribute labels in the sample image set all belong to the same attribute category; Based on the first text description features corresponding to each attribute tag, initial attribute tags with low similarity are selected from the features corresponding to each of the multiple initial attribute tags in the knowledge base as target attribute tags, resulting in multiple target attribute tags, including: Feature extraction is performed on each of the initial attribute tags in the knowledge base to obtain multiple word embedding features; Multiple target clusters are obtained by clustering the multiple word embedding features; For each of the first text description features, based on the distance between the cluster center word embedding features of the multiple target clusters and the first text description features, first target word embedding features are selected from the target clusters to obtain multiple first target word embedding features; The initial attribute labels corresponding to the embedding features of the plurality of first target words are determined as the target attribute labels, thereby obtaining the plurality of target attribute labels.

3. The method according to claim 1, characterized in that, Based on the first text description features corresponding to each attribute tag, initial attribute tags with low similarity are selected from the features corresponding to each of the multiple initial attribute tags in the knowledge base as target attribute tags, resulting in multiple target attribute tags, including: For each of the first sample images: The first sample image is cropped to obtain M image segments, where M is an integer greater than or equal to 2; Based on the similarity between the sub-visual feature corresponding to each of the M image segments and the first text description feature, L first target sub-visual features with lower similarity are screened from the M sub-visual features to serve as first predicted text description features, and L second target sub-visual features with higher similarity are screened to serve as second predicted text description features, wherein 0<L<M / 2, L is an integer, and the first target sub-visual features are different from the second target sub-visual features; According to the similarity between each word embedding feature in a plurality of word embedding features and the first predicted text description feature, the similarity between each said word embedding feature and the second predicted text description feature, and the similarity between each said word embedding feature and the first text description feature, a first similarity value, a second similarity value and a third similarity value corresponding to each said word embedding feature are obtained respectively, wherein the plurality of word embedding features are obtained by performing feature extraction on each of the plurality of initial attribute labels of the knowledge base respectively; Based on the first similarity value, the second similarity value and the third similarity value corresponding to each said word embedding feature, a word embedding feature with a higher first similarity value, and lower second similarity value and lower third similarity value is screened from the plurality of word embedding features as a second target word embedding feature; The initial attribute label corresponding to the second target word embedding feature obtained corresponding to each first sample image is determined as the target attribute label, so as to obtain a plurality of said target attribute labels.

4. The method according to claim 3, characterized in that, The step of screening the second target word embedding feature from the plurality of word embedding features based on the first similarity value, the second similarity value and the third similarity value corresponding to each said word embedding feature comprises: Based on the respective weight coefficients corresponding to the first similarity value, the second similarity value and the third similarity value corresponding to each said word embedding feature, weighted summation is performed on the first similarity value, the second similarity value and the third similarity value to obtain a joint similarity value corresponding to each said word embedding feature; Based on the joint similarity value corresponding to each said word embedding feature, the second target word embedding feature is screened and obtained from the plurality of word embedding features; Wherein, the weight coefficient corresponding to the first similarity value is greater than the weight coefficient corresponding to the second similarity value and the weight coefficient corresponding to the third similarity value.

5. The method according to claim 1, characterized in that, The step of determining the label with the highest similarity to the visual feature of the second sample image based on the similarity between the visual feature of the second sample image and the first text description feature corresponding to each of the plurality of attribute labels, and the similarity between the visual feature of the second sample image and the target text description feature corresponding to each of the plurality of target attribute labels, so as to obtain a detection result representing whether the attribute of the second object of the second sample image exists among the plurality of attribute labels in the sample image set according to whether the label is an attribute label comprises: The multiple target attribute tags are grouped to obtain N sets of target attribute tags, where N is an integer greater than or equal to 2; For each set of target attribute labels, the similarity value corresponding to each of the multiple target attribute labels in the target attribute label set is determined based on the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels in the target attribute label set. Based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, multiple fourth similarity values ​​are obtained; For each set of target attribute tags, an initial detection probability value is determined based on the similarity values ​​corresponding to each of the multiple target attribute tags in the set and the multiple fourth similarity values. The target detection probability value is determined based on the initial detection probability value corresponding to each set of target attribute labels; Based on the target detection probability value, the detection result is determined to identify whether the attribute of the second object in the second sample image exists in the multiple attribute labels of the sample image set.

6. A method for training an image detection model, characterized in that, The method includes: The training samples, including at least the second sample image, are input into the image detection model, and the output is the training detection result used to characterize whether the attribute of the second object in the second sample image exists in multiple attribute labels in the sample image set. The image detection model is trained based on the training detection results corresponding to the second sample image and the sample labels of the second sample image; Wherein, the sample label of the second sample image is determined based on the detection result obtained by the method according to any one of claims 1 to 5. When the detection result indicates that the attribute of the second object of the second sample image exists in multiple attribute labels, the attribute of the second sample image is used as the sample label. When the detection result indicates that the attribute of the second object of the second sample image does not exist in multiple attribute labels, the attribute of the second sample image is either "unknown" or "other" as the sample label.

7. An image detection method, characterized in that, The method includes: The image to be detected is input into a pre-trained image detection model, which outputs a target detection result that characterizes whether the attributes of the object in the image to be detected exist in multiple attribute labels in the sample image set. The pre-trained image detection model is obtained based on the method described in claim 6.

8. The method according to claim 7, characterized in that, The method further includes: If it is determined that the attribute representing the object in the image to be detected does not exist in the multiple attribute labels in the sample image set, a third target word embedding feature is selected from the multiple word embedding features based on the similarity between the visual features of the image to be detected and each word embedding feature in the multiple word embedding features. The multiple word embedding features are obtained by feature extraction from each initial attribute label in the multiple initial attribute labels of the knowledge base. The attribute label corresponding to the embedding feature of the third target word is added to multiple target attribute labels.

9. The method according to claim 8, characterized in that, The method further includes: If it is determined that the detection result indicates that the attribute of the image to be detected exists in multiple attribute labels in the sample image set, the attribute of the object in the image to be detected is determined based on the similarity between the visual features of the image to be detected and each of the first text description features.

10. A sample generation device, characterized in that, The device includes: The acquisition module is used to acquire a sample image set, which includes multiple first sample images and attribute labels of a first object in each first sample image; The first determining module is used to determine the first text description feature corresponding to each of the attribute tags; The second determining module is used to select initial attribute tags with low similarity from the features corresponding to each of the multiple initial attribute tags in the knowledge base as target attribute tags based on the first text description features corresponding to each attribute tag, thereby obtaining multiple target attribute tags, wherein the multiple attribute tags and the multiple target attribute tags are all different; The third determining module is used to determine the label with the highest visual similarity to the second sample image based on the similarity between the visual features of the second sample image and the first text description features corresponding to each of the multiple attribute labels, and the similarity between the visual features of the second sample image and the target text description features corresponding to each of the multiple target attribute labels, so as to obtain a detection result for whether the attribute of the second object in the second sample image exists in the multiple attribute labels in the sample image set, depending on whether the label is an attribute label. The fourth determining module generates training samples based on the detection results corresponding to the second sample image and the second sample image.

Images