Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method, which comprises: obtaining a to-be-processed data set, the to-be-processed data set comprising a plurality of sample pictures and corresponding real labels of the sample pictures; for each sample picture, obtaining a preset number of easily confused labels, the easily confused labels not referring to the same object as the real label; based on the real label and the easily confused label, combining the sample picture with a preset format to obtain difficult sample data corresponding to the sample picture, each sample picture corresponding to at least one difficult sample data; and based on the difficult sample data, constructing a difficult sample data set corresponding to the to-be-processed data set. The present application can solve the problem that a large language model is difficult to quickly find the differentiated rules between similar different objects from complex data, so as to improve the large language model to quickly learn the differentiation of different objects from data and improve the output accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Open-world image classification is a popular research and application area in AI, aiming to identify a wide variety of real-world objects. Unlike closed-set image classification, open-world image classification is not limited by the number of object categories pre-defined during model training, allowing for the generation of one or more labels / names for the objects to be identified. One of the mainstream methods for open-world image classification is based on cross-modal generative models (e.g., LLaVA, LLaVA-Next, Emu) using Visual Question Answering (VQA). To excel in this task, collecting rich and comprehensive training data is crucial for model training. This training data must effectively represent the characteristics and differences of various real-world objects. However, due to the vast variety of real-world objects and the small differences between some, it is difficult to distinguish similar objects, making it challenging for models to quickly discover patterns of differentiation among similar objects from a large dataset. Furthermore, high-quality data collection requires significant human and material resources. Summary of the Invention

[0003] This invention provides a data processing method aimed at addressing the problem that large language models struggle to quickly discover patterns of differentiation between similar objects from massive datasets. This method aims to improve the speed at which large language models learn the differences between objects from data, thereby increasing the accuracy of model output. For each sample image in the dataset to be processed, a predetermined number of easily confused labels are obtained. Based on the true labels and the easily confused labels, the sample images are labeled using a predetermined format to obtain hard sample data corresponding to the sample images. Based on this hard sample data, a hard sample dataset corresponding to the dataset to be processed is constructed. This method can improve the accuracy of the model in open-domain object recognition tasks, especially in challenging scenarios where two types of objects are visually very similar, resulting in a significant improvement.

[0004] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising the following steps:

[0005] Obtain the dataset to be processed, which includes several sample images and the corresponding real labels of the sample images;

[0006] For each sample image, a preset number of easily confused labels are obtained, wherein the easily confused labels and the real labels do not refer to the same object;

[0007] Based on the real labels and the easily confused labels, the sample images are labeled in a preset format to obtain difficult sample data corresponding to the sample images. Each sample image corresponds to at least one difficult sample data.

[0008] Based on the difficult sample data, a difficult sample dataset corresponding to the dataset to be processed is constructed.

[0009] Optionally, for each of the sample images, obtaining a preset number of easily confused tags includes:

[0010] For each of the sample images, a first preset number of first candidate tags are obtained based on image retrieval methods. The first candidate tags do not refer to the same target as the real tags.

[0011] And / or, for each of the sample images, a second preset number of second candidate labels are obtained based on a multimodal large model approach, wherein the second candidate labels do not refer to the same target as the real labels;

[0012] For each sample image, a preset number of easily confused tags are determined based on the first candidate tag and / or the second candidate tag.

[0013] Optionally, obtaining a first preset number of first candidate tags for each sample image based on image retrieval includes:

[0014] Extract the key features of the sample images;

[0015] Based on the key features of the sample images, a search is performed in the image database to obtain multiple target search images and detection tags corresponding to the target search images. The image database includes search images and search tags corresponding to the search images.

[0016] For each of the sample images, a first preset number of first candidate tags are determined based on the search tags corresponding to the multiple target search images.

[0017] Optionally, for each of the sample images, determining a first preset number of first candidate tags based on the tags corresponding to multiple target retrieval images includes:

[0018] For each of the sample images, synonym tags are extracted from the search tags corresponding to the multiple target search images to obtain multiple non-synonymous tags;

[0019] Obtain the number of tags for the non-synonym tags, and the number of target search images for each non-synonym tag "maternal and infant";

[0020] If the number of tags is greater than the first preset number, then, in descending order of the number of target search images, the search tags with a quantity equal to the first preset number are selected as the first candidate tags from the non-synonym tags.

[0021] Optionally, for each of the sample images, obtaining a second preset number of second candidate labels based on a multimodal large model approach includes:

[0022] For each sample image, based on the real labels, multiple object names are output through a large text model to obtain an initial set of easily confused labels. Each sample image corresponds to one such initial set of easily confused labels.

[0023] Based on the sample images and the corresponding initial easily confused tag set, the item names in the initial easily confused tag set are filtered by a multimodal large model to obtain a second preset number of item names as second candidate tags.

[0024] Optionally, the multimodal large model includes a visual encoder and a text encoder. Based on the sample images and the corresponding initial easily confused label set, the multimodal large model filters the item names in the initial easily confused label set to obtain a second preset number of item names as second candidate labels, including:

[0025] The visual encoder is used to extract features from the sample image to obtain the visual features corresponding to the sample image.

[0026] Furthermore, the text encoder extracts features from the item names in the initial set of easily confused tags to obtain the text features corresponding to the item names, with each item name corresponding to one text feature;

[0027] Calculate the similarity between the visual features and the text features, and select a second preset number of the item names as second candidate tags from the initial set of easily confused tags in descending order of similarity.

[0028] Optionally, the step of combining the labels of the sample images based on the real labels and the easily confused labels in a preset format to obtain the difficult sample data corresponding to the sample images includes:

[0029] A question template is constructed based on a preset format. The question template includes an image item, a question prompt item, multiple tag selection items, and answer options.

[0030] The sample image is entered into the image field. The real label is randomly selected from multiple label selection options. A corresponding number of easily confused labels are randomly selected and entered into the remaining label selection options. The real label is entered into the answer options to obtain the difficult sample data corresponding to the sample image.

[0031] Secondly, embodiments of the present invention also provide a data processing apparatus, the data processing apparatus comprising:

[0032] The first acquisition module is used to acquire the dataset to be processed, which includes a number of sample images and the real labels corresponding to the sample images.

[0033] The second acquisition module is used to acquire a preset number of easily confused labels for each sample image, wherein the easily confused labels and the real labels do not refer to the same object;

[0034] The first processing module is used to combine the labels of the sample images in a preset format based on the real labels and the easily confused labels to obtain the difficult sample data corresponding to the sample images, wherein each sample image corresponds to at least one difficult sample data.

[0035] The second processing module is used to construct a difficult sample dataset corresponding to the dataset to be processed based on the difficult sample data.

[0036] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the data processing method provided in embodiments of the present invention.

[0037] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the data processing method provided in the embodiments of the invention.

[0038] In this embodiment of the invention, a dataset to be processed is obtained, which includes several sample images and corresponding ground truth labels. For each sample image, a preset number of easily confused labels are obtained, where the easily confused labels and ground truth labels do not refer to the same object. Based on the ground truth labels and easily confused labels, the sample images are labeled in a preset format to obtain hard sample data corresponding to the sample images, with each sample image corresponding to at least one hard sample data. Based on the hard sample data, a hard sample dataset corresponding to the dataset to be processed is constructed. This invention solves the problem that large language models struggle to quickly discover the differences between similar objects from complex data, thereby improving the accuracy of large language models in quickly learning the differences between different objects from data and enhancing the accuracy of model output. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, Figure 1 This is a flowchart of a data processing method provided in an embodiment of the present invention, which includes the following steps:

[0045] 101. Obtain the dataset to be processed.

[0046] In this embodiment of the invention, the above-described data processing method can be applied to an object recognition platform, which can be built using a server or a distributed server. The object recognition platform includes a database, a data processing application, a knowledge database, a large language model, or a calling interface for a large language model. The dataset to be processed can be obtained through the database interface or through user uploads.

[0047] The dataset to be processed includes several sample images and their corresponding ground truth labels. Specifically, this dataset can be a VQA dataset or a dataset annotated and uploaded by the user.

[0048] The sample images mentioned above can be of any type, such as images containing objects like daily necessities, natural scenery, plants, animals, and vehicles.

[0049] The aforementioned true labels can be understood as the actual category or attribute of the object or scene in the image. For example, if an image contains an apple, then the true label for this image is "apple"; if an image contains an electric toothbrush, then the true label for this image is "electric toothbrush," and so on.

[0050] 102. For each sample image, obtain a preset number of easily confused labels.

[0051] In this embodiment of the invention, after obtaining the dataset to be processed, a preset number of easily confused tags that are most similar to the content of the sample image can be obtained for each sample image.

[0052] The aforementioned easily confused labels do not refer to the same object as the actual labels.

[0053] The preset quantity mentioned above can be a quantity pre-set by the system, such as 1, 3, 5, 10, etc.

[0054] The aforementioned easily confused labels can be understood as labels that are highly similar to each other, easily leading to misclassification by the model. This means that the image is close to other images that do not belong to the same class, according to a certain metric in the feature space. Therefore, for any given image, using image features as a medium, similar samples can be found throughout the training set. If these samples do not have the same label as the image, these different labels are the easily confused labels.

[0055] A preset number of easily confused tags can be obtained by inputting sample images into an image database for retrieval. The aforementioned image database includes the search image and its corresponding search tags.

[0056] A predetermined number of easily confused tags can be obtained by inputting sample images into a multimodal model for retrieval. This multimodal model is trained on a vast amount of world text data and includes knowledge not found in the VQA training set, enabling it to discover even more potential easily confused tags.

[0057] Furthermore, easily confused labels retrieved from the image database can be combined with easily confused labels retrieved from the multimodal model to obtain easily confused labels corresponding to the sample images.

[0058] 103. Based on real labels and easily confused labels, combine the labels of sample images in a preset format to obtain difficult sample data corresponding to the sample images.

[0059] In this embodiment of the invention, each sample image corresponds to at least one difficult sample data.

[0060] The aforementioned preset format can be understood as a training data format pre-set by the system. The preset format includes the combination between sample images and labels. In the preset format, sample images are combined with real labels and easily confused labels to form corresponding difficult sample data.

[0061] The above-mentioned label combination can be understood as a process of combining real labels with easily confused labels according to certain rules, so as to process and analyze data more flexibly and quickly.

[0062] The aforementioned difficult-sample data includes both real labels and easily confused labels.

[0063] 104. Based on the difficult sample data, construct a difficult sample dataset corresponding to the dataset to be processed.

[0064] In this embodiment of the invention, a difficult sample dataset, which is prone to misclassification, can be constructed based on the difficult sample data and corresponds to the dataset to be processed. This difficult sample dataset contains true labels and easily confused or misclassified labels. Training the model using this difficult sample dataset can improve the model's accuracy and robustness.

[0065] In this embodiment of the invention, by adding difficult sample data as one of the training datasets to the model training, the present invention can help to quickly improve the model's recognition ability in open-world image classification tasks.

[0066] In this embodiment of the invention, a dataset to be processed is obtained, which includes several sample images and corresponding ground truth labels. For each sample image, a preset number of easily confused labels are obtained, where the easily confused labels and ground truth labels do not refer to the same object. Based on the ground truth labels and easily confused labels, the sample images are labeled in a preset format to obtain hard sample data corresponding to the sample images, with each sample image corresponding to at least one hard sample data. Based on the hard sample data, a hard sample dataset corresponding to the dataset to be processed is constructed. This invention solves the problem that large language models struggle to quickly discover the differences between similar objects from complex data, thereby improving the accuracy of large language models in quickly learning the differences between different objects from data and enhancing the accuracy of model output.

[0067] It is understood that in the specific implementation of this application, data such as image data, tag data, and knowledge data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of word segmentation tools and large-scale language models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0068] Optionally, in the step of obtaining a preset number of easily confused labels for each sample image, a first preset number of first candidate labels can be obtained for each sample image based on image retrieval; and / or, a second preset number of second candidate labels can be obtained for each sample image based on a multimodal large model; and a preset number of easily confused labels can be determined for each sample image based on the first candidate labels and / or the second candidate labels.

[0069] In this embodiment of the invention, the first candidate label and the real label do not refer to the same target. The second candidate label and the real label do not refer to the same target.

[0070] The training dataset is very large. We can use the efficient vector database Faiss as a retrieval tool. Each sample image is input into the image database for retrieval, and a first preset number of images similar to each sample image are obtained. The first candidate label is then extracted from these similar images.

[0071] The first preset quantity is a quantity pre-set by the system, which can be 5, 10, 15, etc., and can be represented by K1; the second preset quantity can be 5, 10, 15, etc., and can be represented by K2.

[0072] The candidate tags mentioned above can be tags for similar images.

[0073] The aforementioned multimodal large model is trained using massive amounts of world text data, incorporating knowledge not included in the VQA training set, and discovering more potential easily confused tags. This multimodal large model could be the ChatGPT multimodal large model.

[0074] The aforementioned easily confused labels refer to situations where there is a high degree of similarity between labels, which can easily lead to misclassification by the model.

[0075] It should be noted that for each sample image, image retrieval can be used to find other images in the image database that are similar in content but have different labels. The labels of these images are the first candidate labels. The first candidate labels are different from the real labels of the original sample images, but in certain contexts or conditions, they may be mistaken for or confused with the objects or scenes referred to by the real labels.

[0076] In addition to obtaining the first candidate label based on image retrieval, a multimodal large model can be used to automatically analyze the content of each sample image and generate a second, predetermined number of second candidate labels. These second candidate labels are also different from the true labels, but they reflect the confusion or misjudgment that the model may produce when processing the image.

[0077] Furthermore, by using a first preset number of first candidate tags from image retrieval, a preset number of easily confused tags can be obtained for each sample image; by using a second preset number of second candidate tags from the multimodal large model, a preset number of easily confused tags can be obtained for each sample image; by using the first preset number of first candidate tags from image retrieval and / or the second preset number of second candidate tags from the multimodal large model, a preset number of easily confused tags can be obtained for each sample image. The aforementioned preset numbers can be 1, 3, 5, 10, etc., specifically 1 to K1 (the first preset number), or 1 to K2 (the second preset number), or 1 to (K1+K2) (the sum of the first and second preset numbers).

[0078] Optionally, in the step of obtaining a first preset number of first candidate labels for each sample image based on image retrieval, key features of the sample image can be extracted; based on the key features of the sample image, a search can be performed in the image database to obtain multiple target retrieval images and detection labels corresponding to the target retrieval images; for each sample image, based on the retrieval labels corresponding to the multiple target retrieval images, a first preset number of first candidate labels can be determined.

[0079] In this embodiment of the invention, the image database includes search images and corresponding search tags. By comparing the features of sample images with the features of images in the database, visually similar image features can be found.

[0080] Furthermore, the sample images can be input into a feature extraction model to extract features and obtain the key features of the sample images.

[0081] The feature extraction model described above can be a feature extraction model built based on deep learning or machine learning. For example, DINOv2 can be used as the feature extraction model, and the class token of the feature layer before the MLP layer can be selected as the feature representation of the input image. The aforementioned feature layer can capture the key features of the image.

[0082] Feature retrieval in image databases can be performed using an inverted indexing method based on Euclidean distance. Euclidean distance is the straight-line distance between two points. The inverted indexing method is an information retrieval method that quickly retrieves image tags containing image features by mapping image features to image tags containing those features. This Euclidean distance-based inverted indexing method can improve the efficiency of similarity searches on each sample image in a large-scale dataset.

[0083] Optionally, in the step of determining a first preset number of first candidate tags for each sample image based on the tags corresponding to multiple target search images, for each sample image, synonym tags can be removed from the search tags corresponding to multiple target search images to obtain multiple non-synonymous tags; the number of non-synonymous tags and the number of target search images corresponding to each non-synonymous tag can be obtained; if the number of tags is greater than the first preset number, then in descending order of the number of target search images, the first preset number of search tags can be selected as the first candidate tags from the non-synonymous tags.

[0084] In this embodiment of the invention, the purpose of removing synonym tags is to avoid counting duplicate tags with the same meaning, ensuring the uniqueness and accuracy of each tag. Due to the existence of synonyms in Chinese nouns, two tags with different texts may not represent two different objects, such as "car" and "vehicle," "book" and "library," "tomato" and "tomato," etc. Synonym tags are removed because they actually refer to the same object.

[0085] Furthermore, ChatGPT can be used to filter out easily confused tags of the same type, eliminating pseudo-confusing tags that refer to the same object. Specifically, the original tags of the image and the easily confused tags can be paired into noun pairs and provided to ChatGPT, leveraging its powerful textual knowledge to filter out synonyms. The prompt template used is: "Please determine whether A and B refer to the same object. Answer yes if yes, otherwise answer no." Using ChatGPT's "no" answer as the condition, the truly confusing tags are filtered out.

[0086] For example, the potential set of easily confused labels for "vacuum cleaner" includes "steam cleaner", "air purifier", "electric mop", "mite remover", and "dust absorber". Vacuum cleaners come in small handheld forms as well as upright mop-like forms. If it is a picture of a handheld vacuum cleaner, it is unlikely to be visually similar to an "electric mop", but it will have some similarities to a "mite remover". Therefore, the label "electric mop" is not suitable as an easily confused label for this picture.

[0087] In one possible implementation, for each input image, if the number of easily confused tags exceeds K1, then they are filtered. Specifically, all easily confused tags are sorted by the number of images in descending order of quantity, and the top K1 tags are removed as the final result to ensure that the number of easily confused tags for each image does not exceed K1 (where K1 is an experimental empirical value).

[0088] For example, if a search for an image of an "electric toothbrush" yields 20 images that satisfy the Euclidean distance less than the threshold d and whose label is not "electric toothbrush," then 10 of these images are labeled "water flosser," 5 are labeled "electric massager," 3 are labeled "facial cleansing device," and 2 are labeled "electric shaver." If K1 is 3, then the easily confused labels are "water flosser," "electric massager," and "facial cleansing device," while "electric shaver" is filtered out.

[0089] Optionally, in the step of obtaining a second preset number of second candidate labels for each sample image using a multimodal large model approach, for each sample image, based on the real labels, multiple object names can be output through a text large model to obtain an initial easily confused label set; based on the sample image and the corresponding initial easily confused label set, the item names in the initial easily confused label set can be filtered through a multimodal large model to obtain a second preset number of item names as second candidate labels.

[0090] In this embodiment of the invention, each sample image corresponds to an initial set of easily confused labels.

[0091] The aforementioned large-scale text model can output multiple object names or keywords related to the real labels of sample images. This large-scale text model can be a multimodal large-scale model like ChatGPT.

[0092] When generating easily confused labels using a large text model, appropriate cue word templates are needed. For example, cue word templates could be: "Please tell me the names of T objects that are visually similar to A," or "What things look very similar to A? Please list T names," etc. A is the name of the object to be generated as the easily confused label, and T is a suitable empirical value.

[0093] The aforementioned initial easily confused label set is a label set that includes all image labels, including the label of each sample image in the training dataset and the easily confused image labels.

[0094] It should be noted that the "confusing" label refers to images; only images that are visually similar should be considered as "confusing."

[0095] Furthermore, the large text model only uses object names as input to obtain an initial set of easily confused tags. Therefore, it is necessary to filter potential easily confused tags based on the content of the image. These potential easily confused tags may be invalid.

[0096] In one possible implementation, for example, the potential set of easily confused labels for "vacuum cleaner" includes "steam cleaner", "air purifier", "electric mop", "mite remover", and "dust absorber". Vacuum cleaners can be small handheld or upright mop-like. If it is a picture of a handheld vacuum cleaner, it is unlikely to be visually similar to "electric mop", but it will have some similarities to "mite remover". Therefore, the label "electric mop" is not suitable as an easily confused label for this picture.

[0097] Therefore, a multimodal model can be used to filter potentially confusing labels and eliminate inappropriate labels. The aforementioned large multimodal model can be the multimodal large model Siglip.

[0098] Optionally, the multimodal large model includes a visual encoder and a text encoder. In the step of filtering the item names in the initial easily confused label set based on sample images and the corresponding initial easily confused label set to obtain a second preset number of item names as second candidate labels, the visual encoder can extract features from the sample images to obtain the visual features corresponding to the sample images; and the text encoder can extract features from the item names in the initial easily confused label set to obtain the text features corresponding to the item names; the similarity between the visual features and the text features is calculated, and the second preset number of item names are selected from the initial easily confused label set as second candidate labels in descending order of similarity.

[0099] In this embodiment of the invention, each item name corresponds to a text feature.

[0100] The aforementioned visual encoder is responsible for extracting visual features from sample images, such as color, shape, and texture.

[0101] The aforementioned text encoder is responsible for extracting textual features, such as semantic information, from the item names in the initial set of easily confused labels.

[0102] The similarity between visual features and text features can be calculated using cosine similarity or other similarity methods. Then, in order of decreasing similarity, a second preset number of item names can be selected as second candidate labels from the initial set of easily confused labels.

[0103] Specifically, a multimodal large-scale visual encoder is used to extract image features, and then a multimodal large-scale text encoder is used to extract text features for the potential easily confused labels of the image. Then, the image-text similarity is calculated. Only easily confused labels with similarity greater than a preset threshold are retained, and they are sorted in descending order of similarity, with a maximum of K2 labels retained. If no easily confused label has an image-text similarity threshold greater than the preset threshold with the image, then the result with the highest similarity is taken as the easily confused label of the image.

[0104] Optionally, in the step of combining sample images with labels based on real labels and easily confused labels in a preset format to obtain difficult sample data corresponding to the sample images, a question template can be constructed based on the preset format; a sample image is filled in the image item; a real label is randomly selected from multiple label selection items; a corresponding number of easily confused labels are randomly selected and filled in the remaining label selection items; and a real label is filled in the answer option to obtain difficult sample data corresponding to the sample images.

[0105] In this embodiment of the invention, the question template includes an image item, a question prompt word item, multiple tag selection items, and answer options.

[0106] The above preset format can be understood as a training data format pre-set by the system. It can be a question template in the form of multiple-choice questions, as shown below:

[0107] “ Please tell me what the object in the picture above is? A. Object Name 1 B. Object Name 2 C. Object Name 3 D. Object Name 4 Answer:

[0108] in," The image is the prompt for the question "What is the object in the image above?". The tags for "A. Object Name 1", "B. Object Name 2", "C. Object Name 3", and "D. Object Name 4" are the answer choices.

[0109] In one embodiment, the aforementioned question template is used as a reference example input for the multimodal large model, allowing it to generate more VQA training data templates in similar formats. The instruction used to invoke the multimodal large model is: "Please provide me with a prompt word template for constructing VQA data." Reference example: Please tell me what the object in the above image is? A. Object Name 1 B. Object Name 2 C. Object Name 3 D. Object Name 4 Answer: . A new VQA training data template can be obtained by calling the function once, and the output result can replace the reference sample input for this time. The multimodal large model can then be called again; this process can be repeated Q times to obtain Q new and different VQA training data templates. It should be noted that, to ensure the diversity of training data, Q can be 1000 or a larger value.

[0110] For each sample image, the hard sample data includes a true label and 1 to (K1+K2) easily confused labels. The training steps for generating hard sample data include: Step 1, firstly, randomly sample a VQA training data template; Step 2, then randomly sample 3 easily confused labels from the easily confused labels. If there are not enough easily confused labels, sample directly from all training set labels until the number is 3; Step 3, then randomly fill the true label of the image into one of the options A, B, C, and D, and fill in the remaining 3 options after randomly sorting the easily confused labels.

[0111] For example:

[0112] Suppose one of the training image sets is "electric toothbrush," and the true label is "electric toothbrush." ​​The three easily confused labels obtained through sampling are "water flosser," "electric massager," and "facial cleansing device." The sampled VQA training data template is... What appears in the picture? A. Option 1 B. Option 2 C. Option 3 D. Option 4 Answer: . Then, randomly fill in the true labels and easily confused labels into the four options A, B, C, and D: What is shown in the picture? A. Water flosser B. Electric massager C. Electric toothbrush D. Facial cleansing device Answer:

[0113] Steps 1, 2, and 3 create a text-image question-and-answer multiple-choice question with a correct answer. The answer is the letter of the option containing the true label (in the example above, the answer is C). This text-image question-and-answer data in a question-and-answer format can then be used to train a cross-modal generative model. Because the options in the multiple-choice question come from easily confused labels, they are all hard samples. For example, "electric toothbrush" and "water flosser" are two different objects, but apart from the differences in the top parts, their appearances are visually similar. This leads the model to easily misidentify an image of a "water flosser" as an "electric toothbrush" when performing open-domain object recognition tasks. Therefore, for this image of a "water flosser," "electric toothbrush" is an easily confused label, and this image is a typical hard sample.

[0114] like Figure 2 As shown, an embodiment of the present invention provides a data processing apparatus, which includes:

[0115] The first acquisition module 201 is used to acquire a dataset to be processed, which includes a number of sample images and the real labels corresponding to the sample images.

[0116] The second acquisition module 202 is used to acquire a preset number of easily confused labels for each sample image, wherein the easily confused labels and the real labels do not refer to the same object;

[0117] The first processing module 203 combines the labels of the real labels and the easily confused labels in a preset format to obtain the difficult sample data corresponding to the sample images, and each sample image corresponds to at least one difficult sample data.

[0118] The second processing module 204 is used to construct a difficult sample dataset corresponding to the dataset to be processed based on the difficult sample data.

[0119] Optionally, the second acquisition module 202 is further configured to, for each sample image, acquire a first preset number of first candidate labels based on image retrieval, wherein the first candidate labels do not refer to the same target as the real labels; and / or, for each sample image, acquire a second preset number of second candidate labels based on a multimodal large model approach, wherein the second candidate labels do not refer to the same target as the real labels; and for each sample image, determine a preset number of easily confused labels based on the first candidate labels and / or the second candidate labels.

[0120] Optionally, the second acquisition module 202 is further configured to extract key features of the sample image; based on the key features of the sample image, perform a search in the image database to obtain multiple target search images and detection tags corresponding to the target search images, wherein the image database includes search images and search tags corresponding to the search images; for each sample image, determine a first preset number of first candidate tags based on the search tags corresponding to the multiple target search images.

[0121] Optionally, the second acquisition module 202 is further configured to, for each sample image, extract synonym tags from the search tags corresponding to the multiple target search images to obtain multiple non-synonymous tags; acquire the number of tags of the non-synonymous tags, and the number of target search images for each non-synonymous tag (maternal and infant); if the number of tags is greater than a first preset number, then, in descending order of the number of target search images, select the search tags in the non-synonymous tags that equal the first preset number as first candidate tags.

[0122] Optionally, the second acquisition module 202 is further configured to, for each sample image, output multiple object names based on the real labels using a text big model to obtain an initial easily confused label set, with each sample image corresponding to one initial easily confused label set; and based on the sample image and the corresponding initial easily confused label set, filter the item names in the initial easily confused label set using a multimodal big model to obtain a second preset number of item names as second candidate labels.

[0123] Optionally, the multimodal large model includes a visual encoder and a text encoder. The second acquisition module 202 is further configured to extract features from the sample image using the visual encoder to obtain visual features corresponding to the sample image; and to extract features from the item names in the initial easily confused label set using the text encoder to obtain text features corresponding to the item names, with each item name corresponding to one text feature; calculate the similarity between the visual features and the text features, and select a second preset number of item names from the initial easily confused label set as second candidate labels in descending order of similarity.

[0124] Optionally, the first processing module 203 is further configured to construct a question template based on a preset format. The question template includes an image item, a question prompt word item, multiple tag selection items, and answer options. The sample image is filled into the image item, a tag selection item is randomly selected from the multiple tag selection items and the real tag is filled in, a corresponding number of easily confused tags are randomly selected and filled in the remaining tag selection items, and the real tag is filled in the answer options to obtain difficult sample data corresponding to the sample image.

[0125] like Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above data processing methods.

[0126] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to execute data processing methods, wherein:

[0127] The processor 301 executes the calculator program containing the data processing method stored in the memory 302, performing the following steps:

[0128] Obtain the dataset to be processed, which includes several sample images and the corresponding real labels of the sample images;

[0129] For each sample image, a preset number of easily confused labels are obtained, wherein the easily confused labels and the real labels do not refer to the same object;

[0130] Based on the real labels and the easily confused labels, the sample images are labeled in a preset format to obtain difficult sample data corresponding to the sample images. Each sample image corresponds to at least one difficult sample data.

[0131] Based on the difficult sample data, a difficult sample dataset corresponding to the dataset to be processed is constructed.

[0132] Optionally, the process executed by processor 301 to obtain a preset number of easily confused labels for each of the sample images includes:

[0133] For each of the sample images, a first preset number of first candidate tags are obtained based on image retrieval methods. The first candidate tags do not refer to the same target as the real tags.

[0134] And / or, for each of the sample images, a second preset number of second candidate labels are obtained based on a multimodal large model approach, wherein the second candidate labels do not refer to the same target as the real labels;

[0135] For each sample image, a preset number of easily confused tags are determined based on the first candidate tag and / or the second candidate tag.

[0136] Optionally, the process executed by processor 301 to obtain a first preset number of first candidate tags for each sample image based on an image retrieval method includes:

[0137] Extract the key features of the sample images;

[0138] Based on the key features of the sample images, a search is performed in the image database to obtain multiple target search images and detection tags corresponding to the target search images. The image database includes search images and search tags corresponding to the search images.

[0139] For each of the sample images, a first preset number of first candidate tags are determined based on the search tags corresponding to the multiple target search images.

[0140] Optionally, the step of processor 301 performing the step of determining a first preset number of first candidate tags for each of the sample images based on the tags corresponding to the multiple target retrieval images includes:

[0141] For each of the sample images, synonym tags are extracted from the search tags corresponding to the multiple target search images to obtain multiple non-synonymous tags;

[0142] Obtain the number of tags for the non-synonym tags, and the number of target search images for each non-synonym tag "maternal and infant";

[0143] If the number of tags is greater than the first preset number, then, in descending order of the number of target search images, the search tags with a quantity equal to the first preset number are selected as the first candidate tags from the non-synonym tags.

[0144] Optionally, the processor 301 performs the following step: for each sample image, obtaining a second preset number of second candidate labels based on a multimodal large model approach, including:

[0145] For each sample image, based on the real labels, multiple object names are output through a large text model to obtain an initial set of easily confused labels. Each sample image corresponds to one such initial set of easily confused labels.

[0146] Based on the sample images and the corresponding initial easily confused tag set, the item names in the initial easily confused tag set are filtered by a multimodal large model to obtain a second preset number of item names as second candidate tags.

[0147] Optionally, the multimodal large model includes a visual encoder and a text encoder. The processor 301 executes the process of filtering the item names in the initial easily confused tag set based on the sample image and the corresponding initial easily confused tag set using the multimodal large model to obtain a second preset number of item names as second candidate tags, including:

[0148] The visual encoder is used to extract features from the sample image to obtain the visual features corresponding to the sample image.

[0149] Furthermore, the text encoder extracts features from the item names in the initial set of easily confused tags to obtain the text features corresponding to the item names, with each item name corresponding to one text feature;

[0150] Calculate the similarity between the visual features and the text features, and select a second preset number of the item names as second candidate tags from the initial set of easily confused tags in descending order of similarity.

[0151] Optionally, the step of processor 301 performing tag combination on the sample image based on the real label and the easily confused label in a preset format to obtain difficult sample data corresponding to the sample image includes:

[0152] A question template is constructed based on a preset format. The question template includes an image item, a question prompt item, multiple tag selection items, and answer options.

[0153] The sample image is entered into the image field. The real label is randomly selected from multiple label selection options. A corresponding number of easily confused labels are randomly selected and entered into the remaining label selection options. The real label is entered into the answer options to obtain the difficult sample data corresponding to the sample image.

[0154] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the data processing method or application-side data processing method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0155] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0156] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A data processing method, characterized in that, The method includes the following steps: Obtain the dataset to be processed, which includes several sample images and the corresponding real labels of the sample images; For each sample image, a preset number of easily confused labels are obtained, wherein the easily confused labels and the real labels do not refer to the same object; Based on the real labels and the easily confused labels, the sample images are labeled in a preset format to obtain difficult sample data corresponding to the sample images. Each sample image corresponds to at least one difficult sample data. Based on the difficult sample data, a difficult sample dataset corresponding to the dataset to be processed is constructed.

2. The data processing method as described in claim 1, characterized in that, For each of the sample images, obtaining a preset number of easily confused tags includes: For each of the sample images, a first preset number of first candidate tags are obtained based on image retrieval methods. The first candidate tags do not refer to the same target as the real tags. And / or, for each of the sample images, a second preset number of second candidate labels are obtained based on a multimodal large model approach, wherein the second candidate labels do not refer to the same target as the real labels; For each sample image, a preset number of easily confused tags are determined based on the first candidate tag and / or the second candidate tag.

3. The data processing method as described in claim 2, characterized in that, For each of the sample images, obtaining a first preset number of first candidate tags based on image retrieval includes: Extract the key features of the sample images; Based on the key features of the sample images, a search is performed in the image database to obtain multiple target search images and detection tags corresponding to the target search images. The image database includes search images and search tags corresponding to the search images. For each of the sample images, a first preset number of first candidate tags are determined based on the search tags corresponding to the multiple target search images.

4. The data processing method as described in claim 3, characterized in that, For each of the sample images, determining a first preset number of first candidate tags based on the tags corresponding to multiple target retrieval images includes: For each of the sample images, synonym tags are removed from the search tags corresponding to the multiple target search images to obtain multiple non-synonymous tags; Obtain the number of tags for the non-synonym tags, and the number of target search images corresponding to each non-synonym tag; If the number of tags is greater than the first preset number, then, in descending order of the number of target search images, the search tags with a quantity equal to the first preset number are selected as the first candidate tags from the non-synonym tags.

5. The data processing method as described in claim 2, characterized in that, For each of the sample images, obtaining a second preset number of second candidate labels based on a multimodal large model includes: For each sample image, based on the real labels, multiple object names are output through a large text model to obtain an initial set of easily confused labels. Each sample image corresponds to one such initial set of easily confused labels. Based on the sample images and the corresponding initial easily confused tag set, the item names in the initial easily confused tag set are filtered by a multimodal large model to obtain a second preset number of item names as second candidate tags.

6. The data processing method as described in claim 4, characterized in that, The multimodal large model includes a visual encoder and a text encoder. Based on the sample images and the corresponding initial easily confused label set, the multimodal large model filters the item names in the initial easily confused label set to obtain a second preset number of item names as second candidate labels, including: The visual encoder is used to extract features from the sample image to obtain the visual features corresponding to the sample image. Furthermore, the text encoder extracts features from the item names in the initial set of easily confused tags to obtain the text features corresponding to the item names, with each item name corresponding to one text feature; Calculate the similarity between the visual features and the text features, and select a second preset number of the item names as second candidate tags from the initial set of easily confused tags in descending order of similarity.

7. The data processing method according to any one of claims 1 to 6, characterized in that, The step of combining labels of the sample images based on the real labels and the easily confused labels in a preset format to obtain difficult sample data corresponding to the sample images includes: A question template is constructed based on a preset format. The question template includes an image item, a question prompt item, multiple tag selection items, and answer options. The sample image is entered into the image field. The real label is randomly selected from multiple label selection options. A corresponding number of easily confused labels are randomly selected and entered into the remaining label selection options. The real label is entered into the answer options to obtain the difficult sample data corresponding to the sample image.

8. A data processing apparatus, characterized in that, The data processing device includes: The first acquisition module is used to acquire the dataset to be processed, which includes a number of sample images and the real labels corresponding to the sample images. The second acquisition module is used to acquire a preset number of easily confused labels for each sample image, wherein the easily confused labels and the real labels do not refer to the same object; The first processing module is used to combine the labels of the sample images in a preset format based on the real labels and the easily confused labels to obtain the difficult sample data corresponding to the sample images, wherein each sample image corresponds to at least one difficult sample data. The second processing module is used to construct a difficult sample dataset corresponding to the dataset to be processed based on the difficult sample data.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the data processing method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 7.

Images