A data filtering method, device, equipment and medium

Abstract

The application discloses a data filtering method and device, equipment and medium, and relates to the technical field of data processing. The image data and text data of a graph-text pair of data are matched based on keyword information to obtain data corresponding to incomplete matching and complete matching of labeling, a multi-modal sample quality recognition model is trained based on sample data to recognize the initial graph-text pair of data, and the filtered graph-text pair of data is obtained, so as to avoid the problems of low efficiency and low quality of the graph-text pair of data caused by filtering by using an artificial labeling method in the traditional multi-modal large model data filtering process. The recognition model is used for filtering, and the filtering efficiency is improved to a certain extent. In addition, for the ghosting blur condition existing in the matching process, the data quality of the graph-text pair of data is controlled by using uniform keyword information to match the graph-text pair of data when the recognition model is trained, and the quality of the graph-text pair of data is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data filtering method, device, equipment and medium. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, multi-modal large models, as a new type of machine learning technology, have gradually become a hot topic in the field of artificial intelligence. Multi-modal large models can process various media data such as text, images, audio and video, and through learning the association between different modalities, more intelligent information processing can be achieved.

[0003] The data set of the traditional multi-modal large model adopts a manual annotation method during data filtering, and there are hallucination blur conditions when generating image-text pair data, such as inconsistency between the color of the object in the picture and the text description, or statistical error of the number of objects in the picture, etc., which leads to a decrease in the quality of the output image-text pair data.

[0004] Therefore, how to improve the quality of image-text pair data is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a data filtering method, device, equipment and medium to solve the problem of reduced matching degree caused by the use of manual annotation method for the data set of the traditional multi-modal large model, so as to reduce the quality of the image-text pair data output by the multi-modal large model.

[0006] To solve the above technical problems, the present application provides a data filtering method, comprising:

[0007] Obtaining image-text pair data to be filtered;

[0008] Calling a multi-modal sample quality identification model, and taking the image-text pair data to be filtered as input data of the multi-modal sample quality identification model;

[0009] Obtaining output data of the multi-modal sample quality identification model as filtered image-text pair data;

[0010] The multi-modal sample quality identification model is obtained based on annotated sample data for training a multi-modal large model; the annotation process of the annotated sample data specifically includes: obtaining image data and text data corresponding to training image-text pair data and keyword information; performing matching processing on the image data and text data corresponding to the training image-text pair data according to the keyword information to obtain a matching processing result; annotating incomplete matching data and complete matching data of the matching processing result to obtain annotated sample data; the keyword information is feature information corresponding to the text data combined with the image data corresponding to the training image-text pair data.

[0011] In one aspect, the non-complete matching data and the complete matching data of the matching processing result are labeled to obtain labeled sample data, including:

[0012] In the matching processing of the image data and the text data corresponding to the training image-text pair data by the keyword information, if the matching result of the text data is completely matched with the matching result of the image data, the completely matched image data and text data are labeled as positive sample data;

[0013] If the matching result of the text data is not completely matched with the matching result of the image data, the non-completely matched image data and text data are labeled as negative sample data;

[0014] The negative sample data and the positive sample data are used as the labeled sample data.

[0015] In another aspect, the keyword information includes sensitive keyword information and non-sensitive keyword information, and the non-complete matching data and the complete matching data of the matching processing result are labeled to obtain labeled sample data, including:

[0016] In the matching processing of the image data and the text data corresponding to the training image-text pair data by the non-sensitive keyword information, the completely matched image data and text data in each first matching processing result are used as first labeled sample data;

[0017] The non-completely matched image data and text data in each first matching processing result are used as second labeled sample data;

[0018] In the matching processing of the image data and the text data corresponding to the training image-text pair data by the sensitive keyword information, the completely matched image data and text data in each second matching processing result are used as third labeled sample data;

[0019] The non-completely matched image data and text data in each second matching processing result are used as fourth labeled sample data;

[0020] The non-matched image data and text data in each second matching processing result are used as fifth labeled sample data;

[0021] It is judged whether the fifth labeled sample data overlaps with the first labeled sample data and the second labeled sample data;

[0022] If they completely overlap, the fifth labeled sample data is removed; and the first, second, third, and fourth labeled sample data are used as the labeled sample data.

[0023] If they do not completely overlap, then after removing the overlapping data from the first and second annotation sample data respectively in the fifth annotation sample data, the remaining annotation sample data in the fifth annotation sample data is retained; and the remaining annotation sample data in the first, second, third, fourth and fifth annotation sample data are used as the annotation sample data.

[0024] If they do not overlap, the fifth annotation sample data is retained; and the first annotation sample data, the second annotation sample data, the third annotation sample data, the fourth annotation sample data, and the fifth annotation sample data are used as the annotation sample data.

[0025] On the other hand, the process of obtaining the keyword information includes:

[0026] Obtain the initial keyword information corresponding to the training image-text pair data;

[0027] The initial keyword information is used as the keyword information;

[0028] or;

[0029] The initial keyword information is prioritized to determine the sub-keyword information corresponding to each priority level;

[0030] Based on the training graph and text pair data and data filtering conditions, determine the sub-keyword information corresponding to the current target priority;

[0031] The sub-keyword information corresponding to the current target priority is used as the keyword information.

[0032] On the other hand, when the keyword information includes sub-keyword information corresponding to each priority, the process of determining the positive sample data and the negative sample data includes:

[0033] The sub-keyword information corresponding to the first priority is used to match the image data and text data corresponding to the training image-text pair data to obtain the third matching result; so that the image data and text data corresponding to the complete match can be used as positive sample data.

[0034] The sub-keyword information corresponding to the remaining priorities is used to match the image data and text data corresponding to the training image-text pair data to obtain the fourth matching result; so that the image data and text data corresponding to the complete match and incomplete match of the fourth matching result, as well as the image data and text data corresponding to the incomplete match of the third matching result, can be used as negative sample data.

[0035] On the other hand, the process of acquiring the image and text pair data to be filtered includes:

[0036] Obtain initial image data;

[0037] The initial multimodal large model is invoked, and the initial image data is used as the input data of the initial multimodal large model;

[0038] The output data of the initial multimodal large model is obtained and used as the first image-text pair data;

[0039] Determine whether the data volume of the first image-text pair is greater than the preset data volume;

[0040] If so, then randomly select image-text pairs with the same preset data volume from the first image-text pair data as the image-text pair data to be filtered;

[0041] If not, then all of the first image-text pairs are used as the image-text pairs to be filtered.

[0042] On the other hand, the process of acquiring the image-text pair data to be filtered or the training image-text pair data includes:

[0043] Get the current image-text pair data; where the current image-text data is either the image-text pair data to be filtered or the training image-text pair data being labeled.

[0044] The fourth matching result is obtained by matching the current image and text data based on sensitive keyword information;

[0045] In the fourth matching process, it is determined whether the image data and text data of the current image-text pair contain data corresponding to the sensitive keyword information;

[0046] If such data exists, the data corresponding to the sensitive keyword information will be deleted from the current image-text pair data to obtain a new current image-text pair data, which will facilitate subsequent data filtering or training.

[0047] On the other hand, the process of determining the multimodal sample quality identification model includes:

[0048] The multimodal large model is invoked, and the labeled sample data is used as the input data for the multimodal large model.

[0049] The output data of the multimodal large model is obtained and used as the second image-text pair data;

[0050] Determine the accuracy of the positive sample data in the second image-text pair data;

[0051] Determine whether the accuracy of the positive sample data is greater than or equal to the preset accuracy.

[0052] If so, the multimodal large model will be used as the multimodal sample quality identification model;

[0053] If not, return to the step of calling the multimodal large model; and train the multimodal large model using the labeled sample data as input data until the accuracy of the obtained positive sample data is greater than or equal to the preset accuracy.

[0054] On the other hand, the process of determining the multimodal sample quality identification model includes:

[0055] Obtain the embedding layer and visual encoding model of the initial language model;

[0056] The image data of the labeled sample data is input into the visual coding model, and the corresponding image tensor is output.

[0057] The text data of the labeled sample data is input into the embedding layer of the initial language model, and the corresponding text tensor is output.

[0058] The first tensor is obtained by concatenating the text tensor and the image tensor;

[0059] The first tensor is input into the linear transformation layer of the initial language model for tensor decomposition to obtain the second image-text pair data;

[0060] Determine the accuracy of the positive sample data in the second image-text pair data;

[0061] Determine whether the accuracy of the positive sample data is greater than or equal to the preset accuracy.

[0062] If so, the initial language model and the visual encoding model shall be used as the multimodal sample quality recognition model;

[0063] If not, return to the step of obtaining the embedding layer of the initial language model and the visual encoding model for training until the accuracy of the obtained positive sample data is greater than or equal to the preset accuracy.

[0064] On the other hand, the image data of the labeled sample data is input into the visual encoding model, and the corresponding image tensor is output, including:

[0065] The image data corresponding to the training image-text pair data is input into the visual coding model, and the output parameters of the visual coding model are obtained as the second tensor.

[0066] The dimensions of the hidden layers of the visual encoding model are transformed into the dimensions of the hidden layers of the initial language model based on the linear transformation layer and the second tensor, so as to output the corresponding image tensor.

[0067] On the other hand, before inputting the first tensor into the linear transformation layer of the initial language model for tensor decomposition to obtain the second text-image pair data, after concatenating the text tensor and image tensor to obtain the first tensor, the process further includes:

[0068] The first tensor is input into the self-attention layer and fully connected layer of the initial language model to obtain a new first tensor;

[0069] The new first tensor is input into the linear transformation layer of the initial language model to perform tensor decomposition and obtain the second image-text pair data.

[0070] On the other hand, after obtaining the second image-text pair data, the following is also included:

[0071] Obtain the negative sample data and its storage space for the second image-text pair;

[0072] Determine whether the storage space corresponding to the current negative sample data exceeds the preset storage space;

[0073] If the storage space exceeds the limit, it will be cleaned up according to the storage time of the negative sample data to facilitate the storage of new negative sample data.

[0074] To address the aforementioned technical problems, the present invention also provides a data filtering device, comprising:

[0075] The first acquisition module is used to acquire the image-text pairs data to be filtered;

[0076] The calling module is used to call the multimodal sample quality identification model, and the image-text pair data to be filtered is used as the input data of the multimodal sample quality identification model;

[0077] The second acquisition module is used to acquire the output data of the multimodal sample quality recognition model and use it as filtered image-text pair data.

[0078] The multimodal sample quality identification model is trained on a large multimodal model based on labeled sample data. The labeling process of the labeled sample data specifically includes: acquiring image data, text data, and keyword information corresponding to the training image-text pair data; performing matching processing on the image data and text data corresponding to the training image-text pair data according to the keyword information to obtain a matching processing result; labeling the incompletely matched data and completely matched data of the matching processing result to obtain labeled sample data; the keyword information is the feature information corresponding to the text data and image data of the training image-text pair data.

[0079] To address the aforementioned technical problems, the present invention also provides a data filtering device, comprising:

[0080] Memory, used to store computer programs;

[0081] A processor for executing the computer program to implement the steps of the data filtering method as described above.

[0082] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data filtering method described above.

[0083] This invention provides a data filtering method, comprising: acquiring initial image-text pair data; calling a multimodal sample quality identification model, using the initial image-text pair data as input data to the multimodal sample quality identification model; acquiring the output data of the multimodal sample quality identification model, and using it as filtered image-text pair data; wherein, the filtered image-text pair data is less than or equal to the initial image-text pair data; the multimodal sample quality identification model is trained on a large multimodal model based on labeled sample data; the labeling process of the labeled sample data specifically includes: acquiring image data, text data, and keyword information corresponding to the training image-text pair data; performing matching processing on the image data and text data corresponding to the training image-text pair data according to the keyword information to obtain a matching processing result; labeling the incompletely matched data and completely matched data of the matching processing result to obtain labeled sample data; the keyword information is the feature information corresponding to the text data of the training image-text pair data combined with the feature information corresponding to the image data.

[0084] The beneficial effects of this invention lie in its ability to train image and text pairs by combining them with keyword information for matching, resulting in labeled data corresponding to incomplete and complete matches. Based on this labeled sample data, a multimodal sample quality recognition model is trained to recognize the initial image-text pairs, yielding filtered image-text pairs. This avoids the efficiency and quality degradation issues associated with traditional multimodal large-scale model data filtering methods that rely on manual labeling. This invention improves filtering efficiency to a certain extent by using a recognition model for filtering. Furthermore, to address the illusory blurring issues that may occur during matching, this invention employs unified keyword information during the training of the recognition model to control the data quality of the image-text pairs, thereby improving the overall quality of the image-text pairs.

[0085] Secondly, the labeled sample data based on keyword information provided in this embodiment is divided into two types: positive sample data and negative sample data. Positive sample data is labeled "yes," and negative sample data is labeled "no." This simple binary classification clearly defines the matching process, making the labeling process straightforward and improving data processing efficiency. The labeled sample data provided in this embodiment, combined with different keyword information, determines the corresponding labeling process, making the labeling process more refined and improving the accuracy of data processing. The keyword information acquisition scheme provided in this embodiment makes the matching labeling process flexible, increasing the matching diversity of data processing. The labeling process of positive and negative sample data provided in this embodiment, combined with sub-keyword information corresponding to different priorities, increases the flexibility of the labeling process, allowing for labeling based on different sub-keyword information and increasing the diversity of data processing. The process of acquiring the image-text pairs to be filtered provided in this embodiment increases the randomness of data collection, ensuring the authority of subsequent data labeling.

[0086] In addition, the present invention also provides a data filtering device, apparatus, and medium that have the same beneficial effects as the data filtering method described above. Attached Figure Description

[0087] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 A flowchart of a data filtering method provided in an embodiment of the present invention;

[0089] Figure 2 A schematic diagram of a multimodal sample quality identification model provided in an embodiment of the present invention;

[0090] Figure 3 A flowchart of another data filtering method provided in an embodiment of the present invention;

[0091] Figure 4 A structural diagram of a data filtering device provided in an embodiment of the present invention;

[0092] Figure 5 This is a structural diagram of a data filtering device provided in an embodiment of the present invention. Detailed Implementation

[0093] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0094] The core of this invention is to provide a data filtering method, apparatus, device, and medium to solve the problem that the matching degree is reduced due to the manual annotation of datasets in traditional multimodal large models, which leads to the deterioration of the data quality of the image-text pairs output by the multimodal large models.

[0095] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0096] With the rapid development of multimodal large-scale models in computer vision and language, these models can now take images, text, and bounding boxes as input, and output text or bounding boxes after model transformation. Industry-leading multimodal models include Qwen Vision Language (QwenVL), which supports multiple languages ​​such as Chinese and English, and GPT-4V, based on a generative pre-trained deep neural network model (GPT). These models can extract details, text, and coordinates from images, output accurate answers to user questions and images, and possess powerful visual reasoning and instruction-following capabilities. They can accurately complete various downstream tasks such as image description, visual question answering, text extraction, and bounding box description.

[0097] The image understanding and reasoning capabilities of multimodal large-scale models depend on both the model algorithm and the training data, including both the quality and quantity of the data. Currently, leading large-scale models are trained on over 1 billion images, most of which are image-text pairs with image descriptions. While numerous open-source image datasets exist, their image descriptions are often noisy and language-specific. For example, the Laion2B dataset provided by the non-profit organization (Large-scale Artificial Intelligence Open Network, Laion) contains only English image descriptions with poor matching between descriptions and images, making it difficult to translate the dataset's language when training models for other languages, such as Chinese. Furthermore, some datasets are from the internet and contain only images without descriptions. Manually labeling these datasets would be prohibitively time-consuming and labor-intensive. The data filtering method provided in this invention addresses these technical problems.

[0098] Figure 1 A flowchart of a data filtering method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0099] S11: Obtain the image-text pairs to be filtered;

[0100] S12: Call the multimodal sample quality identification model and use the image-text pairs to be filtered as input data for the multimodal sample quality identification model;

[0101] S13: Obtain the output data of the multimodal sample quality identification model and use it as the filtered image-text pair data;

[0102] The multimodal sample quality recognition model is trained on a large multimodal model based on labeled sample data. The labeling process of the labeled sample data specifically includes: acquiring the image data, text data, and keyword information corresponding to the training image-text pair data; performing matching processing on the image data and text data corresponding to the training image-text pair data according to the keyword information to obtain the matching processing result; labeling the incompletely matched data and completely matched data of the matching processing result to obtain labeled sample data; the keyword information is the feature information corresponding to the text data and image data of the training image-text pair data.

[0103] Specifically, the image-text pairs to be filtered can be data downloaded from open-source websites, proprietary datasets, or unprocessed image-text pairs, etc. There are no limitations here; the selection can be based on the actual situation. Image-text pairs consist of images and text. Generally, using images and text together makes it easier for people to understand the essence and characteristics of things. "Image-text" refers to the combined use of images and text. In this embodiment, the image-text pairs are data based on images and text used together in a "pair" format.

[0104] Image-text pairs can be used to enhance semantic alignment between images and text at multiple granularities, or they can consist of visual and textual data. Since visual processing primarily uses image acquisition, image, text, and image-text pairs are thoroughly mixed for training. A portion of the single-modal corpus is used for weak correlation contrast learning in the retrieval system, while a large portion is used for separate text and visual learning. This avoids forgetting issues and forces the network to adapt to both types of data simultaneously, indirectly injecting primal memories from another modality and improving generalization performance.

[0105] The multimodal sample quality identification model in step S12 is a model used for sample quality screening of data from multiple modalities. Its main idea is to establish correlations between different modalities to improve the quality of the screened data. It is trained on a large multimodal model based on labeled sample data. This large multimodal model integrates different media data (such as text, images, audio, and video) and learns the correlations between different modalities to achieve more intelligent information processing. In the large multimodal model, data from different modalities are preprocessed and then input into a deep neural network. After multiple layers of feature extraction and fusion, the final output is the corresponding result.

[0106] In addition, the input data is labeled sample data. The labeling process of this labeled sample data is based on matching the image data and text data of the training image-text pairs with keyword information. That is, based on the feature information corresponding to the keyword information, it is determined whether the image data and text data match. If they match, the matching training image-text pairs are labeled; if they do not match, the unmatched training image-text pairs are labeled. The purpose of the labeled sample data processing in this embodiment is to distinguish the labeled image-text pairs based on the matching degree, so as to facilitate classification in the multimodal sample quality recognition model. The labeling distinction and classification here can be a simple binary classification such as positive and negative samples, or a multi-class classification, in which different levels of sample data are used.

[0107] The matching process in this embodiment can be based on conventional feature processing or similarity matching to determine the similarity between image data and text data. A high similarity indicates a high degree of matching. The matching process is quantified to achieve sample processing of image and text data.

[0108] It is understandable that keyword information is the combination of text data corresponding to training image-text pairs and feature information corresponding to image data. Keyword information can be pre-set, and a large amount of training image-text pairs is filtered based on this keyword information to achieve the division of fully matching data and partially matching data.

[0109] The multimodal large model is trained using labeled sample data. The training process in this embodiment can be based on a limit on the number of iterations or on the accuracy of the model's output parameters using the labeled sample data; no specific limitation is imposed here. The output data of the multimodal sample quality identification model is then used to output the filtered image-text pairs corresponding to the image-text pairs to be filtered.

[0110] This invention provides a data filtering method, comprising: acquiring initial image-text pair data; calling a multimodal sample quality identification model, using the initial image-text pair data as input data to the model; acquiring the output data of the model and using it as filtered image-text pair data; wherein the filtered image-text pair data is less than or equal to the initial image-text pair data; the multimodal sample quality identification model is trained on a large multimodal model based on labeled sample data; the labeling process of the labeled sample data specifically includes: acquiring image data, text data, and keyword information corresponding to the training image-text pair data; performing matching processing on the image data and text data corresponding to the training image-text pair data according to the keyword information to obtain a matching processing result; labeling the incompletely matched data and completely matched data of the matching processing result to obtain labeled sample data; the keyword information is the feature information corresponding to the text data of the training image-text pair data combined with the feature information corresponding to the image data. This invention trains image and text pairs by combining them with keyword information for matching, resulting in labeled data for incomplete and complete matches. A multimodal sample quality recognition model is then trained based on this labeled sample data to identify the initial image-text pairs, yielding filtered image-text pairs. This avoids the inefficiency and quality degradation associated with manual labeling in traditional multimodal large-scale model data filtering. The invention improves filtering efficiency to some extent by using a recognition model for identification and filtering. Furthermore, to address the illusory blurring issues that may occur during matching, this invention employs consistent keyword information during the recognition model training to control the data quality of the image-text pairs, thus improving the overall quality of the image-text pairs.

[0111] In some embodiments, the incompletely matched data and completely matched data of the matching processing results are labeled to obtain labeled sample data, including:

[0112] In the matching process of the image data and text data corresponding to the training image-text pair data by keyword information, if the matching result obtained by the text data completely matches the matching result obtained by the image data, the completely matched image data and text data are labeled as positive sample data.

[0113] If the matching results obtained from the text data do not completely match the matching results obtained from the image data, then the image data and text data that do not completely match are labeled as negative sample data;

[0114] Negative and positive sample data are used as labeled sample data.

[0115] Specifically, the matching results are obtained by matching the image data and text data corresponding to the training image-text pair data according to the keyword information. The keyword information here can be all the keyword information to obtain the final matching result, or it can be based on the main keyword when there are many keywords, or all of them can be matched, but only the complete matching result of the main keyword is considered.

[0116] When the matching results obtained from text data and image data are a perfect match, the corresponding text-image pair can be labeled as positive sample data; when the matching results obtained from text data and image data are not a perfect match, the corresponding text-image pair can be labeled as negative sample data. Both positive and negative sample data are used as labeled sample data.

[0117] The labeled sample data provided in this embodiment, based on keyword information matching and annotation, is divided into two types: positive sample data and negative sample data. Positive sample data is labeled "yes," and negative sample data is labeled "no." This simple binary classification clearly defines the matching process, making annotation processing straightforward and improving data processing efficiency. In some embodiments, keyword information includes sensitive keyword information and non-sensitive keyword information. Labeled sample data is obtained by annotating the incompletely matched and completely matched data from the matching process, including:

[0118] In the first matching result obtained by matching the image data and text data corresponding to the training image-text pair data with non-sensitive keyword information, the image data and text data corresponding to the complete match in each first matching result are used as the first labeled sample data.

[0119] The image data and text data that are not fully matched in each of the first matching processing results are used as the second annotation sample data;

[0120] In the second matching result obtained by matching the image data and text data corresponding to the training image-text pair data with sensitive keyword information, the image data and text data corresponding to the complete match in each second matching result are used as the third annotation sample data.

[0121] The image data and text data that are not fully matched in each of the second matching processing results are used as the fourth annotation sample data;

[0122] The unmatched image data and text data in each of the second matching processing results are used as the fifth annotation sample data;

[0123] Determine whether the fifth labeled sample data overlaps with the first and second labeled sample data;

[0124] If they completely overlap, remove the fifth annotation sample data; and use the first, second, third, and fourth annotation sample data as the annotation sample data.

[0125] If they do not completely overlap, then after removing the overlapping data from the first and second annotation sample data respectively in the fifth annotation sample data, the remaining annotation sample data in the fifth annotation sample data is retained; and the remaining annotation sample data in the first, second, third, fourth and fifth annotation sample data are used as annotation sample data;

[0126] If they do not overlap, the fifth annotation sample data is retained; and the first, second, third, fourth, and fifth annotation sample data are used as annotation sample data.

[0127] Specifically, the keyword information includes sensitive keyword information and non-sensitive keyword information. In other words, sensitive keywords are data that needs to be masked or rejected in subsequent data processing. Therefore, data that exactly matches the sensitive keyword information is labeled. This embodiment distinguishes between sensitive and non-sensitive keywords for labeling, improving the accuracy and precision of the labeling.

[0128] The distinction between sensitive and non-sensitive keyword information can be made based on sensitive features encountered during routine data processing, or on specific features inherent in the current data processing process. Furthermore, there is no limit to the number of sensitive and non-sensitive keyword information items; multiple items may exist, or only one may be present.

[0129] The non-sensitive keyword information is used to match the image data and text data corresponding to the training image-text pair data to obtain the first matching result. The image data and text data that are completely matched are used as the first labeled sample data, and the image data and text data that are not completely matched are used as the second labeled sample data.

[0130] The sensitive keyword information is matched with the image data and text data corresponding to the training image-text pair data to obtain the corresponding second matching result. It should be noted that the second matching result includes three types: complete match, incomplete match, and no match. The difference between this embodiment and the above embodiment is that the no match in this embodiment is for the labeled samples under the sensitive keyword information. The three matching results correspond to the third labeled sample data, the fourth labeled sample data, and the fifth labeled sample data, respectively.

[0131] Furthermore, for a single image-text pair, multiple labeled sample data may exist. To avoid excessive labeling of multiple labeled sample data leading to fine-grained subsequent data filtering, the fifth labeled sample data, which does not contain sensitive keyword information and whose image and text data do not match, needs to be compared with the first and second labeled sample data to eliminate unnecessary labeling information. If overlap exists, a compromise is needed to reduce the amount of data transmitted. If there is complete overlap, the fifth labeled sample data is removed; if there is partial overlap, the overlapping data is deleted, and the remaining labeled samples in the fifth labeled sample data are retained; if there is no overlap, all are retained. Therefore, the final labeled sample data is obtained by combining the first, second, third, and fourth labeled sample data.

[0132] The labeled sample data provided in this embodiment, combined with different keyword information, determines the corresponding labeling process, making the labeling process more refined and improving the accuracy of data processing.

[0133] In some embodiments, the process of obtaining keyword information includes:

[0134] Obtain the initial keyword information corresponding to the training image-text pair data;

[0135] Use the initial keyword information as the keyword information;

[0136] or;

[0137] The initial keyword information is prioritized to determine the sub-keyword information corresponding to each priority level;

[0138] Based on the training graph and text, determine the sub-keyword information corresponding to the current target priority using the data and data filtering conditions;

[0139] Use the subkeyword information corresponding to the current target priority as the keyword information.

[0140] Specifically, keyword information can be all initial keyword information, or it can be sub-keyword information determined based on priority levels. It's important to note that the priority setting takes into account the different keyword information required in different data processing stages. Using priority allows for the separation of various keyword information, enabling hierarchical labeling of training image-text pairs. Determining the target priority requires considering specific data filtering conditions. For example, for image A, if the corresponding main keyword is quantity, then the quantity of objects in the image data would be the first priority. The target priority can be a combination of multiple priority levels forming the keyword information.

[0141] The keyword information acquisition scheme provided in this embodiment makes the matching and annotation process more flexible and improves the matching diversity of data processing.

[0142] In some embodiments, when the keyword information includes sub-keyword information corresponding to each priority level, the process of determining positive sample data and negative sample data includes:

[0143] The sub-keyword information corresponding to the first priority is used to match the image data and text data corresponding to the training image-text pair data to obtain the third matching result; so that the image data and text data corresponding to the complete match can be used as positive sample data.

[0144] The sub-keyword information corresponding to the remaining priorities is used to match the image data and text data corresponding to the training image-text pair data to obtain the fourth matching result; so that the image data and text data corresponding to the complete match and incomplete match of the fourth matching result, as well as the image data and text data corresponding to the incomplete match of the third matching result, can be used as negative sample data.

[0145] Specifically, this embodiment considers the sub-keyword information corresponding to different priorities. An embodiment for determining positive and negative sample data can be based on matching the sub-keyword information of the first priority to obtain a third matching result. The positive sample data is determined based on the image data and text data that completely match in the third matching result. The fourth matching result, obtained by matching the sub-keyword information corresponding to the remaining priorities, is then combined with the image data and text data that do not completely match in the third matching result, and all such data are considered negative sample data.

[0146] The labeling process for positive and negative sample data provided in this embodiment combines sub-keyword information corresponding to different priorities, which improves the flexibility of the labeling process and allows for the division and labeling of different sub-keyword information, thereby increasing the diversity of data processing.

[0147] In some embodiments, both the image-text pairs to be filtered in step S11 and the training image-text pairs being labeled need to undergo preliminary screening to avoid noise in the data obtained after labeling, or to avoid the subsequent data transmission process occupying the same data bit width and consuming resources due to a crude processing method. Therefore, the process of obtaining the image-text pairs to be filtered or the training image-text pairs includes:

[0148] Get the current image-text pair data; where the current image-text data is either the image-text pair data to be filtered or the training image-text pair data being labeled.

[0149] The fourth matching result is obtained by matching the current image and text data based on sensitive keyword information;

[0150] In the fourth matching process, it is determined whether the image data and text data of the current image-text pair contain data corresponding to the sensitive keyword information;

[0151] If such data exists, the data corresponding to the sensitive keyword information will be deleted from the current image-text pair data to obtain a new current image-text pair data, which will facilitate subsequent data filtering or training.

[0152] This embodiment takes into account the noise factor of the data, and requires real-time deletion of useless data in any processing step to ensure the cleanliness of the data and improve its anti-interference ability.

[0153] In some embodiments, the process of acquiring the image-text pair data to be filtered in step S11 includes:

[0154] Obtain initial image data;

[0155] The initial multimodal large model is invoked, and the initial image data is used as the input data for the initial multimodal large model;

[0156] Obtain the output data of the initial multimodal large model and use it as the first image-text pair data;

[0157] Determine whether the data volume of the first image / text pair is greater than the preset data volume;

[0158] If so, then randomly select text-image pairs with the same preset data volume from the first text-image pair data as text-image pairs to be filtered;

[0159] If not, then all first image-text pairs will be treated as image-text pairs to be filtered.

[0160] Specifically, the image-text pairs to be filtered can be based on the current open-source initial multimodal large model to perform image description reasoning on images from open-source datasets or web-scraped images, generating text description data that matches the images. Its code format is as follows:

[0161]

[0162] The output data of the initial multimodal large model is obtained as the first image-text pair dataset. Data is generated for all images in the dataset to form the image-text pair dataset using the following method. This initial multimodal large model identifies the image paths and then generates an answer that matches the prompt words and the image. An example of a model output is shown below:

[0163]

[0164]

[0165] Based on the relationship between the data volume of the first image-text pair and the preset data volume, image-text pairs with the same preset data volume are randomly selected as image-text pairs to be filtered.

[0166] The image and text pair data acquisition process provided in this embodiment improves the randomness of data collection, so as to make the subsequent data labeling more authoritative.

[0167] In some embodiments, the process of determining the multimodal sample quality identification model includes:

[0168] Invoke the multimodal large model; and use the labeled sample data as the input data for the multimodal large model;

[0169] Obtain the output data of the multimodal large model and use it as the second image-text pair data;

[0170] Determine the accuracy of the positive sample data in the second image-text pair;

[0171] Determine whether the accuracy of the positive sample data is greater than or equal to the preset accuracy.

[0172] If so, then the multimodal large model will be used as the multimodal sample quality identification model;

[0173] If not, return to calling the multimodal large model; and train the multimodal large model using labeled sample data as input data until the accuracy of the obtained positive sample data is greater than or equal to the preset accuracy.

[0174] Specifically, the output data of the multimodal large model is obtained as the second image-text pair data. The accuracy of the positive sample data in the second image-text pair data is determined. For example, when the accuracy of the positive sample data recognition is above 85%, the multimodal sample quality recognition model is judged to meet the requirements, and the current training is completed. If the accuracy is not reached, the training continues.

[0175] In some embodiments, the process of acquiring positive and negative sample data involves labeling images and text data in an image-text pair as positive samples if they match perfectly, and as negative samples if there is any mismatch between the images and text data. For example, the first batch of data might sample 5000 positive samples and 5000 negative samples, resulting in the following data format:

[0176]

[0177]

[0178] The above data format provides examples for labeled positive and negative samples. Taking a positive sample as an example, a complete sample consists of an image path, a prompt, and an answer. The prompt part consists of (question + text description). For the answer part, if the data is a positive sample, it is labeled "yes" and if it is a negative sample, it is labeled "no".

[0179] The multimodal sample quality recognition model was trained using the 10,000 positive and negative sample data mentioned above. Table 2 shows the parameters of the multimodal sample quality recognition model. As shown in Table 2, Epoch is the number of training iterations in a given period, such as a parameter of 1; Global Batchsize is the size of the number of input images per batch, such as 64 images per batch; LR is the learning rate, which is the progress of data learning at each step; Optimizer is the optimizer; LR schedule is the learning rate decayer, such as using cosine decay.

[0180] Table 2. Parameters of the Multimodal Sample Quality Identification Model

[0181] Parameter Epoch 1 Global Batchsize 64 LR le-5 Optimizer AdamW LR schedule cosin

[0182] After the model has been trained as described above, for a given image-text pair sample, the input is organized as follows:

[0183]

[0184]

[0185] The model can output "yes" or "no". If the model outputs "yes", it means that the sample is a positive sample. If the model outputs "no", it means that the sample is a negative sample.

[0186] Based on 200 labeled positive samples and 200 labeled negative samples, the multimodal sample quality recognition model trained above is used to evaluate these 400 positive and negative samples. When the recognition accuracy of positive samples is above 85%, the multimodal sample quality recognition model is deemed to meet the requirements. Otherwise, 10,000 positive and negative samples are manually labeled, and the multimodal sample quality recognition model is retrained together with the 10,000 samples trained in the first round. The recognition effect of the model is then evaluated, and this process is repeated until the recognition accuracy of positive samples is above 85%.

[0187] The trained multimodal sample quality identification model is used for filtering, with the following input format:

[0188]

[0189] The model takes an image path and a prompt word as input, where the prompt word is "Does the text description match the image? + image description". After being fed into the multimodal sample quality recognition model, it will output in the following format:

[0190]

[0191]

[0192] The model outputs a "yes" or "no" answer to determine whether a sample is a positive sample. Positive samples are all retained for subsequent model training, while negative samples are discarded or used to generate more image-text pairs.

[0193] In some embodiments, the process of determining the multimodal sample quality identification model includes:

[0194] Obtain the embedding layer and visual encoding model of the initial language model;

[0195] The image data of the labeled sample data is input into the visual encoding model, and the corresponding image tensor is output.

[0196] The labeled sample text data is input into the embedding layer of the initial language model, and the corresponding text tensor is output.

[0197] The first tensor is obtained by concatenating the text tensor and the image tensor;

[0198] The first tensor is input into the linear transformation layer of the initial language model for tensor decomposition to obtain the second text-image pair data;

[0199] Determine the accuracy of the positive sample data in the second image-text pair;

[0200] Determine whether the accuracy of the positive sample data is greater than or equal to the preset accuracy.

[0201] If so, the initial language model and visual encoding model will be used as the multimodal sample quality recognition model;

[0202] If not, return to the steps of obtaining the embedding layer of the initial language model and the visual encoding model for training until the accuracy of the obtained positive sample data is greater than or equal to the preset accuracy.

[0203] Specifically, the training process of the multimodal large model is based on an initial language model and a visual encoding model. The language model (LM) plays a crucial role in natural language processing; its task is to predict the probability of a sentence appearing in a language. To date, the development of language models has progressed through grammatical rule-based language models, statistical language models, and neural network language models. A commonly used evaluation metric for language models is perplexity; the lower the perplexity on test data, the better the language model's modeling performance. Currently, an initial language model is selected. The labeled sample text data is input into the embedding layer of the initial language model to output a text tensor. It should be noted that the initial language model in this embodiment can be any one or more combinations of conventional language models; there are no limitations, and it can be set according to the actual situation.

[0204] A text tensor is a high-dimensional tensor representation that maps input to the model. It's a feature representation process using tensors based on keyword information and text data in the initial language model. The text data is natural language text, and the language model captures the inherent structural features of natural language.

[0205] Similarly, the image tensor is obtained by inputting image data based on labeled sample data into the visual coding model. The visual coding model is a model for processing video understanding tasks. It can handle various video understanding tasks, including classification, localization, retrieval, captioning, and question answering, using a single frozen model. Conventional visual coding models can be convolutional neural networks, autoencoders, recurrent neural networks, long short-term memory networks, etc. The visual coding model in this embodiment can be any one or a combination of conventional visual coding models. There is no limitation here, and it can be set according to the actual situation.

[0206] The obtained text tensor and image tensor are concatenated to obtain the first tensor. It should be noted that the concatenation process combines the vector lengths of the two tensors to facilitate subsequent feature extraction and classification settings. The concatenated first tensor is then input into the embedding layer of the initial language model for tensor decomposition to obtain the second text-image pair data. This tensor decomposition processes the tensor features to obtain positive and negative sample data for the second text-image pair data, and the training result of the model is determined based on the accuracy of the positive sample data.

[0207] In some embodiments, the image data of the labeled sample data is input into the visual coding model, and the corresponding image tensor is output, including:

[0208] Input the image data corresponding to the training image-text pair data into the visual coding model, and obtain the output parameters of the visual coding model as the second tensor;

[0209] The dimensions of the hidden layers of the visual encoding model are transformed into the dimensions of the hidden layers of the initial language model based on the linear transformation layer and the second tensor, so as to output the corresponding image tensor.

[0210] Specifically, the image data of the labeled sample data is input into the visual encoding model to form a tensor representation, that is, the output parameters in matrix form are used as the second tensor. The second tensor is passed through a linear transformation layer to change the hidden layer dimension of the visual encoding model to the hidden layer dimension of the initial language model, so that the hidden layer dimensions of the two models are consistent, thereby improving the quality of the final sample data and facilitating the subsequent assembly to obtain the first tensor.

[0211] In some embodiments, before inputting the first tensor into the linear transformation layer of the initial language model for tensor decomposition to obtain the second text-image pair data, after concatenating the text tensor and image tensor to obtain the first tensor, the method further includes:

[0212] The first tensor is input into the self-attention layer and fully connected layer of the initial language model to obtain a new first tensor;

[0213] The new first tensor is input into the linear transformation layer of the initial language model to perform tensor decomposition and obtain the second text-image pair data.

[0214] Specifically, the first tensor is input into the self-attention layer and fully connected layer of the initial language model to obtain a new first tensor. The self-attention layer ensures that the tensor features obtained from this input are global. The new first tensor is then input into the linear transformation layer of the initial language model to obtain the second image-text pair data. Here, the linear transformation layer is used for binary classification in the multimodal sample quality identification model to distinguish between positive and negative output samples.

[0215] Figure 2 This is a schematic diagram of a multimodal sample quality identification model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the input image data corresponding to the labeled sample data is transformed into a tensor representation by the visual encoding model. After passing through a linear transformation layer, the hidden state-dim dimension of the visual encoding model is transformed into the hidden state-dim dimension of the language model. The text data corresponding to the labeled sample data enters the model and is first transformed into a tensor representation by the language model embedding layer. Then, the image tensor and the text tensor are concatenated to obtain the first tensor, which then enters the self-attention layer and the fully connected layer. In this layer, there can be one or more layers, that is, N layers in this embodiment represent one or more layers, which is not limited here, but N is an integer. Finally, the data passes through the language model embedding layer (linear transformation layer) and outputs the quality result of the model, that is, positive sample or negative sample.

[0216] In this embodiment, a multimodal sample quality recognition model is obtained by combining a language model and a visual coding model and then training it in tensor form, which speeds up data processing and improves the quality of sample data.

[0217] In some embodiments, after obtaining the second image-text pair data, the method further includes:

[0218] Obtain the negative sample data and its storage space for the second image-text pair;

[0219] Determine whether the storage space corresponding to the current negative sample data exceeds the preset storage space;

[0220] If the storage space exceeds the limit, it will be cleaned up according to the storage time of the negative sample data to facilitate the storage of new negative sample data.

[0221] Specifically, negative sample data can be discarded directly, or a portion of negative sample data with a longer storage time can be selectively deleted according to the storage time order or the reserved storage space size, so as to facilitate the storage of new negative sample data. This allows for the simultaneous storage of new negative sample data and the timely updating and discarding of old negative sample data.

[0222] Figure 3A flowchart of another data filtering method provided in an embodiment of the present invention is shown below. Figure 3 As shown, the method includes:

[0223] S21: Input image data;

[0224] S22: Generate image descriptions based on multimodal models;

[0225] S23: Generate image-text pairs based on image descriptions and image data, and randomly sample and select training image-text pairs;

[0226] S24: Label the training image and text data to obtain positive and negative sample data;

[0227] S25: Training a multimodal sample quality recognition model based on positive and negative sample data;

[0228] S26: Perform sample quality inference on the training image-text pairs data to determine whether the sample quality meets the requirements; if yes, proceed to step S27; if no, return to step S23.

[0229] S27: Confirm that the multimodal sample quality identification model training is complete, and input the image-text pair data to be filtered;

[0230] S28: Based on the multimodal sample quality identification model, positive and negative sample identification is performed on the image and text data to be filtered;

[0231] S29: Output positive sample data of images and text.

[0232] The various embodiments corresponding to the data filtering method have been described in detail above. Based on this, the present invention also discloses a data filtering device corresponding to the above method. Figure 4 This is a structural diagram of a data filtering device provided in an embodiment of the present invention. Figure 4 As shown, the data filtering device includes:

[0233] The first acquisition module 11 is used to acquire the image-text pair data to be filtered;

[0234] Module 12 is invoked to call the multimodal sample quality identification model, and the image-text pairs to be filtered are used as input data for the multimodal sample quality identification model.

[0235] The second acquisition module 13 is used to acquire the output data of the multimodal sample quality recognition model and use it as filtered image-text pair data;

[0236] The multimodal sample quality recognition model is trained on a large multimodal model based on labeled sample data. The labeling process of the labeled sample data specifically includes: acquiring the image data, text data, and keyword information corresponding to the training image-text pair data; performing matching processing on the image data and text data corresponding to the training image-text pair data according to the keyword information to obtain the matching processing result; labeling the incompletely matched data and completely matched data of the matching processing result to obtain labeled sample data; the keyword information is the feature information corresponding to the text data and image data of the training image-text pair data.

[0237] In some embodiments, the incompletely matched data and completely matched data of the matching processing results are labeled to obtain labeled sample data, including:

[0238] The first annotation submodule is used to annotate the image data and text data that are completely matched with the image data and text data corresponding to the training image-text pair data by matching the keyword information to obtain the corresponding matching results. If the matching result obtained from the text data completely matches the matching result obtained from the image data, then the completely matched image data and text data are labeled as positive sample data.

[0239] The second annotation submodule is used to annotate the incompletely matched image data and text data as negative sample data if the matching results obtained from the text data and the matching results obtained from the image data are not completely matched.

[0240] The first module is used as a submodule to use negative and positive sample data as labeled sample data.

[0241] In some embodiments, keyword information includes sensitive keyword information and non-sensitive keyword information. Annotated sample data is obtained by labeling the incompletely matched and completely matched data from the matching process, including:

[0242] The second module is used as a submodule to take the image data and text data that are completely matched in each of the first matching results as the first labeled sample data in the first matching result obtained by matching the image data and text data corresponding to the training image-text pair data with non-sensitive keyword information.

[0243] The third module is used as a sub-module to take the image data and text data that are not fully matched in the results of each first matching process as the second annotation sample data.

[0244] The fourth module, as a submodule, is used to take the image data and text data that are completely matched in each of the second matching results as the third labeled sample data in the process of matching the image data and text data corresponding to the training image-text pair data with sensitive keyword information to obtain the corresponding second matching results.

[0245] The fifth module, as a submodule, is used to take the image data and text data that are not fully matched in each of the second matching processing results as the fourth annotation sample data;

[0246] The sixth module, as a submodule, is used to take the unmatched image data and text data as the fifth annotation sample data in each of the second matching processing results;

[0247] The first judgment submodule is used to determine whether the fifth labeled sample data overlaps with the first labeled sample data and the second labeled sample data; if they completely overlap, the first removal submodule is triggered; if they do not completely overlap, the second removal submodule is triggered; if they do not overlap, the retention submodule is triggered.

[0248] The first removal submodule is used to remove the fifth labeled sample data; and to use the first labeled sample data, the second labeled sample data, the third labeled sample data, and the fourth labeled sample data as labeled sample data.

[0249] The second removal submodule is used to remove the overlapping data of the first and second annotation sample data from the fifth annotation sample data respectively, and retain the remaining annotation sample data in the fifth annotation sample data; and use the remaining annotation sample data in the first, second, third, fourth and fifth annotation sample data as annotation sample data;

[0250] The retention submodule is used to retain the fifth annotation sample data; and to use the first annotation sample data, the second annotation sample data, the third annotation sample data, the fourth annotation sample data, and the fifth annotation sample data as annotation sample data.

[0251] In some embodiments, the process of obtaining keyword information includes:

[0252] The first acquisition submodule is used to acquire the initial keyword information corresponding to the training image-text pair data;

[0253] The seventh module, as a submodule, is used to use the initial keyword information as the keyword information;

[0254] or;

[0255] The first sub-module is used to prioritize the initial keyword information and determine the sub-keyword information corresponding to each priority.

[0256] The first determination submodule is used to determine the subkeyword information corresponding to the current target priority based on the training image and text data and data filtering conditions.

[0257] The eighth module, as a submodule, is used to use the subkeyword information corresponding to the current target priority as the keyword information.

[0258] In some embodiments, when the keyword information includes sub-keyword information corresponding to each priority level, the process of determining positive sample data and negative sample data includes:

[0259] The first processing submodule is used to match the image data and text data corresponding to the training image-text pair data with the subkeyword information corresponding to the first priority to obtain the third matching processing result; so that the image data and text data corresponding to the complete match can be used as positive sample data.

[0260] The second processing submodule is used to match the image data and text data corresponding to the training image-text pair data with the subkeyword information corresponding to the remaining priorities to obtain the fourth matching processing result; so that the image data and text data corresponding to the complete match and incomplete match of the fourth matching processing result, and the incomplete match of the third matching processing result, can be used as negative sample data.

[0261] In some embodiments, the process of acquiring the image-text pair data to be filtered includes:

[0262] The second acquisition submodule is used to acquire initial image data;

[0263] The first calling submodule is used to call the initial multimodal large model, using the initial image data as the input data for the initial multimodal large model;

[0264] The third acquisition submodule is used to acquire the output data of the initial multimodal large model and use it as the first image-text pair data;

[0265] The second judgment submodule is used to determine whether the amount of data in the first image and text pair is greater than the preset amount of data; if so, the ninth submodule is triggered; if not, the tenth submodule is triggered.

[0266] The ninth module, as a submodule, is used to randomly select image-text pairs with the same preset data volume from the first image-text pair data as the image-text pair data to be filtered.

[0267] The tenth module, as a submodule, is used to treat all first image-text pairs as image-text pairs to be filtered.

[0268] In some embodiments, the process of acquiring the image-text pair data to be filtered or the training image-text pair data includes:

[0269] The fourth acquisition module is used to acquire the current image-text pair data; wherein, the current image-text data is the image-text pair data to be filtered or the training image-text pair data being labeled.

[0270] The first submodule is used to match the current image and text data based on sensitive keyword information to obtain the fourth matching result;

[0271] The third judgment submodule is used to determine whether the image data and text data of the current image-text pair data contain data corresponding to sensitive keyword information in the fourth matching process result; if they do, the deletion submodule is triggered.

[0272] The deletion submodule is used to delete data that corresponds to sensitive keyword information from the current image-text pair data to obtain new current image-text pair data, so as to facilitate subsequent data filtering or training.

[0273] In some embodiments, the process of determining the multimodal sample quality identification model includes:

[0274] The second calling submodule is used to call the multimodal large model and uses the labeled sample data as the input data for the multimodal large model.

[0275] The fifth acquisition submodule is used to acquire the output data of the multimodal large model and use it as the second image-text pair data;

[0276] The second determination submodule is used to determine the accuracy of the positive sample data in the second image-text pair data;

[0277] The fourth judgment submodule is used to determine whether the accuracy of the positive sample data is greater than or equal to the preset accuracy. If yes, the eleventh submodule is triggered; if no, the second calling submodule is returned until the accuracy of the obtained positive sample data is greater than or equal to the preset accuracy.

[0278] The eleventh module is used as a submodule to use the large multimodal model as a multimodal sample quality identification model.

[0279] In some embodiments, the process of determining the multimodal sample quality identification model includes:

[0280] The sixth submodule is used to obtain the embedding layer and visual encoding model of the initial language model;

[0281] The first input submodule is used to input the image data of the labeled sample data into the visual coding model and output the corresponding image tensor;

[0282] The second input submodule is used to input the text data of the labeled sample data into the embedding layer of the initial language model and output the corresponding text tensor.

[0283] The first splicing submodule is used to splice text tensors and image tensors to obtain the first tensor;

[0284] The tensor decomposition submodule is used to input the first tensor into the linear transformation layer of the initial language model for tensor decomposition to obtain the second text-image pair data.

[0285] The third determination submodule is used to determine the accuracy of the positive sample data in the second image-text pair data;

[0286] The fifth judgment submodule is used to determine whether the accuracy of the positive sample data is greater than or equal to the preset accuracy. If yes, the twelfth submodule is triggered; otherwise, the process returns to the sixth acquisition submodule to perform training until the accuracy of the obtained positive sample data is greater than or equal to the preset accuracy.

[0287] The twelfth module is used as a submodule to use the initial language model and visual coding model as a multimodal sample quality recognition model.

[0288] In some embodiments, the first input submodule includes:

[0289] The third input submodule is used to input the image data corresponding to the training image-text pair data into the visual coding model and obtain the output parameters of the visual coding model as the second tensor.

[0290] The first transformation submodule is used to transform the dimensions of the hidden layers of the visual encoding model into the dimensions of the hidden layers of the initial language model based on the linear transformation layer and the second tensor, so as to output the corresponding image tensor.

[0291] In some embodiments, before the tensor decomposition submodule and after the first assembly submodule, the method further includes:

[0292] The fourth input submodule is used to input the first tensor into the self-attention layer and fully connected layer of the initial language model to obtain a new first tensor;

[0293] The fifth input submodule is used to input the new first tensor into the linear transformation layer of the initial language model to perform tensor decomposition and obtain the second text-image pair data.

[0294] In some embodiments, after obtaining the second image-text pair data, the method further includes:

[0295] The seventh acquisition submodule is used to acquire the negative sample data of the second image-text pair and its corresponding storage space;

[0296] The sixth judgment submodule is used to determine whether the storage space corresponding to the current negative sample data exceeds the preset storage space; if it does, the cleanup submodule is triggered.

[0297] The cleaning submodule is used to clean up the storage space based on the storage time of the negative sample data so that new negative sample data can be stored.

[0298] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0299] For a description of the data filtering device provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above data filtering method.

[0300] Figure 5 A structural diagram of a data filtering device provided in an embodiment of the present invention is shown below. Figure 5 As shown, the device includes:

[0301] Memory 21 is used to store computer programs;

[0302] Processor 22 is used to implement the steps of a data filtering method when executing a computer program.

[0303] The data filtering device provided in this embodiment may include, but is not limited to, tablet computers, laptop computers, or desktop computers.

[0304] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0305] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the data filtering method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the data filtering method, etc.

[0306] In some embodiments, the data filtering device may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0307] Those skilled in the field can understand, Figure 5 The structure shown does not constitute a limitation on the data filtering device and may include more or fewer components than illustrated.

[0308] The processor 22 implements the data filtering method provided in any of the above embodiments by calling instructions stored in the memory 21.

[0309] For an introduction to the data filtering device provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above data filtering method.

[0310] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by processor 22, implements the steps of the data filtering method described above.

[0311] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0312] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be described in detail here, but it has the same beneficial effects as the above data filtering method.

[0313] The data filtering method, apparatus, device, and medium provided by the present invention have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the present invention.

[0314] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

Claims

1. A data filtering method, characterized by, The method comprises the following steps: obtaining image-text pair data to be filtered; calling a multi-modal sample quality identification model, and taking the image-text pair data to be filtered as input data of the multi-modal sample quality identification model; obtaining output data of the multi-modal sample quality identification model as filtered image-text pair data; wherein the multi-modal sample quality identification model is obtained based on multi-modal large model training of labeled sample data; the labeling process of the labeled sample data comprises the following steps: obtaining image data and text data corresponding to training image-text pair data and keyword information; performing matching processing on the image data and the text data corresponding to the training image-text pair data according to the keyword information to obtain a matching processing result; labeling the incomplete matching data and the complete matching data of the matching processing result to obtain labeled sample data; the keyword information is feature information of the text data combined with the image data corresponding to the training image-text pair data; correspondingly, labeling the incomplete matching data and the complete matching data of the matching processing result to obtain labeled sample data comprises the following steps: in the matching processing of the image data and the text data corresponding to the training image-text pair data by the keyword information, if the matching result of the text data is completely matched with the matching result of the image data, the completely matched image data and text data are labeled as positive sample data; if the matching result of the text data is not completely matched with the matching result of the image data, the incompletely matched image data and text data are labeled as negative sample data; the negative sample data and the positive sample data are taken as the labeled sample data; when the keyword information comprises sub-keyword information corresponding to each priority, the determination process of the positive sample data and the negative sample data comprises the following steps: the sub-keyword information corresponding to the first priority is used to perform matching processing on the image data and the text data corresponding to the training image-text pair data to obtain a third matching processing result; so as to take the completely matched image data and text data as positive sample data; the sub-keyword information corresponding to the remaining priorities is used to perform matching processing on the image data and the text data corresponding to the training image-text pair data to obtain a fourth matching processing result; so as to take the completely matched and incompletely matched image data and text data corresponding to the fourth matching processing result and the incompletely matched image data and text data corresponding to the third matching processing result as negative sample data.

2. The data filtering method of claim 1, wherein, the keyword information comprises sensitive keyword information and non-sensitive keyword information, and labeling the incomplete matching data and the complete matching data of the matching processing result to obtain labeled sample data comprises the following steps: in the matching processing of the image data and the text data corresponding to the training image-text pair data by the non-sensitive keyword information, the completely matched image data and text data in each first matching processing result are taken as first labeled sample data; the incompletely matched image data and text data in each first matching processing result are taken as second labeled sample data; In the matching processing of the image data and the text data corresponding to the training image-text pair data by the sensitive keyword information, the completely matched image data and text data in each second matching processing result are taken as third labeled sample data; The incompletely matched image data and text data in each second matching processing result are taken as fourth labeled sample data; The unmatched image data and text data in each second matching processing result are taken as fifth labeled sample data; It is judged whether the fifth labeled sample data overlaps with the first labeled sample data and the second labeled sample data; If the complete overlap, remove the fifth labeled sample data; and the first labeled sample data, the second labeled sample data, the third labeled sample data, the fourth labeled sample data are taken as the labeled sample data; If the incomplete overlap, after removing the data overlapping with the first labeled sample data and the second labeled sample data in the fifth labeled sample data, the remaining labeled sample data in the fifth labeled sample data is reserved; and the first labeled sample data, the second labeled sample data, the third labeled sample data, the fourth labeled sample data and the remaining labeled sample data in the fifth labeled sample data are taken as the labeled sample data; If not overlap, the fifth labeled sample data is reserved; and the first labeled sample data, the second labeled sample data, the third labeled sample data, the fourth labeled sample data and the fifth labeled sample data are taken as the labeled sample data.

3. The data filtering method according to claim 1 or 2, characterized in that, The process of obtaining the keyword information includes: Obtaining initial keyword information corresponding to the training image-text pair data; The initial keyword information is taken as the keyword information; Or; The initial keyword information is prioritized to determine the sub-keyword information corresponding to each priority; According to the training image-text pair data and the data filtering condition, the sub-keyword information corresponding to the current target priority is determined; The sub-keyword information corresponding to the current target priority is taken as the keyword information.

4. The data filtering method of claim 1, wherein, The process of obtaining the to-be-filtered image-text pair data includes: Obtaining initial image data; Calling an initial multi-modal large model, taking the initial image data as the input data of the initial multi-modal large model; Obtaining the output data of the initial multi-modal large model, and taking it as the first image-text pair data; It is judged whether the data amount of the first image-text pair data is greater than a preset data amount; If yes, randomly select the same image-text pair data of the preset data amount in the first image-text pair data as the to-be-filtered image-text pair data; If not, all the first image-text pair data is taken as the to-be-filtered image-text pair data.

5. The data filtering method of claim 1, wherein, The process of obtaining the to-be-filtered image-text pair data or the training image-text pair data includes: Obtaining current image-text pair data; wherein the current image data is to-be-filtered image-text pair data or training image-text pair data being labeled; According to the sensitive keyword information, the matching processing of the current image-text pair data is obtained to obtain the fourth matching processing result; In the fourth matching processing result, it is judged whether the image data and the text data of the current image-text pair data exist data corresponding to the sensitive keyword information; If the sensitive keyword information exists, data corresponding to the sensitive keyword information is deleted from current image-text pair data to obtain new current image-text pair data, so as to facilitate subsequent data filtering or training.

6. The data filtering method of claim 1, wherein, The determination process of the multi-modal sample quality identification model comprises: calling a multi-modal large model; and taking the labeled sample data as input data of the multi-modal large model; obtaining output data of the multi-modal large model and taking the output data as second image-text pair data; determining a correctness rate of positive sample data in the second image-text pair data; judging whether the correctness rate of the positive sample data is greater than or equal to a preset correctness rate; if yes, taking the multi-modal large model as the multi-modal sample quality identification model; if no, returning to the step of calling the multi-modal large model and taking the labeled sample data as input data of the multi-modal large model for training until the correctness rate of the obtained positive sample data is greater than or equal to the preset correctness rate.

7. The data filtering method of claim 1, wherein, The determination process of the multi-modal sample quality identification model comprises: obtaining an embedding layer of an initial language model and a visual encoding model; inputting image data of the labeled sample data into the visual encoding model and outputting corresponding image tensors; inputting text data of the labeled sample data into the embedding layer of the initial language model and outputting corresponding text tensors; splicing the text tensors and the image tensors to obtain first tensors; inputting the first tensors into a linear transformation layer of the initial language model for tensor decomposition to obtain second image-text pair data; determining a correctness rate of positive sample data in the second image-text pair data; judging whether the correctness rate of the positive sample data is greater than or equal to a preset correctness rate; if yes, taking the initial language model and the visual encoding model as the multi-modal sample quality identification model; if no, returning to the step of obtaining the embedding layer of the initial language model and the visual encoding model for training until the correctness rate of the obtained positive sample data is greater than or equal to the preset correctness rate.

8. The data filtering method of claim 7, wherein, Inputting image data of the labeled sample data into the visual encoding model and outputting corresponding image tensors comprises: inputting image data corresponding to the training image-text pair data into the visual encoding model, obtaining output parameters of the visual encoding model as second tensors; changing a dimension of a hidden layer of the visual encoding model to a dimension of a hidden layer of the initial language model according to the linear transformation layer and the second tensors, to output corresponding image tensors.

9. The data filtering method of claim 8, wherein, Before inputting the first tensors into the linear transformation layer of the initial language model for tensor decomposition to obtain the second image-text pair data, after splicing the text tensors and the image tensors to obtain the first tensors, the method further comprises: inputting the first tensors into a self-attention layer and a fully connected layer of the initial language model to obtain new first tensors; inputting the new first tensors into the linear transformation layer of the initial language model for tensor decomposition to obtain the second image-text pair data.

10. The data filtering method of claim 9, wherein, After obtaining the second image-text pair data, the method further comprises: obtaining negative sample data of the second image-text pair data and a storage space to which the negative sample data belongs; judging whether a storage space corresponding to current negative sample data exceeds a preset storage space; If it exceeds, the storage space to which the storage time of the negative sample data belongs is cleaned according to the storage time of the storage negative sample data, so as to store new negative sample data.

11. A data filtering device, characterized by The method comprises the steps of: The first acquisition module is configured to acquire image-text pair data to be filtered. The calling module is configured to call a multi-modal sample quality identification model and input the image-text pair data to be filtered into the multi-modal sample quality identification model. The second acquisition module is configured to acquire output data of the multi-modal sample quality identification model and output the output data as filtered image-text pair data. The multi-modal sample quality identification model is trained based on annotated sample data. The annotation process of the annotated sample data comprises the steps of: acquiring image data and text data corresponding to training image-text pair data and keyword information; performing matching processing on the image data and the text data corresponding to the training image-text pair data according to the keyword information to obtain a matching processing result; and annotating the non-complete matching data and the complete matching data of the matching processing result to obtain annotated sample data. The keyword information is feature information of the image data combined with the text data corresponding to the training image-text pair data. Correspondingly, the annotation of the non-complete matching data and the complete matching data of the matching processing result to obtain annotated sample data comprises the steps of: If the matching result of the text data and the matching result of the image data are completely matched, the completely matched image data and text data are annotated as positive sample data. If the matching result of the text data and the matching result of the image data are not completely matched, the non-completely matched image data and text data are annotated as negative sample data. The negative sample data and the positive sample data are used as the annotated sample data. When the keyword information comprises sub-keyword information corresponding to each priority, the determination process of the positive sample data and the negative sample data comprises the steps of:

12. A data filtering device, characterized by The sub-keyword information corresponding to the first priority is used to perform matching processing on the image data and the text data corresponding to the training image-text pair data to obtain a third matching processing result, so that the completely matched image data and text data are used as positive sample data. The sub-keyword information corresponding to the remaining priorities is used to perform matching processing on the image data and the text data corresponding to the training image-text pair data to obtain a fourth matching processing result, so that the completely matched and non-completely matched image data and text data corresponding to the fourth matching processing result and the non-completely matched image data and text data corresponding to the third matching processing result are used as negative sample data. The method comprises the steps of:

13. A computer-readable storage medium, characterized in that, The memory is configured to store a computer program. The processor is configured to execute the computer program to implement the steps of the data filtering method according to any one of claims 1 to 10. The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the data filtering method according to any one of claims 1 to 10.

Images