Method and device for constructing large-scale Chinese text data, system and storage medium

Abstract

This application provides a method, apparatus, system, and storage medium for constructing large-scale Chinese image-text data, relating to the field of image-text dataset construction technology. By performing language filtering and basic text length filtering on original image-text pairs, calculating information entropy and filtering grammatical integrity on the text in candidate image-text pairs, performing fidelity filtering, pixel entropy calculation, and perceptual clustering deduplication on the images in refined image-text pairs, and calculating cross-modal similarity distance between images and text in diverse image-text pairs and retaining image-text pairs within a preset interval, a target dataset is obtained. These steps work together to achieve the technical effect of efficiently filtering high-quality, large-scale Chinese image-text data from massive amounts of internet data, improving the training effect of visual language pre-trained models.

Description

Technical Field

[0001] This application relates to the field of image and text dataset construction technology, specifically to a method, apparatus, system, and storage medium for constructing large-scale Chinese image and text data. Background Technology

[0002] Currently, in the field of visual language pre-training, training dual-tower models using contrastive learning frameworks has become the mainstream technique. The performance of these models highly depends on large-scale, high-quality image-text pairing datasets. As the parameter size of deep learning models continues to grow, their demand for training data is also expanding. This is especially true for Chinese visual language pre-training models, where high-quality, large-scale Chinese image-text datasets have become one of the key factors restricting model performance improvement. Although the internet contains a vast amount of image-text pairing resources, the raw data usually contains a large amount of noise, low-quality samples, and unsafe content, requiring systematic cleaning and filtering to transform it into effective training data.

[0003] Several schemes for constructing Chinese image-text datasets already exist in related technologies. Some schemes collect raw image-text pairs from the Internet and perform language tag filtering, text length filtering, image-text similarity filtering based on pre-trained visual language models, and image resolution and aspect ratio checks. Other schemes utilize user click-through rates and other behavioral data as auxiliary signals, combining diverse sources of text information to construct datasets.

[0004] In the process of implementing the embodiments of this application, it was found that the related technology has at least the following problems: The filtering process is relatively basic, relying only on a single type of signal or simple threshold judgment. This results in the dataset still containing a large number of semantically irrelevant or low-information samples, making it difficult to meet the overall quality requirements for model training. The limitations of the filtering dimensions cause the semantic distribution of the dataset to be biased towards high-frequency concepts, with insufficient coverage of long-tail semantic content, affecting the model's generalization ability. Some datasets were built a long time ago, and their timeliness cannot cover recently emerging concepts. On the other hand, the solution of introducing user behavior data cannot guarantee the relevance of image-text pairs due to non-semantic noise such as accidental clicks. Therefore, the relevant technologies still cannot efficiently filter high-quality large-scale Chinese image-text data from massive amounts of Internet data, thus limiting the training effect of visual language pre-trained models. Summary of the Invention

[0005] This application provides a method, apparatus, system, and storage medium for constructing large-scale Chinese graphic data.

[0006] The first aspect of this application provides a method for constructing large-scale Chinese image-text data, comprising: acquiring original image-text pairs; performing language filtering and basic text length filtering on the original image-text pairs to obtain candidate image-text pairs; calculating information entropy on the text in the candidate image-text pairs and removing text with information entropy lower than a first preset threshold; performing grammatical integrity filtering on the removed text to obtain refined image-text pairs; performing fidelity filtering on the images in the refined image-text pairs; calculating pixel entropy on the images after fidelity filtering and removing images with pixel entropy lower than a second preset threshold; performing perceptual clustering to remove duplicates on the remaining images to obtain diversified image-text pairs; calculating cross-modal similarity distance between the images and text in the diversified image-text pairs; retaining only image-text pairs whose cross-modal similarity distance is within a preset interval; and obtaining and outputting the target dataset.

[0007] A second aspect of this application provides an apparatus for constructing large-scale Chinese graphic data, including a processor and a memory storing program instructions. The processor is configured to execute the method for constructing large-scale Chinese graphic data as described in the first aspect of this application when running the program instructions.

[0008] A third aspect of the present application provides a system for constructing large-scale Chinese image and text data, comprising: a system body for constructing large-scale Chinese image and text data; and a device for constructing large-scale Chinese image and text data as described in the second aspect of the present application, which is installed on the system body for constructing large-scale Chinese image and text data.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing program instructions that, when executed, cause a computer to perform a method for constructing large-scale Chinese graphic data as described in the first aspect of this application.

[0010] The method, apparatus, system, and storage medium for constructing large-scale Chinese image and text data provided in the embodiments of this application have the following beneficial effects: This application's embodiments perform language filtering and basic text length filtering on original image-text pairs, calculate information entropy and filter grammatical integrity for text in candidate image-text pairs, perform fidelity filtering, pixel entropy calculation, and perceptual clustering deduplication on images in refined image-text pairs, and calculate cross-modal similarity distance between images and text in diverse image-text pairs, retaining image-text pairs within a preset range. This can filter out semantically irrelevant and low-information text samples, improve the visual quality of the image set, reduce semantic redundancy, and obtain samples with reliable image-text relevance. The above steps work together to achieve the technical effect of efficiently filtering high-quality, large-scale Chinese image-text data from massive amounts of Internet data, improving the training effect of visual language pre-trained models. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a method for constructing large-scale Chinese image and text data provided in an embodiment of this application; Figure 2 This is an application diagram illustrating a method for constructing large-scale Chinese image and text data provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the application effect of a method for constructing large-scale Chinese image and text data provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the application effect of another method for constructing large-scale Chinese image and text data provided in the embodiments of this application; Figure 5 This is a schematic diagram of a device for constructing large-scale Chinese graphic data provided in an embodiment of this application.

[0012] Figure label: 800: Device for constructing large-scale Chinese graphic data; 801: Processor; 802: Memory; 803: Communication interface; 804: Bus. Detailed Implementation

[0013] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0014] This application proposes a system for constructing large-scale Chinese image and text data, including a system body for constructing large-scale Chinese image and text data and a device 800 for constructing large-scale Chinese image and text data described below. The system body for constructing large-scale Chinese image and text data includes a data source selection module, a text refinement module, a visual diversification module, a cross-modal cross-batch filtering module, and a control and coordination module. The data source selection module is used to obtain original image and text pairs from massive amounts of Internet data, and to perform language filtering and basic text length filtering on the original image and text pairs, outputting candidate image and text pairs. The input end of the text refinement module is connected to the output end of the data source selection module, and is used to receive candidate image and text pairs, calculate the information entropy of the text in the candidate image and text pairs, remove text whose information entropy is lower than a first preset threshold, and then perform grammatical integrity filtering on the removed text, outputting refined image and text pairs. The input of the visual diversification module is connected to the output of the text refinement module. It receives refined image-text pairs, performs fidelity filtering on the images within these pairs, calculates pixel entropy for the filtered images, removes images with pixel entropy below a second preset threshold, and then performs perceptual clustering to remove duplicates from the remaining images, outputting diversified image-text pairs. The input of the cross-modal cross-batch filtering module is connected to the output of the visual diversification module. It receives diversified image-text pairs, calculates cross-modal similarity distance between the images and text within these pairs, and retains only image-text pairs whose cross-modal similarity distance falls within a preset range, outputting the target dataset. The control and coordination module is connected to the data source selection module, text refinement module, visual diversification module, and cross-modal cross-batch filtering module, respectively, and manages the execution order and data flow of these modules. The large-scale Chinese image-text data construction device 800 is installed within the large-scale Chinese image-text data construction system body. The installation relationships described herein are not limited to placement within the large-scale Chinese image and text data construction system, but also include installation connections with other components of the large-scale Chinese image and text data construction system, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the large-scale Chinese image and text data construction device 800 can be adapted to any feasible large-scale Chinese image and text data construction system, thereby realizing other feasible embodiments. The large-scale Chinese image and text data construction device 800 also includes a processor 801, electrically connected to the aforementioned modules, for controlling the aforementioned modules to perform corresponding actions.

[0015] Figure 1 This is a schematic diagram of a method for constructing large-scale Chinese image and text data provided in an embodiment of this application. Any of the following methods can be executed in a system for constructing large-scale Chinese image and text data, or in a server or terminal device that is communicatively connected to the system for constructing large-scale Chinese image and text data.

[0016] Based on the above-mentioned structure of the system for constructing large-scale Chinese image and text data, such as Figure 1As shown in the figure, this application provides a method for constructing large-scale Chinese image and text data, including: S01, Obtain the original image-text pairs, and perform language filtering and basic text length filtering on the original image-text pairs to obtain candidate image-text pairs.

[0017] S02, calculate the information entropy of the text in the candidate image-text pairs and remove the text whose information entropy is lower than the first preset threshold. Then, filter the removed text for grammatical integrity to obtain refined image-text pairs.

[0018] S03, perform fidelity filtering on the images in the refined image-text pairs, calculate the pixel entropy of the filtered images and remove images with pixel entropy lower than the second preset threshold, and then perform perceptual clustering to remove duplicates on the remaining images to obtain diverse image-text pairs.

[0019] S04: Calculate the cross-modal similarity distance between the image and text in the diverse image-text pairs, retain only the image-text pairs whose cross-modal similarity distance is within the preset range, and obtain and output the target dataset.

[0020] In this embodiment, the original image-text pairs originate from massive amounts of data on the Internet, obtainable from public network datasets. These datasets are collections of images and their associated text extracted from general web crawler data. The images in the original image-text pairs are image files in various formats, and the associated text consists of alternative text, titles, surrounding descriptive text, or user-uploaded tag information from the webpage containing the image. The number of original image-text pairs reaches hundreds of millions or even higher. Language filtering of the original image-text pairs refers to using a language recognition tool to determine the language category of the text in the original image-text pairs, filtering out image-text pairs whose text language is Chinese. The language recognition tool employs a lightweight language recognition model based on fast text classification, which can quickly output language tags for the input text. Basic text length filtering of the original image-text pairs refers to retaining image-text pairs whose text length is within a preset character range. The lower limit of this preset character range is 5 Chinese words, and the upper limit is 60 Chinese words. By setting a lower limit, samples with excessively short text content and a lack of effective semantic information can be filtered out; by setting an upper limit, samples with excessively long text content and the potential to contain a large amount of irrelevant and redundant information can be filtered out. After language filtering and basic text length filtering, the amount of raw data is effectively reduced, and the samples in the candidate image-text pairs meet the basic requirements in terms of language attributes and text length, providing high-quality input for subsequent refined filtering. The data source selection module is used to perform the function corresponding to S01 mentioned above.

[0021] In this embodiment, information entropy is a quantitative indicator for measuring the amount of information in text. For text in a candidate image-text pair, it is considered a sequence of multiple characters or words, and the information entropy is calculated by summing the negative logarithms of the probabilities of each character or word. A higher information entropy indicates a richer variety and more even distribution of different characters or words, and thus a greater amount of information. Conversely, a lower information entropy indicates a large number of repeated characters or words, or a short text sequence resulting in a limited number of distinguishable character types and a smaller amount of information. A first preset threshold, experimentally calibrated, is used to distinguish between high-information-content text and low-information-content text. When the information entropy of a text is lower than the first preset threshold, the image-text pair containing that text is removed. Information entropy filtering effectively filters out meaningless repeated characters, garbled text, overly short greetings, and other low-information-content text.

[0022] In this embodiment, the discarded text undergoes grammatical integrity filtering to further filter out texts that, while meeting the information entropy requirements, are grammatically incomplete. Grammatical integrity filtering is achieved by segmenting the text and making judgments based on the segmentation results. After segmenting the text, it is checked whether the segmentation results contain at least one noun. If the text lacks nouns, its grammatical integrity is insufficient, and it cannot express a clear object or entity, thus it is discarded. The determination of nouns is based on the part-of-speech tagging tool used to tag the part of speech of each segmented word. Furthermore, the number of unknown words in the segmentation results is checked. Unknown words refer to words not included in the segmenter's vocabulary, manifested as special symbols, rare characters, or non-standard expressions. If the number of unknown words exceeds a preset threshold, it indicates that the text contains a large number of characters that cannot be parsed normally, resulting in low semantic reliability, and it is discarded. The preset threshold for the number of unknown words is 5. Through grammatical integrity filtering, the retained text is grammatically more complete, providing more reliable text-side input for subsequent cross-modal alignment. The text refinement module is used to perform the function corresponding to S02 above.

[0023] In this embodiment, fidelity filtering is performed on the images in the refined image-text pair to eliminate image samples with poor visual quality. Fidelity filtering includes checking aspects such as image aspect ratio, minimum side length, and pixel intensity variation. The image aspect ratio should be maintained within a preset range to avoid excessive stretching and distortion in the horizontal or vertical directions; the minimum side length should be greater than a preset pixel threshold to ensure sufficient spatial resolution for subsequent feature extraction. Pixel intensity variation refers to the dispersion of grayscale values ​​of each pixel in the image. For a completely uniform or nearly uniform image, the pixel intensity variation is minimal, indicating that the image does not contain any meaningful visual information and is therefore invalid and should be eliminated.

[0024] In this embodiment, pixel entropy is a quantitative indicator for measuring the amount of information in an image. The image is considered a two-dimensional matrix composed of multiple pixels, each with a grayscale value. Pixel entropy is calculated by summing the negative logarithms of the probabilities of each pixel's grayscale value. A higher pixel entropy indicates a richer and more uniform distribution of grayscale levels, and thus a greater amount of visual information. Conversely, a lower pixel entropy indicates a concentrated grayscale distribution, monotonous image content, and low visual information. A second preset threshold, experimentally calibrated, is used to distinguish between images with high and low visual information. When the pixel entropy of an image is lower than the second preset threshold, the image-text pair containing that image is removed. Pixel entropy filtering effectively filters out invalid images with solid-color backgrounds, overexposed or underexposed images, and images with extremely simple content.

[0025] In this embodiment, perceptual clustering is performed on the remaining images to reduce visual redundancy and semantic repetition at the perceptual level. A pre-trained visual model is used to extract the embedding vectors of each image in the remaining images. These embedding vectors are semantic representations of the images in high-dimensional space, reflecting their visual content and semantic information. The cosine distance between the embedding vectors of each image is calculated; a smaller cosine distance indicates greater perceptual similarity between the two images. Images with a cosine distance less than a preset clustering threshold are grouped into the same image group. This process uses a disjoint-set data structure (DFS) algorithm to achieve efficient non-overlapping clustering. For each resulting image group, the centroid of the group is calculated. The centroid is the average vector of all image embedding vectors within the group. The image closest to the centroid is retained as the unique representative of the group, while other images within the group are discarded. Perceptual clustering deduplication effectively reduces image-level redundancy while maintaining the visual diversity level of the dataset, resulting in a more balanced semantic distribution of the dataset. The visual diversity module performs the function corresponding to S03 described above.

[0026] In this embodiment, cross-modal similarity distance is a quantitative indicator for measuring the degree of semantic alignment between images and text. A pre-trained cross-modal expert model is used to encode the images and text in diverse image-text pairs, obtaining image embedding vectors and text embedding vectors. The L2 distance between these two embedding vectors is then calculated. A smaller L2 distance indicates that the image and text are closer in semantic space, and their semantic relevance is higher; a larger L2 distance indicates that the image and text are farther apart in semantic space, and their semantic relevance is lower. The preset interval has a lower limit and an upper limit, retaining only image-text pairs whose cross-modal similarity distance falls within the preset interval. When the lower limit is set, samples with excessively small L2 distances are eliminated, as these samples may exhibit abnormally close false matches due to a lack of actual correspondence between the image and text in terms of content. When the upper limit is set, samples with excessively large L2 distances are eliminated, as these samples have low image-text relevance. The lower limit of the preset interval is 1.06, and the upper limit is 1.24, retaining only image-text pairs whose L2 distance falls within this interval. By employing a distance interval filtering strategy, low-relevance samples can be filtered out, along with noise samples with abnormally high similarity due to images containing a large amount of text, thus obtaining a more reliable target dataset with image-text relevance. The final output target dataset is a large-scale Chinese image-text dataset, which can be directly used for training visual language pre-training models. The cross-modal cross-batch filtering module is used to execute the function corresponding to S04 above. The control and coordination module controls the data source selection module, text refinement module, visual diversification module, and cross-modal cross-batch filtering module to execute sequentially in the order of S01 to S04, and coordinates the data flow between each module.

[0027] The method for constructing large-scale Chinese image and text data provided in this application, by sequentially completing data source selection, text refinement, visual diversification, and cross-modal cross-batch filtering, can filter out high-quality, semantically balanced, and image-text aligned large-scale Chinese image and text datasets from massive original image and text pairs.

[0028] Optionally, language filtering and basic text length filtering are performed on the original image-text pairs, including: filtering Chinese text in the original image-text pairs based on language recognition tools; and retaining image-text pairs whose text length is within a preset word count range.

[0029] In this embodiment, the language recognition tool is pre-trained on a large corpus of text labeled with language categories, enabling it to output the probability distribution of the language category of the input text. Filtering Chinese text from the original image-text pairs based on the language recognition tool means retaining image-text pairs whose language labels are determined to be Chinese, and discarding those labeled as other languages. In this process, a confidence threshold is set, retaining only image-text pairs whose confidence in the Chinese labels output by the language recognition tool is higher than this threshold, thus reducing the false positive rate. The preset character count range has a lower limit of 5 Chinese words and an upper limit of 60 Chinese words. The text length can be counted using various methods. One method counts the number of Chinese characters, with one Chinese character counted as one character unit. Another method counts the number of tokens after Chinese word segmentation, with one token counted as one character unit. A suitable counting method can be selected based on the size of the dataset and the required processing efficiency. When using the character counting method, the preset character count range has a lower limit of 5 characters and an upper limit of 60 characters. When using the word counting method, the preset character count range has a lower limit of 3 words and an upper limit of 30 words. Regardless of the counting method used, the core purpose is to eliminate extreme samples that are too short or too long, so that the remaining text lengths are distributed within a range that is conducive to subsequent semantic analysis.

[0030] In this way, a large number of unqualified samples can be eliminated at the entrance of the entire filtering pipeline with extremely low computational cost, so that the candidate image-text pairs meet the basic requirements in terms of language attributes and text length, providing quality-controllable input for subsequent text refinement and cross-modal alignment filtering.

[0031] Optionally, the removed text is filtered for grammatical integrity, including: segmenting the removed text into words; discarding text that lacks nouns; and discarding text that has more than a preset threshold of unknown words.

[0032] In this embodiment, the word segmentation tool employs a segmenter based on a statistical model or a neural network model, capable of outputting the boundary position and part-of-speech tag of each word element in the input text. The determination of nouns depends on the part-of-speech tagging results; when the part-of-speech tag of a word element belongs to the noun category, that word element is identified as a noun. Text lacking nouns means that the text does not contain any nominal components representing specific objects, entities, or concepts. Such text cannot carry clear semantic content and should not be retained in the dataset. Discarding text lacking nouns ensures that the retained text contains at least one clear object or entity, enabling the text side of the image-text pair to possess basic semantic referential capabilities. Due to the openness and dynamism of the Chinese language, the vocabulary of any segmenter cannot cover all possible Chinese words. Rare characters, internet slang, proper nouns, misspelled words, and words containing special characters may all be marked as unknown words. A preset threshold of 5 is set; that is, when the number of unknown words in a text reaches 6 or more, the image-text pair containing that text is discarded. In another implementation, the proportion of unknown words in the total number of words in the text can be calculated. When the proportion exceeds a preset threshold, the unknown words are discarded. This method can adapt to the actual situation of texts of different lengths.

[0033] In this way, text quality can be finely controlled from two dimensions: grammatical integrity and parsability. This makes the text in the refined image-text pair more grammatically complete and more standardized in terms of lexical coverage, providing higher-quality text input for subsequent information entropy calculation and cross-modal alignment.

[0034] Optionally, perceptual clustering is performed on the remaining images to remove duplicates, including: extracting the embedding vectors of each image in the remaining images; calculating the cosine distance between each embedding vector in the remaining images; grouping images whose cosine distance is less than a preset clustering threshold into the same image group; and retaining the image in each image group that is closest to the centroid of the image group.

[0035] In this embodiment, the pre-trained visual model for extracting image embedding vectors employs a visual encoder in a dual-tower visual language model trained based on contrastive learning, capable of encoding input images of arbitrary size into feature vectors of fixed dimensions. Before extracting the embedding vectors, the images need to be preprocessed, including scaling them to a fixed input size and normalizing them. The dimension of the embedding vectors depends on the configuration of the visual encoder used, with common dimensions being 512, 768, or 1024. A preset clustering threshold is used to determine whether two images should be grouped into the same image group. When the cosine distance between the embedding vectors of two images is less than the threshold, the two images are considered perceptually highly similar. The value of the preset clustering threshold directly affects the granularity and intensity of deduplication. The smaller the threshold, the more lenient the deduplication condition, and the higher the diversity of retained images; the larger the threshold, the stricter the deduplication condition, and the fewer images retained. During the image grouping process, a disjoint-set data structure is used to achieve efficient connectivity clustering, merging image pairs that meet the similarity condition into the same set one by one, ultimately resulting in each set corresponding to an image group.

[0036] This allows for the identification of images that are semantically highly similar but differ in pixel details, thus achieving more accurate deduplication at the semantic level and improving the semantic distribution balance of the dataset.

[0037] Optionally, the images in the refined image-text pair are subjected to fidelity filtering, including: retaining images in the refined image-text pair whose aspect ratio is within a preset ratio range and whose minimum side length is greater than a preset pixel threshold; and removing images in the refined image-text pair whose pixel intensity change is lower than a preset change threshold.

[0038] In this embodiment, the preset aspect ratio range is between 1 / 3 and 3, meaning the image's aspect ratio is greater than or equal to 1 / 3 and less than or equal to 3. When the image's aspect ratio exceeds this range, the image is severely elongated in the horizontal or vertical direction, and its visual content typically loses normal recognizability due to geometric distortion. The preset pixel threshold is 100 pixels, meaning the smaller of the image's width and height is greater than 100 pixels. This limitation can exclude thumbnails or icon-type images with excessively low resolution and severely lacking visual details. The preset variation threshold is used to distinguish between valid and invalid images. For 8-bit grayscale images, the preset variation threshold is set to a small value. By eliminating images with excessively low pixel intensity variation, images with solid color backgrounds, test cards, and monochrome fills that do not contain meaningful visual content can be excluded.

[0039] In this way, the visual fidelity of an image can be quickly evaluated from both geometric and pixel statistical properties. Images that are obviously distorted, have insufficient resolution, or lack visual information can be removed with low computational cost, providing better visual quality image input for subsequent pixel entropy calculation and perceptual clustering deduplication.

[0040] Optionally, the fidelity filtering of the images in the refined image-text pair further includes: using Laplacian variance to detect the blur of the fidelity-filtered images in the refined image-text pair; and removing images in the refined image-text pair whose Laplacian variance is lower than a preset blur threshold.

[0041] In this embodiment, the Laplacian variance is calculated as follows: First, the color image is converted to a grayscale image. Then, the Laplacian operator is applied to the grayscale image for convolution to obtain a Laplacian response map. The variance of all pixel values ​​in the response map is then calculated; this variance is the Laplacian variance. A preset blur threshold of 1000 is used. When the Laplacian variance is less than 1000, the image-text pair containing that image is removed. In another implementation, Laplacian variance detection can be performed after the image is scaled to a uniform size to eliminate the influence of image resolution differences on the detection results.

[0042] This effectively filters out blurry images caused by motion blur, inaccurate focus, or excessive compression, ensuring that the retained images meet the visual clarity requirements of the pre-trained model for input image quality.

[0043] Optionally, information entropy calculation is performed on the text in the candidate image-text pair, including: calculating the information entropy based on the sum of the negative logarithms of the occurrence probabilities of each character in the text of the candidate image-text pair.

[0044] In this embodiment, information entropy is calculated using characters in the text as the basic statistical unit. For each character, the logarithm of its occurrence probability is calculated and negative. Then, the negative logarithms of all characters are weighted and summed according to their respective probabilities to obtain the information entropy of the text. The more diverse the types of characters in the text and the more evenly the occurrence probabilities of each character, the larger the summation result and the higher the information entropy. Conversely, when the types of characters in the text are singular or a few characters occupy the majority of the frequency, the smaller the summation result and the lower the information entropy. The calculation result of information entropy is related to the text length. For extremely short texts, even if the character distribution is even, the upper limit of its information entropy is limited. In one implementation, the information entropy of the text in the candidate image-text pair can be calculated first, and then a quantile can be selected as a threshold according to the distribution of information entropy. The text with the lowest 5% information entropy is discarded as low-quality text.

[0045] In this way, the information content of the text in the candidate image-text pairs can be evaluated using a unified information theory standard, effectively eliminating texts with low information content, and significantly improving the semantic density of the text in the refined image-text pairs.

[0046] Optionally, pixel entropy calculation is performed on the image after fidelity filtering, including: calculating pixel entropy based on the negative logarithm of the probability of occurrence of gray values ​​of each pixel in the image after fidelity filtering.

[0047] In this embodiment, after converting a color image to a grayscale image, the grayscale values ​​of each pixel are obtained. The grayscale values ​​range from 0 to 255, resulting in 256 grayscale levels. For each grayscale level, the logarithm of its occurrence probability is calculated and negative. Then, the negative logarithms of all grayscale levels are weighted and summed according to their respective probabilities to obtain the pixel entropy of the image. The richer the distribution of grayscale levels in the image and the more uniform the occurrence probability of each grayscale level, the higher the pixel entropy; when the grayscale level distribution in the image is concentrated, the pixel entropy is lower. In one implementation, a pixel entropy value can be selected as a threshold, and images with pixel entropy below this threshold are considered low-information images.

[0048] In this way, the visual information content of the images after fidelity filtering can be evaluated using a unified information theory standard, effectively eliminating images with low visual information content, and significantly improving the visual information density of images in diverse image-text pairs.

[0049] Optionally, obtaining the original image-text pairs also includes: filtering the source of the original image-text pairs based on a preset blacklist, and using a lightweight classifier to perform a preliminary content security screening of the original image-text pairs.

[0050] In this embodiment, the preset blacklist is a pre-constructed list containing identifiers of low-quality or unreliable sources. The source information of the original text-image pair is the Uniform Resource Locator, domain name, or website identifier of the webpage where the image is located. Filtering the source of the original text-image pair based on the preset blacklist means removing text-image pairs whose source information matches any entry in the blacklist. The lightweight classifier used for the initial content security screening is a classification model with approximately 1 million parameters, capable of outputting a security risk score or category label for the input text or image. The initial content security screening can be performed separately at the text and image dimensions, or jointly at the text-image pair dimension.

[0051] In this way, the raw data can be pre-screened from the two dimensions of source legality and content security, eliminating a large number of unqualified samples with low computational cost and reducing the waste of computational resources in subsequent processing stages.

[0052] Optionally, before calculating the information entropy of the text in the candidate image-text pairs, the method further includes: unifying the text in the candidate image-text pairs into Simplified Chinese.

[0053] In this embodiment, the unification to Simplified Chinese is achieved through a character conversion tool. An open Chinese conversion tool is used to convert traditional Chinese characters in the text to their corresponding simplified Chinese characters. The character conversion tool maintains a mapping table between traditional and simplified Chinese characters. For each character in the input text, its corresponding simplified Chinese character is searched in the mapping table; if it exists, it is replaced; otherwise, the original character is retained. The unification to Simplified Chinese not only includes character-level conversion but also the unification of full-width and half-width characters, converting full-width English letters, numbers, and punctuation marks to half-width forms.

[0054] This eliminates the differences between simplified and traditional characters in text data, providing a unified text input for subsequent processing steps such as information entropy calculation, syntactic integrity filtering, and cross-modal alignment, which helps improve the consistency and accuracy of each processing step.

[0055] Optionally, before unifying the text in the candidate image-text pairs to Simplified Chinese, the method further includes: performing language recognition on the text in the candidate image-text pairs to confirm that the text in the candidate image-text pairs is Chinese text.

[0056] In this embodiment, language recognition is performed before unifying to Simplified Chinese to ensure that all subsequent processing operations target Chinese text. One implementation uses a fast classifier based on character-level features, which quickly determines the language category using the character's encoding range or statistical distribution characteristics. Since Chinese characters are distributed within specific Unicode encoding intervals, the proportion of characters belonging to these intervals in the text can be statistically analyzed to quickly determine if the text is Chinese. Another implementation uses a neural network-based language classifier that outputs probability distributions for multiple language categories. Once the text in the candidate image-text pair is confirmed to be Chinese, the determination is made based on whether the probability of the Chinese category output by the language recognition model is higher than a preset confidence threshold.

[0057] This allows for a secondary verification of the language attributes of candidate image-text pairs before they enter the text refinement process, preventing non-Chinese texts from being missed in the initial language recognition screening and thus entering subsequent processing stages.

[0058] Optionally, after filtering the removed text for grammatical integrity and before calculating the information entropy of the text in the candidate image-text pairs, the method further includes: removing emojis and special characters from the text in the candidate image-text pairs.

[0059] In this embodiment, emoticons refer to the emoticon characters defined in the Unicode standard, and special characters refer to uncommon symbols other than regular Chinese characters, English letters, numbers, and Chinese punctuation marks. These emoticons and special characters do not contribute substantial semantic information to the text, but may introduce noise and interfere with the calculation of information entropy and the results of word segmentation. Removing emoticons and special characters is achieved by constructing a set of allowed characters. Each character in the text is traversed, and only characters belonging to the allowed character set are retained; characters not belonging to the allowed character set are deleted or replaced with empty strings. The allowed character set includes Chinese characters, English letters, numbers, and commonly used Chinese punctuation marks. In one implementation, the removal operation may further include compressing consecutively repeating characters, compressing multiple consecutively appearing identical characters into a single character.

[0060] This allows for further purification of the text content, eliminating the influence of non-semantic characters on subsequent information entropy calculations, and making the information entropy filtering results more accurately reflect the semantic information content of the text.

[0061] Optionally, after calculating the information entropy of the text in the candidate image-text pairs and removing texts with information entropy lower than a first preset threshold, the method further includes: using a content security detection model to perform security filtering on the text in the refined image-text pairs.

[0062] In this embodiment, the content security detection model has approximately 20 million parameters, offering stronger detection capabilities and higher accuracy compared to the lightweight classifier used in the data source selection phase. The content security detection model performs security classification on text at a finer granular level, identifying specific security categories such as advertising content, politically sensitive content, pornographic and vulgar content, and violent and terrorist content, and can output a risk score for each security category. When using the content security detection model for filtering, a comprehensive risk score threshold is set; when the risk score of a text exceeds this threshold, the text-image pair containing that text is removed. In one implementation, a third-party security detection service can be used for auxiliary judgment, obtaining the text's security review results by calling an external application programming interface.

[0063] This allows for further purification of the dataset from the perspective of text content security, eliminating samples containing unsafe content and ensuring that the final output target dataset meets security and compliance requirements.

[0064] Optionally, after performing fidelity filtering on the images in the refined image-text pair, the method further includes: performing security filtering on the fidelity-filtered images using an image content security detection model.

[0065] In this embodiment, the image content security detection model employs a multi-label classifier based on a deep convolutional neural network. It is trained on image data containing unsafe categories such as pornography, violence, terror, and politically sensitive images, and can output a risk score for each unsafe category for the input image. When using the image content security detection model for filtering, a comprehensive risk score threshold or a specific threshold for a particular security category is set. When the risk score of an image exceeds the corresponding threshold, the image-text pair containing that image is removed. The image content security detection model can be deployed on a graphics processor to achieve efficient batch inference.

[0066] In this way, the dataset can be cleaned from the perspective of image content security, eliminating samples containing unsafe visual content and ensuring the compliance of the target dataset in terms of content security.

[0067] Optionally, after obtaining the target dataset, the method further includes: summarizing the image-text pairs from different processing batches, and performing perceptual clustering to remove duplicates on the summarized image-text pairs again.

[0068] In this embodiment, due to the large scale of the original data, the entire filtering pipeline is divided into multiple processing batches. Each processing batch independently performs steps such as data source selection, text refinement, visual diversification, and cross-modal cross-batch filtering on its assigned subset of original image-text pairs, obtaining the intermediate results for that batch. After all processing batches are completed, the image-text pairs from different processing batches are aggregated. During the aggregation process, redundant image-text pairs across batches may occur due to batch division, meaning that the same or highly similar image-text pairs are included in different processing batches. To eliminate this cross-batch redundancy, the aggregated image-text pairs need to be deduplicated again using perceptual clustering, the specific implementation of which is the same as the perceptual clustering deduplication in the visual diversification stage. The preset clustering threshold for cross-batch redundancy removal is the same as the preset clustering threshold in the visual diversification stage. Since the amount of data after cross-batch aggregation is still very large, to improve deduplication efficiency, local sensitive hashing and other approximate nearest neighbor retrieval techniques can be used after extracting the embedding vectors to accelerate the search for similar images.

[0069] This eliminates duplicate samples across batches introduced by batch partitioning, resulting in a lower overall redundancy in the final output target dataset. It maintains semantic diversity while achieving higher data purity and a more compact data size.

[0070] In practical applications, such as Figure 2As shown, the entire process includes a data source selection step, a text reinforcement step, a visual diversification step, and a cross-modal cross-batch filtering step. In the data source selection step, language control (Label='zho') is applied to the original image-text pairs to filter Chinese text. Image-text pairs from low-quality sources are removed through source reliability filtering (baidu.com). Basic text length constraints (5~60 words) are applied to the text, and initial content safety screening is performed. This step outputs 475.1M image-text pairs to the text reinforcement step. In the TextReinforcement step, following the output of the data source selection step, the text in the received image-text pairs is standardized to Simplified Chinese, and grammatical integrity is filtered (Nouns, [UNK]) to retain text with complete grammatical structure. The text is filtered for information density (emojis, special chars) to remove text with low information content, and content safety is performed. This step outputs 397.2M image-text pairs to the Visual Diversification step. In the Visual Diversification step, following the output of the Text Refinement step, visual fidelity filtering (Edge Length & pixel) is performed on the images in the received image-text pairs to remove images with poor visual quality. Image information density filtering is performed to remove images with low visual information content. Perceptual and Semantic Redundancy: Union-Find Algorithm is used to remove image-level redundancy, and image content safety filtering is performed. This step outputs 178.6M image-text pairs to the Cross-Modal Cross-Batch Filtering step.In the Cross-Modal Cross-Batch Filtering step, following the output of the visual diversification step, a Cross-Modal Alignment Assessment is performed on the received image-text pairs. Samples meeting the image-text relevance requirements are selected based on cross-modal similarity distance. The figure shows weak semantic correlation samples and image-caption matching samples. Then, Cross-Batch Redundancy Removal eliminates duplicate image-text pairs from different batches. An example of a redundant image-text pair to be removed is shown in the figure (Apollo space rocket light). This step ultimately outputs 100M image-text pairs as the target dataset.

[0071] Figure 3 This paper illustrates the application effect of a large-scale Chinese image and text data construction method provided in the embodiments of this application, specifically comparing the two indicators of text content word density and text perplexity of different datasets. Figure 3The diagram includes (a) a comparison of text content word density on the left and (b) a comparison of text perplexity on the right. Each diagram contains three curves, corresponding to the target dataset (DanQing) constructed in this application, the Wukong dataset (Wukong) from related technologies, and the Zero dataset (Zero) from related technologies. In the (a) text content word density comparison diagram on the left, the horizontal axis represents the text content word density, ranging from 0.4 to 1.0, and the vertical axis represents the cumulative distribution ratio, ranging from 0.0 to 1.0. The three curves represent the cumulative distribution of text content word density for the three datasets, with the curve closer to the right indicating a higher text word density for that dataset. In areas of low word density, the curve for the target dataset (DanQing) is located to the lower left of the other two curves, indicating that the proportion of low word density samples in the target dataset is lower than in the other datasets. In areas of high word density, the curve for the target dataset (DanQing) is located to the upper right of the other two curves, indicating that the proportion of high word density samples in the target dataset is higher than in the other datasets. In the text perplexity comparison graph on the right (b), the horizontal axis represents text perplexity, ranging from 0 to 80, and the vertical axis represents the cumulative distribution ratio, ranging from 0.0 to 1.0. The three curves represent the cumulative distribution of text perplexity for the three datasets, with the curve closer to the left indicating lower text perplexity. Throughout the entire horizontal axis range, the curve for the target dataset (DanQing) is located to the left of the other two curves, indicating that the perplexity of the target dataset is lower than that of the other datasets at the same cumulative distribution ratio. This demonstrates that the target dataset constructed in this application outperforms the Wukong dataset and the Zero dataset in related technologies in terms of text semantic coherence.

[0072] Figure 4 This paper illustrates the application effect of another large-scale Chinese image and text data construction method provided in the embodiments of this application. Specifically, it compares the matching ability of models trained on different datasets for multiple images under two different concepts, which is used to evaluate the model's ability to distinguish emerging concepts. Figure 4The chart includes a comparison of the Wukong concept identification results (a) on the left and a comparison of the Xiaomi concept identification results (b) on the right, each presented as a horizontal bar chart. In both charts, the horizontal axis represents the matching ability value, and the vertical axis represents the model or dataset category, from top to bottom: Baseline model, TaShu dataset, Wukong dataset, Zero dataset, and DanQing dataset. In the Wukong concept identification result comparison chart (a) on the left, each row contains three bars of different colors, representing the matching ability value of the model or dataset for three types of Wukong-related images: Wukong from Dragon Ball, Wukong from Journey to the West, and Wukong from Black Myth. The baseline model has a matching ability value of 0.334 for the Wukong image from Dragon Ball, 0.328 for the Wukong image from Journey to the West, and 0.340 for the Wukong image from Black Myth. The matching ability scores for the following datasets are as follows: The *Tashu* dataset has a matching ability of 0.332 for Goku images from *Dragon Ball*, 0.322 for Goku images from *Journey to the West*, and 0.350 for Goku images from *Black Myth*. The *Goku* dataset has a matching ability of 0.323 for Goku images from *Dragon Ball*, 0.325 for Goku images from *Journey to the West*, and 0.345 for Goku images from *Black Myth*. The zero dataset has a matching ability of 0.330 for Goku images from *Dragon Ball*, 0.331 for Goku images from *Journey to the West*, and 0.342 for Goku images from *Black Myth*. The target dataset has a matching ability of 0.321 for Goku images from *Dragon Ball*, 0.330 for Goku images from *Journey to the West*, and 0.347 for Goku images from *Black Myth*. In the comparison chart of Xiaomi concept identification results (b) on the right, each row contains three bars of different colors, representing the matching ability of the model or dataset for three Xiaomi-related images: the Xiaomi brand logo, Xiaomi phones, and Xiaomi SU7. Specifically, the baseline model has a matching ability of 0.334 for the Xiaomi brand logo image, 0.328 for the Xiaomi phone image, and 0.340 for the Xiaomi SU7 image. The Tashu dataset has a matching ability of 0.332 for the Xiaomi brand logo image, 0.322 for the Xiaomi phone image, and 0.350 for the Xiaomi SU7 image. The Wukong dataset has a matching ability of 0.323 for the Xiaomi brand logo image, 0.325 for the Xiaomi phone image, and 0.345 for the Xiaomi SU7 image.The zero dataset achieves a matching ability score of 0.330 on the Xiaomi brand logo image, 0.331 on the Xiaomi phone image, and 0.342 on the Xiaomi SU7 image. The target dataset achieves a matching ability score of 0.321 on the Xiaomi brand logo image, 0.330 on the Xiaomi phone image, and 0.347 on the Xiaomi SU7 image. It can be seen that the target dataset obtained using the large-scale Chinese image and text data construction method provided in this application demonstrates matching ability scores superior to or equal to the zero dataset and the Wukong dataset in related technologies for the emerging concept categories of "Black Myth Wukong" and "Xiaomi SU7" (which appeared after 2024). This indicates that the target dataset constructed in this application has a stronger ability to distinguish emerging concepts and can effectively improve the visual language pre-training model's ability to learn and understand contemporary knowledge with strong timeliness.

[0073] Combination Figure 5 As shown, this application embodiment provides a large-scale Chinese image and text data construction apparatus 800, including a processor 801 and a memory 802. Optionally, the apparatus may further include a communication interface 803 and a bus 804. The processor 801, communication interface 803, and memory 802 can communicate with each other via the bus 804. The communication interface 803 can be used for information transmission. The processor 801 can call logical instructions in the memory 802 to execute the large-scale Chinese image and text data construction method of the above embodiment.

[0074] Furthermore, the logic instructions in the aforementioned memory 802 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0075] The memory 802, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 801 executes functional applications and data processing by running the program instructions / modules stored in the memory 802, thereby realizing the method for constructing large-scale Chinese image and text data in the above embodiments.

[0076] The memory 802 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 802 may include high-speed random access memory and may also include non-volatile memory.

[0077] This application provides a computer-readable storage medium storing computer-executable instructions configured to execute the aforementioned method for constructing large-scale Chinese graphic data.

[0078] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for constructing large-scale Chinese image and text data, characterized in that, include: Obtain the original image-text pairs, and perform language filtering and basic text length filtering on the original image-text pairs to obtain candidate image-text pairs; The information entropy of the text in the candidate image-text pairs is calculated and the text with information entropy lower than the first preset threshold is removed. The removed text is then filtered for grammatical integrity to obtain refined image-text pairs. The images in the refined image-text pairs are subjected to fidelity filtering. The images after fidelity filtering are subjected to pixel entropy calculation and images with pixel entropy lower than the second preset threshold are removed. Then, the remaining images are subjected to perceptual clustering to remove duplicates, resulting in diverse image-text pairs. For diverse image-text pairs, calculate the cross-modal similarity distance between the image and the text, retain only image-text pairs whose cross-modal similarity distance is within a preset range, and obtain and output the target dataset.

2. The method according to claim 1, characterized in that, The original image-text pairs are filtered by language and basic text length, including: Chinese text in the original image-text pairs was filtered using a language recognition tool; Retain text-image pairs whose original text length is within the preset character limit.

3. The method according to claim 1, characterized in that, Perform grammatical integrity filtering on the removed text, including: Perform word segmentation on the removed text; Discard any text that lacks nouns after the removal process; Discard text in which the number of unknown terms exceeds a preset threshold after removal.

4. The method according to claim 1, characterized in that, Perceptual clustering is performed on the remaining images for deduplication, including: Extract the embedding vectors of each image from the remaining images; Calculate the cosine distance between each embedding vector in the remaining image; Images whose cosine distance is less than a preset clustering threshold are grouped into the same image group; Keep the image in each image group that is closest to the centroid of the image group.

5. The method according to claim 1, characterized in that, Fidelity filtering is applied to images within refined text-image pairs, including: Retain images from refined image-text pairs whose aspect ratio is within a preset range and whose minimum side length is greater than a preset pixel threshold. Images in the refined image-text pair whose pixel intensity changes are below a preset change threshold are removed.

6. The method according to claim 1, characterized in that, The information entropy of the text in the candidate image-text pairs is calculated, including: For the text in the candidate image-text pair, the information entropy is calculated by summing the negative logarithms of the probability of each character appearing in the text of the candidate image-text pair.

7. The method according to claim 1, characterized in that, Pixel entropy calculation is performed on the image after fidelity filtering, including: For the image after fidelity filtering, the pixel entropy is calculated by summing the negative logarithms of the probability of each pixel's grayscale value appearing in the image after fidelity filtering.

8. A device for constructing large-scale Chinese image and text data, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute, when running the program instructions, a method for constructing large-scale Chinese graphic data as described in any one of claims 1 to 7.

9. A system for constructing large-scale Chinese image and text data, characterized in that, include: The system ontology for constructing large-scale Chinese image and text data; and, The apparatus for constructing large-scale Chinese image and text data as described in claim 8 is installed on the main body of the system for constructing large-scale Chinese image and text data.

10. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed, they cause the computer to perform the method for constructing large-scale Chinese graphic data as described in any one of claims 1 to 7.

Images