Sorting and simplifying method and system for mixed modal data and medium

Abstract

The invention discloses a sorting and simplifying method and system for mixed modal data and a medium. The method comprises the steps that an original data file is input, content inspection and semantic analysis are conducted on the original data file, and multi-modal data in the original data file are recognized and classified into subsets of multiple specific modals; each subset is sent to a corresponding deduplication pipeline, and each deduplication pipeline correspondingly executes specific modal data deduplication; and each deduplication pipeline completes deduplication of the specific modal data and outputs simplified data. According to the method, the multi-modal data recognition accuracy is high, the data volume can be remarkably reduced through targeted duplicate removal after classification, and original network data can be converted into a high-quality training corpus.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to methods, systems and media for sorting and simplifying mixed-modal data. Background Technology

[0002] The effectiveness of modern large-scale machine learning models is inextricably linked to the scale and quality of their training data. Web crawler data is a primary source of such data, but leveraging this raw web data presents significant data engineering challenges. Raw web data often manifests as a chaotic “digital swamp,” a term defined as an unstructured and heterogeneous mixture with two key obstacles: first, inherent heterogeneity, where raw web data chaotically mixes multiple modalities within a corpus, such as images (.png, .jpg), audio (wav, mp3), text (.txt, .json), obscure binary files (.bin), and metadata links; and second, the dataset is rife with redundancy, including not only exact repetitions but also more subtle near-duplicates. Examples include content from templates, paraphrases, cross-site aggregations, or different encodings of the same audio.

[0003] While the importance of data quality is well-known, existing data preparation and cleaning methods fall short in dealing with the complexities of the "digital swamp": First, existing data cleaning frameworks primarily target error correction in structured or semi-structured data (such as relational tables), implicitly assuming that the input data has been classified and is single-modal, lacking the ability to handle raw, mixed, and messy web data; second, modality-specific NDD techniques, although near-duplicate detection (NDD) techniques for modality-specific data (e.g., for text data) are already very advanced, they also assume that the input data has been pre-classified. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a method, system, and medium for sorting and simplifying mixed modal data, thereby solving the problem of the lack of an integrated and automated method in the prior art to guide and process the initial chaotic state of raw mixed modal data collection.

[0005] To address the aforementioned technical problems, this application provides a method for sorting and simplifying mixed-modal data, comprising the following steps: First, inputting an original data file, the original data file including mixed multimodal data; Second, performing content inspection and semantic parsing on the original data file, identifying and classifying the multimodal data therein into subsets of multiple specific modalities; Third, each subset is sent to a corresponding deduplication pipeline, and each deduplication pipeline performs deduplication of the specific modal data respectively; Fourth, each deduplication pipeline completes the deduplication of the specific modal data and outputs simplified data.

[0006] In some embodiments, in the first step, the mixed multimodal data includes multiple specific modal data of video, audio and / or text, and each specific modal data includes multiple types.

[0007] In some embodiments, the second step includes: reading the identifier bytes in the header of the original data file to quickly identify common binary formats; calculating the proportion of printable characters in the original data file and using a threshold to distinguish binary data based on the corpus and non-text; JSON semantic parsing, with the trigger condition being that files that are still unclear after the first two steps are uniformly designated as unknown categories.

[0008] In some embodiments, in the third step, for specific modal data that is image data, image data deduplication is performed, corresponding to the execution by the first deduplication pipeline: using the CLIP-VIT-B-16 model, the input image is resized to 256×256 pixels and cropped to 224×224 pixels, corresponding to the output being the extracted 512-dimensional embedding vector; all 512-dimensional embedding vectors are L2 normalized, and the 512-dimensional embedding vectors are placed on a unit hypersphere, and cosine similarity is calculated; GPU-accelerated K-Means clustering is used, the high-dimensional semantic space is divided into multiple clusters and iterated; within each cluster, a pairwise cosine similarity matrix is ​​calculated, i.e., for each cluster... Each image in Find its maximum similarity to any other image in the cluster. Threshold determination, setting a distance threshold. Corresponding to the similarity threshold ,if Then retain the image. Otherwise, treat it as a duplicate and discard it.

[0009] In some embodiments, in the third step, for specific modal data that is audio data, audio data deduplication is performed, which is performed by the second deduplication pipeline: time-frequency feature extraction, inputting the audio signal, converting it into a complex spectrum D through a short-time Fourier transform, and then applying a logarithmic transform. The logarithmic spectrum is obtained to compress the dynamic range and emphasize weak signal conduction. Adaptive binarization is performed by calculating a global adaptive threshold for the logarithmic spectrum using Otsu's thresholding method. The time-frequency cell is set to 1 if and only if its amplitude exceeds: Otsu's threshold + an empirically optimized 6dB offset; otherwise, it is set to 0. Feature normalization involves resampling the logarithmic spectrum to normalize it to a standard 32×128 resolution, then flattening this 32×128 matrix to output a 4096-dimensional binary feature vector. LSH signatures are performed using MinHash: a MinHash signature matrix is ​​generated for the 4096-dimensional binary feature vectors of n samples. LSH bucketing is then performed: the MinHash signature matrix is ​​divided into b strips, each with r rows. Sample pairs with the same hash value in any strip are grouped as candidate duplicate pairs. Similarity verification is performed only on the candidate duplicate pairs, calculating Jaccard similarity. Similarity verification methods: , Threshold judgment, similarity threshold If the Jaccard similarity is greater than 0.7, the candidate duplicate pair is considered a duplicate, and one of the samples is randomly retained.

[0010] In some embodiments, in the third step, for specific modal data that is text data, text data deduplication is performed, which is performed by a third deduplication pipeline: Text standardization involves preprocessing, including case normalization, merging redundant whitespace, removing special characters, and trimming leading and trailing spaces. Multi-granularity N-gram feature extraction is performed: for character-level N-grams, a sliding window with k=3 is used to generate 3-grams; for word-level N-grams, after word segmentation, word-level 3-grams are generated using k=3. Finally, the character and word sets are merged. The final feature set is the union of these two sets; Jaccard similarity is calculated as follows: ; Layered filtering and optimization employ a three-layer filtering mechanism: (a) Length pre-filtering: If the length difference between two texts exceeds 50%, they are assumed not to be duplicates and the comparison is skipped; (b) Sample-based comparison: Each new text is compared only with the S most recently retained texts; (c) Threshold judgment: The similarity threshold τ = 0.8. If the Jaccard similarity > τ, it is classified as a near-duplicate.

[0011] This application also discloses a system for sorting and simplifying mixed modal data, comprising: an input unit for inputting an original data file, the original data file including mixed multimodal data; a classification unit for performing content inspection and semantic parsing on the original data file, and classifying the multimodal data therein into subsets of multiple specific modalities; a deduplication unit for sending each subset into a corresponding deduplication pipeline, and each deduplication pipeline performing deduplication of the specific modal data respectively; and an output unit for outputting simplified data after each deduplication pipeline has completed the deduplication of the specific modal data.

[0012] This application also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the preceding methods.

[0013] The beneficial effects of this application are as follows: This application discloses a method, system, and medium for sorting and simplifying mixed-modal data. The method includes inputting an original data file, performing content inspection and semantic parsing on the original data file, and identifying and classifying the multimodal data therein into subsets of multiple specific modalities; each subset is sent to a corresponding deduplication pipeline, and each deduplication pipeline performs deduplication of the specific modal data respectively; each deduplication pipeline completes the deduplication of the specific modal data and outputs simplified data. This method has high accuracy in identifying multimodal data, and the targeted deduplication after classification can significantly reduce the amount of data, transforming the original network data into a high-quality training corpus. Attached Figure Description

[0014] Figure 1 This is a flowchart of an embodiment of a method for sorting and simplifying mixed-modal data according to this application; Figure 2 This is a flowchart of data classification in an embodiment of a method for sorting and simplifying mixed-modal data according to this application; Figure 3 This is a flowchart of image data deduplication in an embodiment of a method for sorting and simplifying mixed modal data according to this application. Detailed Implementation

[0015] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0016] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0017] The embodiments will now be described in detail with reference to the accompanying drawings.

[0018] refer to Figure 1 This embodiment provides a method for sorting and simplifying mixed-modal data, including the following steps: S101: Step 1, input the raw data file, which includes mixed multimodal data; S102: The second step is to perform content inspection and semantic parsing on the original data file, and to identify and classify the multimodal data therein into a subset of multiple specific modalities; S103: In the third step, each subset is sent to the corresponding deduplication pipeline, and each deduplication pipeline performs deduplication of specific modal data respectively. S104: Fourth step, each of the deduplication pipelines completes the deduplication of the specific modal data and outputs simplified data.

[0019] Through the above steps, this application provides an end-to-end data sorting and simplification method specifically designed to clean up "digital swamps". This invention embodies a strategic "Tiage-and-Clean" design concept, which aims to automatically transform raw, messy raw network data into a high-quality, non-redundant corpus optimized for downstream machine learning applications.

[0020] Preferably, in the first step S101, the raw data file includes mixed multimodal data, such as the raw WARC archive file from Common Crawl, which contains multimodal data such as video, audio, and text. Each specific modality of data can include multiple types, such as image data, which can include multiple different types of image data such as .png and .jpg.

[0021] In step S102, a lightweight but robust classifier can be used that reads unstructured raw data files and uses a series of content-driven inspections, including magic number sniffing, printability analysis, and JSON semantic parsing, to classify mixed input files into subsets of specific modalities (e.g., images, audio, text).

[0022] Furthermore, in step S102, the classifier (Sorter) is the first key subsystem of this invention, implementing a "multi-layered hybrid heuristic" approach. It is a content-driven inspection cascade designed to achieve high throughput and robustness to "dirty" raw network data (such as missing / incorrect extensions). Figure 2 As shown, the method (process) is as follows: S201: Read the identifier bytes (e.g., '0x89 PNG', 'RIFF') in the file header of the original data file to quickly identify common binary formats (such as PNG / GIF, RIFF / WAV); S202: Calculate the proportion of printable characters in the original data file. Use a "printability threshold" to distinguish between corpus-based (e.g., .txt, .json) and non-text binary data. This step can be considered printability analysis and can be performed concurrently with step S201.

[0023] S203: JSON semantic parsing, trigger condition: for files that are still ambiguous after the first two steps (e.g., files that are identified as "text" but do not have a clear extension, or files with a common extension such as .json).

[0024] This is a non-obvious depth check; this application parses the JSON structure and performs the following: (1) Field voting: Check the name of the JSON key (e.g., “text”, “image_url”, “caption”); (2) URL suffix analysis: Check the URL in the field value (e.g., if a JSON file is full of URLs pointing to .jpg files, it may be classified as image metadata).

[0025] The purpose of JSON semantic parsing is to infer the true modality of the data included in the original data file, rather than just its container type.

[0026] Furthermore, in step S102, if the original data file is subjected to content inspection and semantic parsing, and there is still data whose modality type cannot be determined, such as missing or incorrect file extensions and incorrectly formatted containers, then these are uniformly designated as the "unknown" category.

[0027] Furthermore, regarding step S103, this invention intentionally avoids using a single, uniform semantic model (e.g., a massive multimodal model) to process all data, as the computational cost in the petabyte-scale "digital swamp" is prohibitively high. Instead, this invention employs a pragmatic hybrid-depth strategy, specifically manifested as follows: For image data, redundancy is often semantic (e.g., different photos of the same object). Therefore, this invention argues that the high computational cost of using deep semantic models (such as CLIP) is necessary and justified.

[0028] For text and audio: redundancy is often syntactic (e.g., boilerplate text) or acoustic (e.g., different encodings of the same recording). For these modalities, this invention prioritizes computational efficiency and throughput, employing highly optimized non-semantic methods (e.g., multi-granularity N-grams and acoustic fingerprinting). These techniques serve as powerful and scalable first-round filters.

[0029] Therefore, the core of the scalability and practicality of this invention lies in the specific engineering trade-off between the depth of the cleaning algorithm and the modal redundancy properties, which involves using deep semantic models (such as CLIP) and non-semantic methods (such as multi-granularity N-gram and acoustic fingerprinting) in the third step of S103.

[0030] Furthermore, such as Figure 3 As shown, for specific modal data, image data deduplication is performed, which is executed by the first deduplication pipeline: S301: Employs the CLIP-VIT-B-16 model. The input image is resized to 256×256 pixels, center-cropped to 224×224, converted to a tensor, and normalized. The corresponding output is the extracted 512-dimensional embeddings.

[0031] S302: L2 normalization. All 512-dimensional embedding vectors are L2 normalized. This is a key prerequisite for subsequent steps, ensuring that the vectors lie on the unit hypersphere, so that the computationally expensive Euclidean distance can be replaced by the computationally cheaper and equivalent cosine similarity (calculated via dot product).

[0032] S303: Hierarchical clustering, using GPU-accelerated K-Means clustering (implemented via the Faiss library). This choice perfectly matches step S302 (L2 normalization), ensuring that K-Means clustering is meaningful in the high-dimensional semantic space. Preferably, the high-dimensional semantic space is divided into k=2000 sub-regions (clusters), iterated 100 times. This invention employs a "spherical K-Means variant," a non-obvious choice that uses cosine distance instead of Euclidean distance in standard K-Means.

[0033] S304: Calculate the pairwise cosine similarity matrix within each cluster, i.e., for each cluster... Each image in Find its maximum similarity to any other image in the cluster. .

[0034] S305: Threshold determination, distance threshold Corresponding to similarity threshold ,if Then retain the image. Otherwise, treat it as a near-duplicate and discard it. The "divide and conquer" method of the present invention reduces complexity. Table 1 shows the parameter definitions for image deduplication using the above method.

[0035]

[0036] Furthermore, for specific modal data, audio data deduplication is performed, corresponding to the second deduplication pipeline. The goal is to identify acoustically identical or nearly identical recordings (e.g., different encodings, different noise levels), prioritizing computational throughput. The method involves converting the audio into a high-dimensional binary vector and using Locality Sensitive Hashing (LSH) for efficient matching. Specifically, this includes: Step 401: Time-frequency feature extraction. Input audio signal y, convert it into a complex spectrum D through short-time Fourier transform (STFT) (using the Hanning window), and then apply logarithmic transform. To compress the dynamic range and emphasize weak signal-guided characteristics.

[0037] Step 402: Adaptive Binarization. This is a key and innovative step in the audio processing of this invention, aiming to generate a noise-insensitive binary feature. First, a globally adaptive threshold for the logarithmic spectrogram is calculated using Otsu's thresholding method. A time-frequency cell is set to 1 (true) if and only if its amplitude exceeds: the Otsu threshold + an empirically optimized 6dB offset; otherwise, it is set to 0. This method effectively separates significant acoustic features from background noise.

[0038] Step 403, feature standardization, involves resampling and normalizing the binary spectrogram to a standard 32×128 resolution, then flattening this 32×128 matrix to output a 4096-dimensional (32×128=4096) binary feature vector (fingerprint). The goal is to ensure that all samples have consistent dimensions, regardless of the original audio duration.

[0039] Step 404: Efficient Similarity Retrieval Based on LSH. LSH signature is performed using MinHash: MinHash signatures are generated for the 4096-dimensional fingerprint matrix of n samples; LSH bucketing: The signature matrix is ​​divided into b strips, each strip having r rows (h=b×r). Sample pairs with the same hash value in any strip (i.e., collisions) are grouped into "candidate near-duplicate pairs".

[0040] Step 405: Similarity verification. Perform the computationally expensive Jaccard similarity verification only on "candidate pairs" and not all possible pairs. Jaccard similarity verification method: .

[0041] Step 406: Threshold determination. The similarity threshold r = 0.7. If the Jaccard similarity > 0.7, the pair is considered nearly duplicated, and one of the samples is randomly retained. This algorithm achieves a time complexity far superior to existing technologies. Table 2 shows the parameter definitions for audio deduplication using the above method.

[0042]

[0043] Furthermore, for specific modal data, text data deduplication is performed, which is handled by a third deduplication pipeline. A highly optimized multi-granularity N-gram method is employed to eliminate syntactic repetitions (e.g., boilerplate documents, copied and pasted articles), prioritizing the highest computational throughput.

[0044] Step 501: Text Standardization. Perform rigorous preprocessing: case normalization, merging redundant whitespace, deleting special characters, and trimming leading and trailing spaces, etc. Step 502: Multi-granularity N-gram feature extraction. This is a key and novel step in the text processing of this invention. This invention fuses two feature sets of different granularities to simultaneously capture local spelling patterns and broader phrase structures.

[0045] For character-level N-grams, a sliding window with k=3 is used to generate 3-grams (e.g., “the” → {“the”}); for word-level N-grams, after word segmentation, k=3 is used to generate word-level 3-grams (e.g., “the quick brown” → {“the quick brown”}).

[0046] Further integration involves merging the character set and the word set, namely: The final feature set is the union of these two sets. This solves the problem that single-granularity methods (such as pure character N-grams) cannot capture phrase structure, while pure word N-grams are sensitive to spelling errors.

[0047] Step 503: Jaccard similarity calculation, the calculation method is as follows: .

[0048] Step 504: Layered filtering and optimization. This is the second key and innovative step in the text processing of this invention, designed to address the scalability issue. The method employs a three-layer filtering mechanism: (a) Length pre-filtering: If the length difference between two texts exceeds 50%, they are assumed not to be duplicates and the comparison is skipped; (b) Sample-based comparison: This invention does not compare each new text with all the retained texts; instead, it compares it only with the S most recently retained texts.

[0049] (c) Threshold judgment: similarity threshold τ=0.8. If Jaccard similarity>τ, it is classified as nearly duplicate.

[0050] Step 505: Cross-partition deduplication. As a final step, this invention ensures deduplication across the training, test, and validation sets to prevent data leakage.

[0051] Similarly, the time complexity of the above methods for text processing is significantly reduced. Table 3 shows the parameter definitions for text deduplication using the above methods.

[0052]

[0053] The core contribution of this invention is to provide a novel, end-to-end "Tiage-and-Clean" framework for cleaning up "digital swamps".

[0054] In terms of overall architecture, this invention provides a data cleaning system (MMdedup framework) comprising: (a) a classifier module (Sorter) for classifying mixed modality input data into modality-specific subsets based on content-driven heuristic rules; and (b) multiple dedicated deduplication pipelines, each targeting a modality subset (such as images, audio, and text) and applying a near-duplicate detection algorithm specific to that modality.

[0055] In terms of implementation, this invention provides a data cleaning method comprising two stages of a "Triage-and-Clean" architecture: (a) firstly, an automated triage step is performed to separate heterogeneous data streams; (b) then, a specific modality deduplication step is performed on each separated stream. Specifically, it includes: The method for classifying mixed-modal data performs a specific multi-level cascading check: (a) first, it applies format identification and printability analysis based on file header byte features for rapid classification; (b) then applies JSON semantic parsing to ambiguous files (such as JSON), which includes "field voting" and "URL suffix analysis" to infer their content modality.

[0056] A scalable semantic image deduplication method includes the following steps: (a) extracting 512-dimensional embeddings using CLIP and performing L2 normalization; (b) applying GPU-accelerated spherical K-Means (using cosine distance) to cluster the normalized embeddings; and (c) computing pairwise cosine similarity only within each cluster (a "divide and conquer" strategy) to achieve deduplication with lower complexity.

[0057] The audio deduplication method includes the following steps: (a) generating an STFT logarithmic spectrogram; (b) applying adaptive binarization using a specific 0tsuThreshold +6 dB formula to create a noise-robust binary fingerprint; (c) normalizing the fingerprint to a fixed 4096-dimensional vector; and (d) validating the fingerprint using MinHash LSH and Jaccard. Perform efficient matching.

[0058] A scalable syntactic text deduplication method comprises the following steps: (a) "multi-granularity" feature extraction, which is the union of character-level 3-gram sets and word-level 3-gram sets; and (b) employing a "layered pass" mechanism that combines length pre-filtering and sample-based comparison (comparing only the S most recent samples). This application also provides a computer-readable storage medium storing program code that can be called by a processor to execute the methods described in the above method embodiments.

[0059] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, computer-readable storage media include nontransitory computer-readable media. A computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed in an appropriate form.

[0060] Therefore, this application discloses a method, system, and medium for sorting and simplifying mixed-modal data. The method includes inputting an original data file, performing content inspection and semantic parsing on the original data file, and identifying and classifying the multimodal data therein into subsets of multiple specific modalities. Each subset is fed into a corresponding deduplication pipeline, which performs deduplication for the specific modality data. Each deduplication pipeline completes the deduplication of the specific modality data and outputs simplified data. This method has high accuracy in identifying multimodal data, and the targeted deduplication after classification can significantly reduce the data volume, transforming original network data into a high-quality training corpus.

[0061] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A method for sorting and simplifying mixed-modal data, characterized in that, Including the following steps: The first step is to input the raw data file, which includes mixed multimodal data; The second step is to perform content inspection and semantic parsing on the original data file, and to identify and classify the multimodal data therein into subsets of multiple specific modalities; Third, each subset is sent to the corresponding deduplication pipeline, and each deduplication pipeline performs deduplication of specific modal data respectively. Fourth, each deduplication pipeline completes the deduplication of the specific modal data and outputs simplified data.

2. The method for sorting and simplifying mixed-modal data according to claim 1, characterized in that, In the first step, the mixed multimodal data includes multiple specific modal data of video, audio and / or text, and each specific modal data includes multiple types.

3. The method for sorting and simplifying mixed-modal data according to claim 1, characterized in that, The second step includes: Read the identifier bytes in the header of the original data file to quickly identify common binary formats; Calculate the proportion of printable characters in the original data file and use a threshold to distinguish between binary data based on the corpus and non-text data. For files that remain unclear after the first two steps, JSON semantic parsing is triggered, and they are uniformly designated as unknown.

4. The method for sorting and simplifying mixed-modal data according to claim 1, characterized in that, In the third step, image data deduplication is performed for specific modal data, which is image data, and is executed by the first deduplication pipeline: Using the CLIP-VIT-B-16 model, the input image is resized to 256×256 pixels and cropped to 224×224 pixels at the center, resulting in an output of a 512-dimensional embedding vector. All 512-dimensional embedding vectors are L2 normalized, and the 512-dimensional embedding vectors are located on a unit hypersphere. The similarity is calculated using cosine similarity. GPU-accelerated K-Means clustering divides the high-dimensional semantic space into multiple clusters and iterates through them; Within each cluster, a pairwise cosine similarity matrix is ​​calculated, i.e., for each cluster... Each image in Find its maximum similarity to any other image in the cluster. ; Threshold determination, setting a distance threshold Corresponding to the similarity threshold ,if Then retain the image. Otherwise, treat it as a duplicate and discard it.

5. The method for sorting and simplifying mixed-modal data according to claim 1, characterized in that, In the third step, for specific modal data that is audio data, audio data deduplication is performed, which is executed by the second deduplication pipeline: Time-frequency feature extraction involves inputting an audio signal, converting it into a complex spectrum D using a short-time Fourier transform, and then applying a logarithmic transform. Obtain the logarithmic spectrum; Adaptive binarization is performed by calculating a global adaptive threshold for the logarithmic spectrum using Otsu's thresholding method, wherein the corresponding time-frequency cell is set to 1 if and only if its magnitude exceeds: Otsu threshold + an empirically optimized 6dB offset, otherwise it is set to 0; Feature standardization involves resampling the logarithmic spectrum to normalize it to a standard 32×128 resolution, then flattening this 32×128 matrix to output a 4096-dimensional binary feature vector. LSH signatures are performed using MinHash-based LSH: a MinHash signature matrix is ​​generated for the 4096-dimensional binary feature vectors of n samples; then LSH bucketing is performed: the MinHash signature matrix is ​​divided into b strips, each strip has r rows, and sample pairs with the same hash value in any strip are grouped into candidate duplicate pairs; Similarity verification is performed only on the candidate duplicate pairs, calculating the Jaccard similarity score. The Jaccard similarity verification method is as follows: ， Threshold judgment, similarity threshold If the Jaccard similarity is greater than 0.7, the candidate duplicate pair is considered a duplicate, and one of the samples is randomly retained.

6. The method for sorting and simplifying mixed-modal data according to claim 1, characterized in that, In the third step, for specific modal data that is text data, text data deduplication is performed, which is executed by the third deduplication pipeline: Text standardization involves preprocessing, including: case normalization, merging redundant whitespace, removing special characters, and trimming leading and trailing spaces. Multi-granularity N-gram feature extraction: For character-level N-grams, a sliding window with k=3 is used to generate 3-grams; for word-level N-grams, after word segmentation, word-level 3-grams are generated using k=3. The character set and word set are merged and integrated, that is: The final feature set is the union of these two sets; Jaccard similarity calculation, the calculation method is as follows: ； Layered filtering and optimization, employing a three-layer filtering mechanism: (a) Length pre-filtering: If the length difference between two texts exceeds 50%, they are assumed not to be duplicates and the comparison is skipped; (b) Sample-based comparison: Each new text is compared only with the S most recently retained texts; (c) Threshold judgment: similarity threshold τ=0.

8. If Jaccard similarity>τ, it is classified as nearly duplicate.

7. A sorting and simplification system for mixed-modal data, characterized in that, include An input unit is used to input a raw data file, which includes mixed multimodal data; A classification unit is used to perform content inspection and semantic parsing on the original data file, and to identify and classify the multimodal data therein into a subset of multiple specific modalities; A deduplication unit is used to send each subset into the corresponding deduplication pipeline, and each deduplication pipeline performs deduplication of specific modal data respectively. The output unit is used to deduplicatize the specific modal data of each of the deduplication pipelines and output simplified data.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images