A method and device for automatically generating Chinese image-text pair data based on LVLM

Through the automated Chinese graphic and text data generation method based on LVLM, the existing Chinese graphic and text data quality and small number of data sets are solved, and high-quality and diversified data set generation is realized, which improves the training and training of the model. Generation ability.

CN118570566BActive Publication Date: 2024-11-22ROCK AI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411052852.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-22
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The existing Chinese pictures and texts have low quality and small number of data sets, resulting in limited model training effects and generalization capabilities. The images and text topics in the data set are concentrated in specific fields and lack broadness.

Method used

Using an automated Chinese graphic and text data generation method based on LVLM (CLIP and large vision-language model), by obtaining images and generating instructions, using the LVLM model to generate text, and perform truncated sample filtering and repeatability filtering, and computed with the Chinese CLIP model. Graphic and text similarity, delete mismatched data, and form a high-quality and diverse data set.

Benefits of technology

It improves the quality and quantity of Chinese pictures and texts, enhances the diversity and representativeness of the data, and improves the training effect and generation ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570566B_ABST
    Figure CN118570566B_ABST
Patent Text Reader

Abstract

The present invention provides an automatic Chinese image-text data generation method and device based on LVLM, which solves the problem of low quality and small quantity of Chinese data in existing Chinese image-text data. It has the following advantages: (1) The same LVLM is used to generate Chinese data and translate English data, which simplifies the architecture of the solution; (2) By combining the generation and translation methods, the source of Chinese image-text data is expanded and the diversity of data is increased; (3) The generated / translated samples are post-processed in multiple stages to further improve the quality of Chinese corpus; (4) The CLIP model is used to compare the similarity of images and texts, which further improves the quality of Chinese corpus.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a method and device for automatically generating Chinese image-text pair data based on LVLM. Background Art

[0002] In the field of artificial intelligence, multimodal data fusion technology, especially the combination of images and text, has become a research hotspot. This type of data not only enriches the input dimension of machine learning models, but also promotes the rapid development of deep learning in application fields such as visual question answering, image description generation, and visual positioning. Among them, the Chinese image-text pair dataset, as an important bridge connecting vision and semantics, is crucial to promoting multimodal understanding and generation research in the Chinese context.

[0003] Although many works have been devoted to creating Chinese image-text pair datasets in recent years, these datasets often have the following limitations: (1) Compared with the English field, high-quality and large-scale Chinese image-text pair datasets are relatively scarce, which limits the training effect and generalization ability of the model; (2) In the existing datasets, the matching degree of some image-text pairs is not high, and the text description may not match the image content, causing the model to learn incorrect associations; (3) The image and text topics in the dataset are often concentrated in certain specific fields, lacking breadth and representativeness, which affects the model's comprehensive understanding and generation capabilities.

[0004] In view of the above limitations, automated Chinese image-text pair data generation and cleaning technology came into being. This technology aims to automatically mine, match and clean Chinese image-text pair data from massive Internet resources through algorithms, in order to build a large-scale, high-quality, diverse and real-time updated dataset. Summary of the invention

[0005] The present invention provides an automatic Chinese image-text pair data generation method and device based on LVLM, so as to solve the problem that the Chinese data in the existing Chinese image-text pair data is of low quality and small quantity.

[0006] In the first aspect, the present invention provides a method and device for automatically generating Chinese image-text pair data based on LVLM (CLIP and Large Vision-Language Model), which is applicable to the case where only images are included, and specifically includes the following steps:

[0007] Step S1, acquiring image D1 and generating instruction prompt-1;

[0008] Step S2, input D1 and prompt-1 into the LVLM model to form text T1;

[0009] Step S3, filtering the truncated samples of T1 to form text T2;

[0010] Step S4. Perform repetitive filtering on T2 to form text T3;

[0011] Step S5. Calculate the similarity S-1 between D1 and T3 through the Chinese CLIP model. If S-1 is less than the threshold γ, delete D1; otherwise, retain D1.

[0012] Preferably, in step S1, the prompt-1 is used to guide the LVLM model to generate a specific type of response or perform a specific task.

[0013] Preferably, in step S3, perform truncated sample filtering on T1 to form T2, which specifically includes the following steps:

[0014] Step S301. Judge the character length L of T1 T1 and the size of the character threshold max_new_tokens. If L T1 is less than max_new_tokens, set T2 as T1 and execute step S4; otherwise, truncate the characters in T1 that exceed max_new_tokens to form T1-truncate;

[0015] Step S302. Judge whether the last character character-1 of T1-truncate is a Chinese full stop. If character-1 is a Chinese full stop, set T2 as T1-truncate; otherwise, delete the content after the last Chinese full stop in T1-truncate to form T2.

[0016] Preferably, in step S4, perform repetitive filtering on T2 to form T3, which specifically includes the following steps:

[0017] Step S401. Divide T2 into L sentences to form text T3;

[0018] Step S402. Perform deduplication processing on T3 to obtain L' sentences after deduplication processing;

[0019] Step S403. If L'<L, delete D1; otherwise, execute step S5.

[0020] Preferably, in step S402, perform the deduplication processing by performing set operations on the L sentences.

[0021] Preferably, when the computing resources are insufficient, remove step S5, directly retain the image D1 and output the text T3.

[0022] In a second aspect, the present invention provides an automated Chinese image-text pair data generation method based on LVLM, which is applicable to the situation including an image and an English description of the image, and specifically includes the following steps:

[0023] Step A1: Obtain image D2 and its English description D2-description;

[0024] Step A2: Calculate the similarity S-2 between D2 and D2-description through the Chinese CLIP model. If S-2 is less than the threshold γ, delete D2; otherwise, retain D2.

[0025] Step A3: Generate instruction prompt-2 and input prompt-2 and D2-description into the LVLM model to translate D2-description into Chinese, forming text T4.

[0026] Step A4: Perform truncated sample filtering on T4 to form text T5.

[0027] Step A5: Perform duplicate filtering on T5 to form text T6.

[0028] Preferably, in step A4, performing truncated sample filtering on T4 to form text T5 specifically includes the following steps:

[0029] Step A401: Judge the character length L of T4 T4 and the size of the character threshold max_new_tokens. If L T4 is less than max_new_tokens, set T5 as T4 and execute step A5; otherwise, truncate the characters in T4 that exceed max_new_tokens to form T4-truncate.

[0030] Step A402: Judge whether the last character character-2 of T4-truncate is a Chinese full stop. If character-2 is a Chinese full stop, set T5 as T4-truncate; otherwise, delete the content after the last Chinese full stop in T4-truncate to form T5.

[0031] Preferably, in step A5, performing duplicate filtering on T5 to form text T6, the data processing method in step A5 specifically includes the following steps:

[0032] Step A501: Divide T5 into N sentences to form text T6.

[0033] Step A502: Perform deduplication processing on T6 to obtain N' sentences after deduplication processing.

[0034] Step A503: If N' < N, delete D2; otherwise, output text T6.

[0035] Preferably, in step A502, a set operation is performed on the N sentences to implement the deduplication process.

[0036] Preferably, when computing resources are insufficient, step A2 is deleted.

[0037] In a third aspect, the present invention provides an automatic Chinese image-text pair data generation device based on LVLM, which is applicable to the case where only images are included, and specifically includes the following modules:

[0038] The first initialization module is used to obtain the image D1 and generate the instruction prompt-1;

[0039] A first text generation module, connected to the first initialization module, for inputting D1 and prompt-1 into the LVLM model to form text T1;

[0040] A first truncated sample filtering module, connected to the first text generating module, for filtering the truncated samples of T1 to form text T2;

[0041] A first repetitive filtering module, connected to the first truncated sample filtering module, is used to perform repetitive filtering on T2 to form text T3;

[0042] The first sample processing module is connected to the first repeatability filtering module and is used to calculate the similarity S-1 between D1 and T3 through the Chinese CLIP model. If S-1 is less than a threshold value γ, D1 is deleted; otherwise, D1 is retained.

[0043] Preferably, in the first initialization module, the prompt-1 is used to guide the LVLM model to generate a specific type of response or perform a specific task.

[0044] Preferably, the first truncated sample filtering module specifically includes the following submodules:

[0045] The first truncation submodule is used to determine the character length L of T1 T1 With the size of the character threshold max_new_tokens, if L T1 If it is less than max_new_tokens, T2 is set to T1 and the first repetitive filtering module is executed. Otherwise, the characters in T1 exceeding max_new_tokens are truncated to form T1-truncate.

[0046] The second truncation sub-module, connected to the first truncation sub-module, is used to determine whether the last character character-1 of T1-truncate is a Chinese full stop. If character-1 is a Chinese full stop, then set T2 as T1-truncate; otherwise, delete the content after the last Chinese full stop in T1-truncate to form T2.

[0047] Preferably, the first repeatability filtering module specifically includes the following sub-modules:

[0048] The first repeatability sub-module is used to divide T2 into L sentences to form text T3;

[0049] The second repeatability sub-module, connected to the first repeatability sub-module, is used to perform deduplication processing on T3 to obtain L' deduplicated sentences;

[0050] The third repeatability sub-module, connected to the second repeatability sub-module, is used to delete D1 when L'<L; otherwise, execute the first sample processing module.

[0051] Preferably, in the connection of the first repeatability sub-module, the deduplication processing is implemented by performing set operations on L sentences.

[0052] Fourthly, an LVLM-based automated Chinese image-text pair data generation device, applicable to the situation including an image and an English description of the image, specifically includes the following modules:

[0053] The second initialization module is used to obtain image D2 and the English description D2-description of D2;

[0054] The second sample processing module, connected to the second initialization module, is used to calculate the similarity S-2 between D2 and D2-description through the Chinese CLIP model. If S-2 is less than the threshold γ, then delete D2; otherwise, retain D2;

[0055] The second text generation module, connected to the second sample processing module, is used to generate instruction prompt-2 and input prompt-2 and D2-description into the LVLM model, and translate D2-description into Chinese to form text T4;

[0056] The second truncated sample filtering module, connected to the second text generation module, is used to perform truncated sample filtering on T4 to form text T5;

[0057] The second repeatability filtering module, connected to the second truncated sample filtering module, is used to perform repeatability filtering on T5 to form text T6.

[0058] Preferably, the second truncated sample filtering module specifically includes the following sub-modules:

[0059] The third truncation sub-module is used to judge the character length L of T4 T4 and the size of the character threshold max_new_tokens. If L T4 is less than max_new_tokens, set T5 to T4 and execute the second duplicate filtering module. Otherwise, truncate the characters in T4 that exceed max_new_tokens to form T4-truncate;

[0060] The fourth truncation sub-module is connected to the third truncation sub-module and is used to judge whether the last character character-2 of T4-truncate is a Chinese full stop. If character-2 is a Chinese full stop, set T5 to T4-truncate. Otherwise, delete the content after the last Chinese full stop in T4-truncate to form T5.

[0061] Preferably, the second duplicate filtering module specifically includes the following sub-modules:

[0062] The fourth duplicate sub-module is used to divide T5 into N sentences to form text T6;

[0063] The fifth duplicate sub-module is connected to the fourth duplicate sub-module and is used to perform duplicate removal processing on T6 to obtain N' sentences after duplicate removal processing;

[0064] The sixth duplicate sub-module is connected to the fifth duplicate sub-module and is used to delete D2 when N'<N. Otherwise, output text T6.

[0065] Preferably, in the fifth duplicate sub-module, the duplicate removal processing is implemented by performing a set operation on N sentences.

[0066] On the other hand, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements a method for generating automated Chinese graphic-text pair data based on LVLM according to any one of the first aspect or the second aspect of the present application.

[0067] On the other hand, the present invention also provides an electronic device. The electronic device includes: a memory storing a computer program; a processor communicatively connected to the memory and executing a method for generating automated Chinese graphic-text pair data based on LVLM according to any one of the first aspect or the second aspect of the present application when calling the computer program.

[0068] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0069] The present invention provides an automatic Chinese image-text data generation method and device based on LVLM, which solves the problem of low quality and small quantity of Chinese data in existing Chinese image-text data. It has the following advantages: (1) The same LVLM is used to generate Chinese data and translate English data, which simplifies the architecture of the solution; (2) By combining the generation and translation methods, the source of Chinese image-text data is expanded and the diversity of data is increased; (3) The generated / translated samples are post-processed in multiple stages to further improve the quality of Chinese corpus; (4) The CLIP model is used to compare the similarity of images and texts, which further improves the quality of Chinese corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which constitute part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention. In the accompanying drawings:

[0071] Figure 1 It is a flow chart of a situation in an automatic Chinese image-text pair data generation method based on LVLM in a preferred embodiment of the present invention.

[0072] Figure 2 It is a flowchart of another situation in an automatic Chinese image-text pair data generation method based on LVLM in a preferred embodiment of the present invention.

[0073] Figure 3 It is a structural schematic diagram of a situation in an automatic Chinese image-text data generation device based on LVLM in a preferred embodiment of the present invention.

[0074] Figure 4 It is a structural schematic diagram of another situation in an automatic Chinese image-text data generation device based on LVLM in a preferred embodiment of the present invention.

[0075] Figure 5 It is a flow chart of an automatic Chinese image-text pair data generation method based on LVLM according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0076] The present invention provides a method and device for automatically generating Chinese text-image data based on LVLM. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0077] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. Example

[0078] like Figure 1 and Figure 5 (left) shows that the method for generating Chinese image-text pair data based on LVLM described in this embodiment is applicable to the case where only images are included, and specifically includes the following steps:

[0079] Step S1, obtain image D1 and generate instruction prompt-1, such as: please describe this picture.

[0080] Among them, the prompt-1 is used to guide the LVLM model to generate a specific type of response or perform a specific task.

[0081] Step S2: Input D1 and prompt-1 into the LVLM model to form text T1.

[0082] Step S3: truncate and filter samples of T1 to form text T2.

[0083] Optionally, step S3 specifically includes the following steps:

[0084] Step S301, determine the character length L of T1 T1 With the size of the character threshold max_new_tokens, if L T1 If it is less than max_new_tokens, T2 is set to T1 and step S4 is executed. Otherwise, the characters in T1 exceeding max_new_tokens are truncated to form T1-truncate.

[0085] Step S302, determine whether the last character character-1 of T1-truncate is a Chinese period. If character-1 is a Chinese period, set T2 to T1-truncate. Otherwise, delete the content after the last Chinese period in T1-truncate to form T2.

[0086] Step S4, perform repetitive filtering on T2 to form text T3.

[0087] Optionally, step S4 specifically includes the following steps:

[0088] Step S401: Divide T2 into L sentences to form text T3.

[0089] Step S402: Remove duplicates from T3 to obtain L' sentences after duplicate removal.

[0090] Optionally, the duplicate removal process is implemented by performing a set operation on the L sentences.

[0091] Step S403: If L' < L, delete D1; otherwise, execute step S5.

[0092] Step S5: Calculate the similarity S-1 between D1 and T3 using the Chinese CLIP model. If S-1 is less than the threshold γ, delete D1; otherwise, retain D1. Embodiment

[0093] As Figure 2 and Figure 5 (right) shows, an automated Chinese image-text pair data generation method based on LVLM according to this embodiment is applicable to the situation including an image and an English description of the image, and specifically includes the following steps:

[0094] Step S6: Obtain image D2 and its English description D2-description.

[0095] Step S7: Calculate the similarity S-2 between D2 and D2-description using the Chinese CLIP model. If S-2 is less than the threshold γ, delete D2; otherwise, retain D2.

[0096] Step S8: Generate an instruction prompt-2 (e.g., please translate into Chinese) and input prompt-2 and D2-description into the LVLM model to translate D2-description into Chinese to form text T4.

[0097] Step S9: Perform truncated sample filtering on T4 to form text T5.

[0098] Optionally, step S9 specifically includes the following steps:

[0099] Step S901: Judge the character length L of T4 T4 and the size of the character threshold max_new_tokens. If L T4If it is less than max_new_tokens, set T5 to T4 and execute step S10. Otherwise, truncate the characters in T4 that exceed max_new_tokens to form T4-truncate.

[0100] Step S902: Determine whether the last character character-2 of T4-truncate is a Chinese full stop. If character-2 is a Chinese full stop, set T5 to T4-truncate. Otherwise, delete the content after the last Chinese full stop in T4-truncate to form T5.

[0101] Step S10: Perform duplicate filtering on T5 to form text T6.

[0102] Optionally, step S10 specifically includes the following steps:

[0103] Step S1001: Divide T5 into N sentences to form text T6.

[0104] Step S1002: Perform deduplication processing on T6 to obtain N' deduplicated sentences.

[0105] Optionally, the deduplication processing is implemented by performing a set operation on N sentences.

[0106] Step S1003: If N'<N, delete D2. Otherwise, output text T6. Embodiment

[0107] As Figure 3 shown, an automated Chinese image-text pair data generation device based on LVLM described in this embodiment specifically includes the following modules:

[0108] The first initialization module is used to obtain image D1 and generate instruction prompt-1.

[0109] The first text generation module is connected to the first initialization module and is used to input D1 and prompt-1 into the LVLM model to form text T1.

[0110] The first truncated sample filtering module is connected to the first text generation module and is used to perform truncated sample filtering on T1 to form text T2.

[0111] Among them, the first truncated sample filtering module specifically includes the following sub-modules:

[0112] The first truncation sub-module is used to judge the character length L of T1 T1 and the size of the character threshold max_new_tokens. If L T1If it is less than max_new_tokens, set T2 to T1 and execute the first repetitive filtering module; otherwise, truncate the characters in T1 that exceed max_new_tokens to form T1-truncate.

[0113] A second truncation sub-module, connected to the first truncation sub-module, is used to determine whether the last character character-1 of T1-truncate is a Chinese full stop. If character-1 is a Chinese full stop, set T2 to T1-truncate; otherwise, delete the content after the last Chinese full stop in T1-truncate to form T2.

[0114] A first repetitive filtering module, connected to the first truncated sample filtering module, is used to perform repetitive filtering on T2 to form text T3.

[0115] Among them, the first repetitive filtering module specifically includes the following sub-modules:

[0116] A first repetitive sub-module, used to divide T2 into L sentences to form text T3.

[0117] A second repetitive sub-module, connected to the first repetitive sub-module, is used to perform deduplication processing on T3 to obtain L' deduplicated sentences.

[0118] Optionally, the deduplication processing is implemented by performing a set operation on L sentences.

[0119] A third repetitive sub-module, connected to the second repetitive sub-module, is used to delete D1 when L' < L; otherwise, execute the first sample processing module.

[0120] A first sample processing module, connected to the first repetitive filtering module, is used to calculate the similarity S-1 between D1 and T3 through a Chinese CLIP model. If S-1 is less than the threshold γ, delete D1; otherwise, retain D1. Embodiment

[0121] As Figure 4 shown, an automated Chinese image-text pair data generation device based on LVLM described in this embodiment, for the case of including an image and an English description of the image, specifically includes the following modules:

[0122] A second initialization module, used to obtain image D2 and the English description D2-description of D2.

[0123] The second sample processing module is connected to the second initialization module and is used to calculate the similarity S-2 between D2 and D2-description through the Chinese CLIP model. If S-2 is less than a threshold γ, D2 is deleted; otherwise, D2 is retained.

[0124] The second text generation module is connected to the second sample processing module, and is used to generate instruction prompt-2 and input prompt-2 and D2-description into the LVLM model, and translate D2-description into Chinese to form text T4.

[0125] The second truncated sample filtering module is connected to the second text generating module and is used to perform truncated sample filtering on T4 to form text T5.

[0126] The second truncated sample filtering module specifically includes the following submodules:

[0127] The third truncation submodule is used to determine the character length L of T4 T4 With the size of the character threshold max_new_tokens, if L T4 If it is less than max_new_tokens, T5 is set to T4 and the second repetitive filtering module is executed. Otherwise, the characters in T4 exceeding max_new_tokens are truncated to form T4-truncate.

[0128] The fourth truncation submodule is connected to the third truncation submodule and is used to determine whether the last character character-2 of T4-truncate is a Chinese period. If character-2 is a Chinese period, T5 is set to T4-truncate. Otherwise, the content after the last Chinese period in T4-truncate is deleted to form T5.

[0129] The second repetitive filtering module is connected to the second truncated sample filtering module and is used to perform repetitive filtering on T5 to form text T6.

[0130] Wherein, the second repetitive filtering module specifically includes the following submodules:

[0131] The fourth repetitive submodule is used to divide T5 into N sentences to form text T6.

[0132] The fifth repetition submodule is connected to the fourth repetition submodule and is used to perform a deduplication process on T6 to obtain N' sentences after the deduplication process.

[0133] Optionally, the deduplication process is implemented by performing a set operation on N sentences.

[0134] The sixth repeating sub-module, connected to the fifth repeating sub-module, is used to delete D2 when N'<N, and vice versa, output text T6.

[0135] The specific embodiments of the present invention have been described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, all equivalent transformations and modifications made without departing from the spirit and scope of the present invention should be covered within the scope of the present invention.

Claims

1. An automatic Chinese picture-text pair data generation method based on LVLM, characterized in that: Applicable to the case of only containing images, which specifically includes the following steps: Step S1: Obtain image D1 and generate instruction prompt-1; Step S2: Input D1 and prompt-1 into the LVLM model to form text T1; Step S3: Perform truncated sample filtering on T1 to form text T2; Step S4: Perform duplicate filtering on T2 to form text T3; Step S5: Calculate the similarity S-1 between D1 and T3 through the Chinese CLIP model. If S-1 is less than the threshold γ, delete D1; otherwise, retain D1. In step S3, performing truncated sample filtering on T1 to form T2 specifically includes the following steps: Step S301, determine the character length L of T1 T1 With the size of the character threshold max_new_tokens, if L T1 If it is less than max_new_tokens, T2 is set to T1 and step S4 is executed. Otherwise, the characters in T1 exceeding max_new_tokens are truncated to form T1-truncate. Step S302: Judge whether the last character character-1 of T1-truncate is a Chinese full stop. If character-1 is a Chinese full stop, set T2 as T1-truncate; otherwise, delete the content after the last Chinese full stop in T1-truncate to form T2; In step S4, performing duplicate filtering on T2 to form T3 specifically includes the following steps: Step S401: Divide T2 into L sentences to form text T3; Step S402: Perform deduplication processing on T3 to obtain L' deduplicated sentences; Step S403: If L'<L, delete D1; otherwise, execute step S5; Among them, in step S402, the deduplication processing is implemented by performing set operations on the L sentences.

2. An automatic Chinese picture-text pair data generation method based on LVLM, characterized in that: Applicable to the case of containing an image and an English description of the image, which specifically includes the following steps: Step A1: Obtain image D2 and the English description D2-description of D2; Step A2: Calculate the similarity S-2 between D2 and D2-description through the Chinese CLIP model. If S-2 is less than the threshold γ, delete D2; otherwise, retain D2; Step A3: Generate instruction prompt-2 and input prompt-2 and D2-description into the LVLM model to translate D2-description into Chinese to form text T4; Step A4: Perform truncated sample filtering on T4 to form text T5; Step A5: Perform duplicate filtering on T5 to form text T6. In step A4, performing truncated sample filtering on T4 to form text T5 specifically includes the following steps: Step A401: Determine the character length L of T4 T4 With the size of the character threshold max_new_tokens, if L T4 If it is less than max_new_tokens, T5 is set to T4 and step A5 is executed. Otherwise, the characters in T4 exceeding max_new_tokens are truncated to form T4-truncate. Step A402: Judge whether the last character character-2 of T4-truncate is a Chinese full stop. If character-2 is a Chinese full stop, set T5 as T4-truncate; otherwise, delete the content after the last Chinese full stop in T4-truncate to form T5; In step A5, performing duplicate filtering on T5 to form text T6 specifically includes the following steps: Step A501: Divide T5 into N sentences to form text T6; Step A502: Perform deduplication processing on T6 to obtain N' deduplicated sentences; Step A503: If N' < N, then delete D2; otherwise, output text T6. Among them, in step A502, the deduplication process is implemented by performing set operations on N sentences.

3. An automatic Chinese picture-text pair data generation device based on LVLM, characterized in that: Applicable to the case of only containing images, specifically including the following modules: The first initialization module is used to obtain image D1 and generate instruction prompt-1. The first text generation module, connected to the first initialization module, is used to input D1 and prompt-1 into the LVLM model to form text T1. The first truncated sample filtering module, connected to the first text generation module, is used to perform truncated sample filtering on T1 to form text T2. The first duplicate filtering module, connected to the first truncated sample filtering module, is used to perform duplicate filtering on T2 to form text T3. The first sample processing module, connected to the first duplicate filtering module, is used to calculate the similarity S-1 between D1 and T3 through the Chinese CLIP model. If S-1 is less than the threshold γ, then delete D1; otherwise, retain D1. Among them, The first truncated sample filtering module specifically includes the following sub-modules: The first truncation submodule is used to determine the character length L of T1 T1 With the size of the character threshold max_new_tokens, if L T1 If it is less than max_new_tokens, T2 is set to T1 and the first repetitive filtering module is executed. Otherwise, the characters in T1 exceeding max_new_tokens are truncated to form T1-truncate. The second truncation sub-module, connected to the first truncation sub-module, is used to determine whether the last character character-1 of T1-truncate is a Chinese full stop. If character-1 is a Chinese full stop, then set T2 to T1-truncate; otherwise, delete the content after the last Chinese full stop in T1-truncate to form T2. The first duplicate filtering module specifically includes the following sub-modules: The first duplication sub-module is used to divide T2 into L sentences to form text T3. The second duplication sub-module, connected to the first duplication sub-module, is used to perform deduplication processing on T3 to obtain L' deduplicated sentences. The third duplication sub-module, connected to the second duplication sub-module, is used to delete D1 when L' < L; otherwise, execute the first sample processing module.

4. An automatic Chinese picture-text pair data generation device based on LVLM, characterized in that: Applicable to the case of containing an image and an English description of the image, specifically including the following modules: The second initialization module is used to obtain image D2 and the English description D2-description of D2. The second sample processing module, connected to the second initialization module, is used to calculate the similarity S-2 between D2 and D2-description through the Chinese CLIP model. If S-2 is less than the threshold γ, then delete D2; otherwise, retain D2. The second text generation module, connected to the second sample processing module, is used to generate instruction prompt-2 and input prompt-2 and D2-description into the LVLM model to translate D2-description into Chinese to form text T4. The second truncated sample filtering module, connected to the second text generation module, is used to perform truncated sample filtering on T4 to form text T5. The second duplicate filtering module, connected to the second truncated sample filtering module, is used to perform duplicate filtering on T5 to form text T6. Among them, The second truncated sample filtering module specifically includes the following sub-modules: The third truncation submodule is used to determine the character length L of T4 T4 With the size of the character threshold max_new_tokens, if L T4 If it is less than max_new_tokens, T5 is set to T4 and the second repetitive filtering module is executed. Otherwise, the characters in T4 exceeding max_new_tokens are truncated to form T4-truncate. The fourth truncation sub-module, connected to the third truncation sub-module, is used to determine whether the last character character-2 of T4-truncate is a Chinese full stop. If character-2 is a Chinese full stop, set T5 to T4-truncate; otherwise, delete the content after the last Chinese full stop in T4-truncate to form T5. The second duplicate filtering module specifically includes the following sub-modules: The fourth duplicate sub-module is used to divide T5 into N sentences to form text T6. The fifth duplicate sub-module, connected to the fourth duplicate sub-module, is used to perform duplicate removal processing on T6 to obtain N' sentences after duplicate removal processing. The sixth duplicate sub-module, connected to the fifth duplicate sub-module, is used to delete D2 when N'<N; otherwise, output text T6.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1-2.

6. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in any one of claims 1-2.