Method for generating multi-modal large model pre-training interlaced data based on professional domain knowledge

Abstract

The invention discloses a method for generating multi-modal large model pre-training interlaced data based on professional domain knowledge. The method comprises the following steps: S1, cleaning original professional domain knowledge data; s2, performing duplicate removal operation on the cleaned professional domain knowledge data; s3, segmenting the professional domain knowledge data after duplicate removal according to comma; s4, extracting the feature of one piece of segmented data, and searching in an image library by using the obtained feature to search for an image feature closest to the feature; s5, judging whether the similarity between the characteristics of the data and the characteristics of the closest image is greater than an image-text similarity threshold value, if so, placing the image in front of the data, otherwise, skipping; and S6, repeating the steps S4 and S5 until all the segmented data are processed, and generating final professional field interlaced data. According to the scheme, a sufficient number of diversified interlaced data can be constructed, so that the understanding ability of the multi-modal large model is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer science and artificial intelligence, and in particular to a method for generating multimodal large model pre-training interleaved data based on professional domain knowledge. Background Art

[0002] Interleaved data of professional domain knowledge is crucial to improving the understanding ability of multimodal large models. However, the acquisition of interleaved data based on professional domain knowledge is currently facing challenges. Since professional domain knowledge data is usually plain text data and is often subject to confidentiality and privacy restrictions, the amount of interleaved data of professional domain knowledge is small and difficult to obtain. Summary of the invention

[0003] The present invention mainly solves the technical problems in the prior art that the amount of interleaved data of professional domain knowledge is small and is not conducive to the training of multimodal large models. It provides a method for generating interleaved data for pre-training of multimodal large models based on professional domain knowledge, which can automatically construct interleaved data to improve the comprehension ability of multimodal large models.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a method for generating multimodal large model pre-training interleaved data based on professional domain knowledge, comprising the following steps: S1. Clean the original professional knowledge data; S2. De-duplication operation is performed on the cleaned professional domain knowledge data; S3, splitting the professional field knowledge data after deduplication by commas; S4, extracting a feature of the segmented data, and using the obtained feature to search in the image library to find the image feature closest to the feature; S5, determining whether the similarity between the feature of the data and the feature of the closest image is greater than the image-text similarity threshold, if so, placing the image before the data, otherwise skipping; S6. Repeat steps S4 and S5 until all segmented data are processed to generate final professional field interleaved data.

[0005] The final generated professional field interleaved data is in the following form: < Figure 1 >Professional field knowledge 1<Figure 2>Professional field knowledge 2Professional field knowledge 3<Figure 4>Professional field knowledge 4…… The professional domain knowledge 3 is the situation where no image larger than the threshold is found.

[0006] Preferably, the step S1 is specifically as follows: S101. Preprocess the original professional domain knowledge data through text parsing and character encoding processing to ensure the integrity and readability of the data; S102. Using regular expression matching, mark the privacy-related content in the data as sensitive information, such as personal name, phone number, email address, WeChat ID, QQ ID, etc.; S103. Replace sensitive information with an empty string to protect user privacy.

[0007] The data cleaning module is designed to process sensitive information in professional field data through automated methods, protect user privacy and ensure the accuracy and consistency of data.

[0008] Preferably, the preprocessing in step S101 includes but is not limited to text parsing and character encoding processing, wherein text parsing specifically extracts useful information from the original professional domain knowledge data and structures it, because some professional data may exist in files such as PDF, which requires certain text parsing; Character encoding is mainly to convert text into a coding format that can be processed by computers, while ensuring consistency and losslessness during the processing. Specifically: Encoding standardization: Convert text data into a unified encoding format, such as UTF-8, to avoid garbled characters caused by different encoding formats; Remove noise characters: clean up irrelevant characters or noise characters that may exist in the text, such as HTML tags, special symbols, etc. Normalization: Process special characters and formats in text to normalize them. For example, convert full-width characters to half-width characters and convert uppercase letters to lowercase letters (as needed).

[0009] Preferably, the step S2 is specifically as follows: S201, segmenting the cleaned professional domain knowledge data according to line breaks to obtain several segments of text data, and then segmenting each segment of the segmented text data and extracting features to obtain several feature vectors; for example, the HanLP segmentation tool can be used for segmentation; and the text data feature extraction can be completed by using MinHash; S202, horizontally splicing all feature vectors to obtain text data features, that is, the feature vector extracted from a section of text data is a vector of combined text data features, and the dimension (number of rows) of the feature vector is the same as the number of rows of the text data features, and then obtaining a hash signature matrix according to the text data features and the random permutation table to complete the dimensionality reduction of the text data features; S203, splitting the hash signature matrix into several groups in the vertical direction, that is, each eigenvector is also split into several groups; S204, compare the feature vectors in the hash signature matrix in pairs. If at least one group of hash values ​​in the two feature vectors is the same, the text data corresponding to the two feature vectors are placed in the same hash bucket as candidate similar items; that is, as long as there is one group that is the same, they are placed in the same hash bucket, and no judgment is made later whether there are any other similarities; for example, the dimension of the feature vector is 20, which is evenly divided into 5 groups, each group contains 4 rows, that is, each feature vector is split into 5 groups of small vectors with 4 rows each, and then when comparing two feature vectors, the two groups of small vectors are compared. If any group of small vectors in the first feature vector is the same as any group of small vectors in the second feature vector, the text data corresponding to the two feature vectors will be placed in the same hash bucket; S205. In each hash bucket, calculate the similarity between any two text data. If the similarity is greater than the text similarity threshold, the two text data are determined to be duplicate data. Remove any one of the two duplicate text data from the cleaned professional domain knowledge data. Repeat this process until all combinations of two text data are compared.

[0010] The reasons for segmentation first and then word segmentation are as follows: 1. Structured processing Knowledge data in professional fields usually have certain structures and formats, such as paragraphs, lists, titles, etc. By segmenting with line breaks, the data can be initially separated into natural paragraphs or semantic units, making subsequent processing more orderly and structured.

[0011] 2. Improve word segmentation effect Direct word segmentation may cause some problems due to processing the entire block of text, such as inaccurate or missing word segmentation. After line break segmentation, the word segmenter can segment smaller and semantically clear text blocks, improving the accuracy and efficiency of word segmentation.

[0012] 3. Reduce noise and errors Knowledge data in professional fields may contain a large number of proper nouns and terms, and direct word segmentation is prone to errors or misclassification. By segmenting with line breaks, these data can be divided into more manageable small segments, reducing noise and errors in the word segmentation process.

[0013] Preferably, in step S203, the hash signature matrix is ​​divided into several groups in any one of the following three ways: (1) Fixed-length grouping: The hash signature matrix is ​​grouped according to fixed length, and each group contains a fixed number of rows. The advantage of this method is that it is simple and intuitive, but it may cause some segments to be too long or too short, affecting the accuracy of matching. (2) Division based on the ratio of the total number of rows in the hash signature matrix: Divide the total number of rows in the hash signature matrix by an integer, and the quotient is the number of rows in each group. This method is relatively flexible, but needs to be adjusted according to specific circumstances. (3) Dynamically determine the segments: Adaptively adjust the number and length of segments according to the changes in the features in the hash signature matrix.

[0014] Preferably, in the dynamically determined segmentation, feature change detection includes: Detect data distribution changes: Detect changes in hash values ​​of specific patterns in the hash signature matrix; some rows of the hash signature matrix may contain specific patterns or frequently occurring hash values. By detecting changes in these patterns, it can be determined whether segmentation is needed; Detect differences between rows: Detect the difference in hash values ​​between rows; if the difference between rows in a certain part is small, they can be grouped together. If the difference suddenly increases, segmentation is performed here; Statistical characteristics: Use statistical methods to detect data characteristics, such as mean, variance, standard deviation, etc. When these statistical characteristics change significantly, they are used as the basis for segmentation; Adaptive adjustment strategies include: Dynamic window: Using the sliding window method, the window size is dynamically adjusted when traversing the hash signature matrix, and the segmentation points are determined according to the changes in the features; for example, the initial window can be set to a small value, and then the window size is gradually adjusted according to the changes in the data; Threshold setting: Set a change threshold for a certain feature, and segment when the change in the feature value exceeds the threshold; for example, you can set a change threshold for the mean or variance of the hash value, and segment when it exceeds the threshold.

[0015] Clustering algorithm: Use a clustering algorithm (such as k-means or hierarchical clustering) to cluster the rows of the hash signature matrix, and determine the segmentation based on the clustering results, so that the rows in each group can be made more similar.

[0016] Preferably, in step S205, the cosine similarity is used to calculate the similarity between any two text data, and the text similarity threshold is 0.8.

[0017] Preferably, the image-text similarity threshold is 0.6.

[0018] Preferably, the image library is obtained by extracting features from a public image dataset (such as Laion) using an image feature extraction model (such as altclip) and then storing the features in a vector database.

[0019] The substantial effect of the present invention is that it is possible to utilize existing professional domain knowledge to construct a sufficient amount of diverse interleaved data for training of large multimodal models to improve understanding capabilities, without the need for human involvement, with a high degree of automation and resource saving. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a method for generating multimodal large model pre-training interleaved data based on professional domain knowledge of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0022] Embodiment: This embodiment provides a method for generating multimodal large model pre-training interleaved data based on professional domain knowledge, such as Figure 1 As shown, the following steps are included: S1. Use the data cleaning module to clean the original professional domain knowledge data; S2. Use the data deduplication module to perform deduplication operations on the cleaned professional domain knowledge data; S3, splitting the professional field knowledge data after deduplication by commas; S4. Extract a feature of the segmented data, and use the obtained feature to search in the image library to find the image feature closest to the feature; the feature extraction can use the Altering the Language Encoder inCLIP (AltCLIP) model; S5, determining whether the similarity between the feature of the data and the feature of the closest image is greater than the image-text similarity threshold, if so, placing the image before the data, otherwise skipping; S6. Repeat steps S4 and S5 until all segmented data are processed to generate final professional field interleaved data.

[0023] The final generated professional field interleaved data is in the following form: < Figure 1 >Professional field knowledge 1<Figure 2>Professional field knowledge 2Professional field knowledge 3<Figure 4>Professional field knowledge 4…… The professional domain knowledge 3 is the situation where no image larger than the threshold is found.

[0024] Step S1 is specifically as follows: S101. Preprocess the original professional domain knowledge data through text parsing and character encoding processing to ensure the integrity and readability of the data; S102. Using regular expression matching, mark the privacy-related content in the data as sensitive information, such as personal name, phone number, email address, WeChat ID, QQ ID, etc.; S103. Replace sensitive information with an empty string to protect user privacy.

[0025] The data cleaning module is designed to process sensitive information in professional field data through automated methods, protect user privacy and ensure the accuracy and consistency of data.

[0026] The preprocessing in step S101 includes but is not limited to text parsing and character encoding processing, wherein text parsing specifically extracts useful information from the original professional domain knowledge data and structures it, because some professional data may exist in files such as PDF, which requires certain text parsing; Character encoding is mainly to convert text into a coding format that can be processed by computers, while ensuring consistency and losslessness during the processing. Specifically: Encoding standardization: Convert text data into a unified encoding format, such as UTF-8, to avoid garbled characters caused by different encoding formats; Remove noise characters: clean up irrelevant characters or noise characters that may exist in the text, such as HTML tags, special symbols, etc. Normalization: Process special characters and formats in text to normalize them. For example, convert full-width characters to half-width characters and convert uppercase letters to lowercase letters (as needed).

[0027] Step S2 is specifically as follows: S201, segmenting the cleaned professional domain knowledge data according to line breaks to obtain several segments of text data, and then segmenting each segment of the segmented text data and extracting features to obtain several feature vectors; for example, the HanLP segmentation tool can be used for segmentation; and the text data feature extraction can be completed by using MinHash; S202, horizontally splicing all feature vectors to obtain text data features, that is, the feature vector extracted from a section of text data is a vector of combined text data features, and the dimension (number of rows) of the feature vector is the same as the number of rows of the text data features, and then obtaining a hash signature matrix according to the text data features and the random permutation table to complete the dimensionality reduction of the text data features; S203, splitting the hash signature matrix into several groups in the vertical direction, that is, each eigenvector is also split into several groups; S204, compare the feature vectors in the hash signature matrix in pairs. If at least one group of hash values ​​in the two feature vectors is the same, the text data corresponding to the two feature vectors are placed in the same hash bucket as candidate similar items; that is, as long as there is one group that is the same, they are placed in the same hash bucket, and no judgment is made later whether there are any other similarities; for example, the dimension of the feature vector is 20, which is evenly divided into 5 groups, each group contains 4 rows, that is, each feature vector is split into 5 groups of small vectors with 4 rows each, and then when comparing two feature vectors, the two groups of small vectors are compared. If any group of small vectors in the first feature vector is the same as any group of small vectors in the second feature vector, the text data corresponding to the two feature vectors will be placed in the same hash bucket; S205. In each hash bucket, calculate the similarity between any two text data. If the similarity is greater than the text similarity threshold, the two text data are determined to be duplicate data. Remove any one of the two duplicate text data from the cleaned professional domain knowledge data. Repeat this process until all combinations of two text data are compared.

[0028] The reasons for segmentation first and then word segmentation are as follows: 1. Structured processing Knowledge data in professional fields usually have certain structures and formats, such as paragraphs, lists, titles, etc. By segmenting with line breaks, the data can be initially separated into natural paragraphs or semantic units, making subsequent processing more orderly and structured.

[0029] 2. Improve word segmentation effect Direct word segmentation may cause some problems due to processing the entire block of text, such as inaccurate or missing word segmentation. After line break segmentation, the word segmenter can segment smaller and semantically clear text blocks, improving the accuracy and efficiency of word segmentation.

[0030] 3. Reduce noise and errors Knowledge data in professional fields may contain a large number of proper nouns and terms, and direct word segmentation is prone to errors or misclassification. By segmenting with line breaks, these data can be divided into more manageable small segments, reducing noise and errors in the word segmentation process.

[0031] In step S203, the hash signature matrix is ​​divided into several groups using any one of the following three methods: (1) Fixed-length grouping: The hash signature matrix is ​​grouped according to fixed length, and each group contains a fixed number of rows. The advantage of this method is that it is simple and intuitive, but it may cause some segments to be too long or too short, affecting the accuracy of matching. (2) Division based on the ratio of the total number of rows in the hash signature matrix: Divide the total number of rows in the hash signature matrix by an integer, and the quotient is the number of rows in each group. This method is relatively flexible, but needs to be adjusted according to specific circumstances. (3) Dynamically determine the segments: Adaptively adjust the number and length of segments according to the changes in the features in the hash signature matrix.

[0032] In the dynamic determination segmentation, feature change detection includes: Detect data distribution changes: Detect changes in hash values ​​of specific patterns in the hash signature matrix; some rows of the hash signature matrix may contain specific patterns or frequently occurring hash values. By detecting changes in these patterns, it can be determined whether segmentation is needed; Detect differences between rows: Detect the difference in hash values ​​between rows; if the difference between rows in a certain part is small, they can be grouped together. If the difference suddenly increases, segmentation is performed here; Statistical characteristics: Use statistical methods to detect data characteristics, such as mean, variance, standard deviation, etc. When these statistical characteristics change significantly, they are used as the basis for segmentation; Adaptive adjustment strategies include: Dynamic window: Using the sliding window method, the window size is dynamically adjusted when traversing the hash signature matrix, and the segmentation points are determined according to the changes in the features; for example, the initial window can be set to a small value, and then the window size is gradually adjusted according to the changes in the data; Threshold setting: Set a change threshold for a certain feature, and segment when the change in the feature value exceeds the threshold; for example, you can set a change threshold for the mean or variance of the hash value, and segment when it exceeds the threshold.

[0033] Clustering algorithm: Use a clustering algorithm (such as k-means or hierarchical clustering) to cluster the rows of the hash signature matrix, and determine the segmentation based on the clustering results, so that the rows in each group can be made more similar.

[0034] In step S205, the cosine similarity is used to calculate the similarity between any two text data, and the text similarity threshold is 0.8.

[0035] The image-text similarity threshold is 0.6.

[0036] The image library is obtained by extracting features from a public image dataset (such as Laion) using an image feature extraction model (such as altclip) and then storing the features in a vector database.

[0037] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0038] Although this article uses more terms such as data cleaning, deduplication, feature extraction, and similarity, it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A method for generating multimodal large model pre-training interleaved data based on professional domain knowledge, characterized in that: The following steps are involved: S1. Clean the original professional knowledge data; S2. De-duplication operation is performed on the cleaned professional domain knowledge data; S3, splitting the professional field knowledge data after deduplication by commas; S4, extracting a feature of the segmented data, and using the obtained feature to search in the image library to find the image feature closest to the feature; S5, determining whether the similarity between the feature of the data and the feature of the closest image is greater than the image-text similarity threshold, if so, placing the image before the data, otherwise skipping; S6. Repeat steps S4 and S5 until all segmented data are processed to generate final professional field interleaved data.

2. According to claim 1, a method for generating multimodal large model pre-training interleaved data based on professional domain knowledge is characterized in that: The step S1 is specifically as follows: S101, preprocessing the original professional domain knowledge data through text parsing and character encoding processing; S102, using a regular expression matching method to mark the privacy-related content in the data as sensitive information; S103. Replace sensitive information with an empty string.

3. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 2, characterized in that: The text parsing in step S101 specifically involves extracting useful information from the original professional domain knowledge data and structuring it; The character encoding is as follows: Coding standardization: convert text data into a unified coding format; Remove noise characters: clean up irrelevant characters or noise characters that may exist in the text; Normalization: Process special characters and formats in text to make it normalized.

4. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 1 or 2, characterized in that: The step S2 is specifically as follows: S201, segmenting the cleaned professional domain knowledge data according to line breaks to obtain a number of text data segments, and then segmenting each segment of the text data and extracting features to obtain a number of feature vectors; S202, horizontally concatenating all feature vectors to obtain text data features, and then obtaining a hash signature matrix according to the text data features and a random permutation table; S203, splitting the hash signature matrix into a plurality of groups in the vertical direction, that is, each eigenvector is also split into a plurality of groups; S204, comparing the feature vectors in the hash signature matrix in pairs, if at least one set of hash values ​​in two feature vectors is the same, then putting the text data corresponding to the two feature vectors into the same hash bucket as candidate similar items; S205. In each hash bucket, calculate the similarity between any two text data. If the similarity is greater than the text similarity threshold, the two text data are determined to be duplicate data. Remove any one of the two duplicate text data from the cleaned professional domain knowledge data. Repeat this process until all combinations of two text data are compared.

5. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 4 is characterized in that: In step S203, the hash signature matrix is ​​divided into several groups using any one of the following three methods: (1) Fixed-length grouping: The hash signature matrix is ​​grouped according to a fixed length, and each group contains a fixed number of rows; (2) Division based on the ratio of the total number of rows in the hash signature matrix: Divide the total number of rows in the hash signature matrix by an integer, and the quotient obtained is the number of rows contained in each group; (3) Dynamically determine the segments: Adaptively adjust the number and length of segments according to the changes in the features in the hash signature matrix.

6. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 5, characterized in that: In the dynamically determined segmentation, feature change detection includes: Detect data distribution changes: Detect changes in hash values ​​of specific patterns in the hash signature matrix; Detect differences between rows: Detect the difference in hash values ​​between rows; Statistical characteristics: Use statistical methods to detect the characteristics of data; Adaptive adjustment strategies include: Dynamic window: Use the sliding window method to dynamically adjust the window size when traversing the hash signature matrix and determine the segmentation point according to the feature changes; Threshold setting: set the change threshold of a certain feature, and segment it when the change of the feature value exceeds the threshold; Clustering algorithm: Use a clustering algorithm to cluster the rows of the hash signature matrix, and determine the segmentation based on the clustering results.

7. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 4, characterized in that: In step S205, the cosine similarity is used to calculate the similarity between any two text data, and the text similarity threshold is 0.

8.

8. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 1, characterized in that: The image-text similarity threshold is 0.

6.

9. The method for generating multimodal large model pre-training interleaved data based on professional domain knowledge according to claim 1, characterized in that: The image library is obtained by extracting features from a public image data set using an image feature extraction model and then storing the features in a vector database.

Images