Intelligent manufacturing-oriented large model pre-training method and system
By performing cluster analysis and singular value decomposition on text corpora in the field of intelligent manufacturing, a suitable context window size is determined, which solves the problem of model illusion in the pre-training of large language models, achieves a balance between the complexity and accuracy of large model reasoning, and improves the accuracy of assisted decision-making.
Patent Information
- Application Number
- CN202511247189.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-02
- Publication Date
- 2025-11-04
AI Technical Summary
In the pre-training of large language models for the field of intelligent manufacturing, the fixed context window in existing technologies leads to model illusion, which cannot balance the complexity of large model inference and the accuracy of pre-training.
By performing cluster analysis on the vocabulary set of each paragraph in the text corpus, semantic ambiguity, low-frequency consistency, and high-frequency redundancy interference are obtained. Singular value decomposition is used to determine the appropriate context window size, and a training dataset is constructed for large-scale model pre-training.
It effectively solves the model illusion problem, balances the complexity of large model reasoning with pre-training accuracy, and improves the accuracy of large language models in assisting decision-making in the field of intelligent manufacturing.
Smart Images
Figure CN120892818A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of large language model text processing, in particular to a large model pre-training method and system for intelligent manufacturing. BACKGROUND
[0002] At present, intelligent manufacturing technology is the key driving force for industrial development, and intelligent manufacturing can achieve more efficient and accurate decision-making with the help of large language models, thereby continuously evolving the transformation and upgrading of the intelligent manufacturing industry. However, in the pre-training of large language models for the intelligent manufacturing field, due to the limitations of the context window during the pre-training of large language models, the phenomenon of model hallucination is prone to occur in large language models, which can lead to errors, lack of evidence support, or logical contradictions in the intelligent manufacturing-related knowledge generated by large language models, thereby affecting the accuracy of the actual application of large language models in the intelligent manufacturing field, such as decision-making assistance, report generation, and intelligent question answering.
[0003] In the pre-training of large language models for intelligent manufacturing, most existing technologies use fixed context window text sequences to pre-train large language models. However, a larger context window can easily lead to the consideration of more redundant information during large model training, which can increase the complexity of large model reasoning, while a smaller context window can easily lead to the inability of large models to fully capture the semantic relationships between context words, thereby reducing the accuracy of large language model pre-training. Therefore, most existing technologies use fixed context window text sequences to pre-train large language models, which cannot balance the complexity of large model reasoning and the accuracy of large language model pre-training, and cannot effectively solve the problem of model hallucination in large language models after pre-training. SUMMARY
[0004] To solve the above technical problems, the purpose of the present application is to provide a large model pre-training method and system for intelligent manufacturing, and the technical solutions adopted are as follows: The present application provides a large model pre-training method for intelligent manufacturing, comprising the following steps: Obtain a text corpus, and perform word segmentation processing on each text corpus in the text corpus to obtain a vocabulary dataset for each text corpus; Cluster the semantic relationships between the words in the paragraph vocabulary set of each paragraph in each text corpus to extract each cluster of the paragraph vocabulary set of each paragraph, obtain the semantic ambiguity of each cluster according to the fluctuation and average level of the semantic relationships between the words in the cluster, and obtain the low-frequency consistency degree of each cluster by the change of the occurrence frequency of each word in the cluster in the paragraph vocabulary set, and then obtain the high-frequency redundant interference degree of each cluster in combination with the semantic ambiguity; singular value decomposition is performed on the high-frequency redundant interference degrees of all the cluster clusters corresponding to each paragraph in each text corpus, and the lexical suspiciousness of each text corpus is obtained by using the singular values obtained after the decomposition and the average level of all the high-frequency easy interference degrees corresponding to each paragraph of each text corpus, and then the window adaptation value of each text corpus is obtained by combining the similarity between the paragraph vocabulary sets of different paragraphs in each text corpus. The training data set of the LLM large model is obtained by the window adaptation value of each text corpus, so as to pre-train the LLM large model.
[0005] Preferably, the cluster clusters of the paragraph vocabulary set of each paragraph are further included: The paragraph vocabulary set of each paragraph of each text corpus is counted from the lexical data set of each text corpus, the normalized Google distance between any two words in the paragraph vocabulary set is calculated, and the cluster clusters are obtained by clustering the paragraph vocabulary set by using a clustering algorithm, wherein the normalized Google distance between any two words is used as the metric distance of the clustering algorithm.
[0006] Preferably, the method for obtaining the semantic ambiguity of each cluster cluster is that the ratio of the dispersion degree of the normalized Google distance between any two different words in each cluster cluster to the average of all the normalized Google distances is used as the semantic ambiguity of each cluster cluster.
[0007] Preferably, the method for obtaining the low-frequency consistency of each cluster cluster is: The frequency of each word in each cluster cluster appearing in the paragraph vocabulary set is counted, the frequency distribution vector of each cluster cluster is obtained by arranging all the frequencies of the words in each cluster cluster in ascending order, the average of all the elements in the frequency distribution vector of each cluster cluster and the average of all the elements in the first difference vector of the frequency distribution vector are counted respectively, and the reciprocal of the sum of the two averages is used as the low-frequency consistency of each cluster cluster.
[0008] Preferably, the method for obtaining the high-frequency redundant interference degree of each cluster cluster is: ; wherein, is the high-frequency redundant interference degree of the jth cluster cluster, is the semantic ambiguity of the jth cluster cluster, is the low-frequency consistency of the jth cluster cluster, is a constant to avoid zero denominator.
[0009] Preferably, the method for obtaining the lexical suspiciousness of each text corpus is: ; ; wherein, a lexical suspiciousness of the ith text corpus, a number of row vectors in a redundancy distribution matrix of the ith text corpus, a singularity weight of the ith row vector in the redundancy distribution matrix of the ith text corpus, a mean value of the ith row vector in the redundancy distribution matrix of the ith text corpus.
[0010] Preferably, the high-frequency redundancy interference degrees of all the cluster groups in each paragraph in each text corpus are arranged from small to large to form a redundancy interference vector of each paragraph, and the redundancy interference vectors of all the paragraphs are arranged according to the paragraph order to obtain the redundancy distribution matrix of each text corpus. The redundancy distribution matrix is subjected to singular value decomposition to extract the singular values of the row vectors, and the normalized results of the singular values of the row vectors are taken as the singularity weights of the row vectors in the redundancy distribution matrix.
[0011] Preferably, the method for obtaining the window adaptation value of each text corpus is as follows: wherein, a window adaptation value of the ith text corpus, is a floor function, is a preset initial value of the context window size, is an exponential function with a natural constant as the base number, is a maximum value of the preset context window size, a lexical suspiciousness of the ith text corpus, a paragraph similarity of the ith text corpus, wherein the mean value of the Jaccard similarity coefficient between the paragraph lexical sets of any two paragraphs in the ith text corpus is the paragraph similarity of the ith text corpus.
[0012] Preferably, the method for obtaining the training data set of the LLM large model pre-training is as follows: taking the window adaptation value of each text corpus as the size of the context window in each text corpus, performing word segmentation processing on the text vectors of all the context windows in each text corpus in the text corpus library, and taking the text vectors of all the context windows after the word segmentation processing as the training data set.
[0013] The embodiment of the present application also provides a large model pre-training system for intelligent manufacturing, which comprises a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the large model pre-training method for intelligent manufacturing according to any one of the above embodiments when executing the computer program.
[0014] As can be seen from the above, the large model pre-training method and system for intelligent manufacturing provided by the present application have at least the following beneficial effects: The present application considers that the prior art mostly adopts a fixed context window of a text sequence for pre-training of a large language model, cannot consider the complexity of large model inference and the accuracy of large language model pre-training, and cannot effectively solve the problem of model hallucination of the large language model after pre-training. Therefore, the present application performs clustering analysis on the paragraph vocabulary set of each paragraph in the text corpus, and accurately measures the interference degree of the semantics between different vocabularies in the clustering cluster according to the semantic ambiguity and low-frequency consistency of all vocabularies in each clustering cluster, which is used to avoid the problem of increasing the complexity of large language model inference when setting a large context window for pre-training of the large language model in the subsequent; Further, the present application constructs a redundancy distribution matrix through the high-frequency redundancy interference degree, and accurately measures the vocabulary suspiciousness of the text corpus by using the singular value decomposition method, which is used to set a more suitable context window for the text corpus in the subsequent; at the same time, the present application more accurately determines the context window size suitable for each text corpus by combining the paragraph closeness of each text corpus with the vocabulary suspiciousness of each text corpus, thereby avoiding the problem of setting a context window that is too large or too small, so that the complexity of large model inference and the accuracy of large language model pre-training can be considered at the same time in the subsequent, thereby solving the problem of model hallucination of the large language model after pre-training. BRIEF DESCRIPTION OF DRAWINGS
[0015] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.
[0016] Figure 1 A step flow chart of a large model pre-training method for intelligent manufacturing provided by the present application. DETAILED DESCRIPTION
[0017] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following describes the specific implementation, structure, features and effects of a large model pre-training method and system according to the present application in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.
[0018] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.
[0019] The specific scheme of the large model pre-training method and system for intelligent manufacturing provided by the present application will be specifically described below in combination with the drawings.
[0020] Please refer to Figure 1 which shows the step flow chart of the large model pre-training method for intelligent manufacturing provided by an embodiment of the present application, including the following steps: Step 1: Obtain the text corpus, and perform word segmentation processing on each text corpus in the text corpus to obtain the vocabulary dataset of each text corpus.
[0021] Before the large model pre-training for intelligent manufacturing, the text corpus in the field of intelligent manufacturing needs to be extracted. In the present embodiment, the text corpus in the field of intelligent manufacturing is obtained on the computer internet. Each text corpus in the text corpus is a selected text material in the field of intelligent manufacturing from academic papers, journals or blogs, which is used for subsequent pre-training of the large model.
[0022] Preprocess the text corpus. Preferably, in the present embodiment, the punctuation marks and meaningless characters in each text corpus in the text corpus are removed by using a regular matching algorithm to obtain the long text data of each text corpus. Further, by installing the Jieba word segmentation tool, the long text data of each text corpus is segmented by using the Jieba word segmentation technology, and the set composed of the segmentation results of the long text data is denoted as the vocabulary dataset of each text corpus in the field of intelligent manufacturing. The regular matching algorithm and the Jieba word segmentation technology are both known technologies, and the specific process will not be described in detail.
[0023] Second step: clustering the semantic relationship between the words in the paragraph vocabulary set of each paragraph in each text corpus to extract the clustering clusters of the paragraph vocabulary set of each paragraph, obtaining the semantic ambiguity of each clustering cluster according to the fluctuation and average level of the semantic relationship between the words in the clustering cluster, obtaining the low-frequency consistency of each clustering cluster through the change of the occurrence frequency of each word in the paragraph vocabulary set in the paragraph, and then obtaining the high-frequency redundant interference degree of each clustering cluster in combination with the semantic ambiguity.
[0024] Since there is a certain degree of difference in the semantic association between context words in different text corpora, different text corpora are suitable for pre-training of large models using text sequences of different context window sizes. For example, the semantic association between context words in a certain text corpus is low and the vocabulary data set contains a lot of redundant information. At this time, a smaller context window is suitable, thereby reducing the influence of redundant information on training accuracy during large model training. Conversely, a larger context window is suitable, thereby enabling the large model to fully capture the semantic relationship between context words during training.
[0025] Therefore, in the present embodiment, the i-th text corpus in the intelligent manufacturing field text corpus library is taken as an example, the paragraph vocabulary set of each paragraph is extracted from the vocabulary data set of the i-th text corpus, and the normalized Google distance between any two different words in the paragraph vocabulary set is calculated. All words in the paragraph vocabulary set are input to the k-means clustering algorithm, the normalized Google distance between two words is used as the measurement distance of the k-means clustering algorithm, and the number of clustering clusters is determined by the elbow rule. All clustering clusters of the paragraph vocabulary set are obtained by using the k-means clustering algorithm. Normalized Google distance and k-means clustering algorithm are both known technologies, and the specific process will not be repeated.
[0026] Further, the normalized Google distance between any two different words in each clustering cluster of the paragraph vocabulary set is calculated, the dispersion degree of all normalized Google distances is taken as the numerator, the mean of all normalized Google distances is taken as the denominator, and the ratio of the numerator to the denominator is taken as the semantic ambiguity of each clustering cluster. It should be noted that in the process of ratio calculation, in order to avoid the case that the denominator is 0, a constant is added to the denominator to avoid the denominator being zero. The value range of the constant can be (0.01, 0.05), and the specific value of the constant for avoiding the denominator being zero in the present embodiment is 0.02.
[0027] The greater the semantic ambiguity, the more ambiguous the semantic relationship between different words in the cluster is represented, and the more difficult it is for the large language model to fully capture the semantic relationship between different words in the cluster during pre-training. It should be noted that the method of measuring the degree of dispersion can be variance, standard deviation or coefficient of variation. In this embodiment, the standard deviation is used to measure the degree of dispersion.
[0028] At the same time, the frequency of each word in each cluster appearing in the paragraph word set is counted, and the frequencies corresponding to all words in each cluster are arranged in ascending order to obtain the frequency distribution vector of each cluster. In this embodiment, if the frequencies corresponding to different words are equal, a random arrangement is preferred. It can be understood that the more consistent the frequency of all words in the frequency distribution vector changes, and the lower the frequency of all words appears, the smaller the feature representing the information redundancy of all words in the cluster is, and the less the semantic relationship between different words in the cluster is affected by information redundancy.
[0029] Therefore, the mean of all elements in the first-order difference vector of the frequency distribution vector of each cluster is counted, which is preferably denoted as the first mean in this embodiment, representing the consistency of the frequency change of all words in the frequency distribution vector, and the mean of all elements in the frequency distribution vector of each cluster is counted, which is preferably denoted as the second mean in this embodiment, representing the average frequency of all words in the frequency distribution vector. Further, the reciprocal of the sum of the first mean and the second mean is taken as the low-frequency consistency of each cluster. The smaller the low-frequency consistency, the higher the amount of redundant information existing in all words in the cluster, and the easier it is to interfere with the semantic relationship between different words in the paragraph.
[0030] Through the above analysis of the semantic ambiguity and low-frequency consistency of different clusters, the high-frequency redundant interference degree of the jth cluster is calculated as follows: In the formula, is the high-frequency redundant interference degree of the jth cluster, is the semantic ambiguity of the jth cluster, is the low-frequency consistency of the jth cluster, is to avoid the denominator being a constant of zero, and its value range is (0.01, 0.05), and the value is 0.02 in this embodiment.
[0031] The high-frequency redundant interference degree reflects the degree of interference of the semantic relationship between different words in the cluster by high-frequency redundant information. The greater the high-frequency redundant interference degree, the greater the interference of the semantic relationship between different words in the cluster by high-frequency redundant information. At this time, setting a larger context window for pre-training of the large language model will increase the complexity of the inference of the large language model, thereby easily causing the large language model to appear model hallucination after pre-training.
[0032] Third step: singular value decomposition is performed on the high-frequency redundant interference degree of all cluster clusters corresponding to each paragraph in each text corpus, and the singular values obtained after decomposition and the average level of all high-frequency easy interference degrees corresponding to each paragraph in each text corpus are used to obtain the lexical suspiciousness of each text corpus. The similarity between the paragraph word sets of different paragraphs in each text corpus is combined to further obtain the window adaptation value of each text corpus.
[0033] Further, for each paragraph of each text corpus, the high-frequency redundant interference degrees of all cluster clusters corresponding to each paragraph in the text corpus are arranged in order from small to large to obtain the redundant interference vector of each paragraph in the text corpus. If the high-frequency redundant interference degrees of different cluster clusters are equal, they are randomly arranged in this embodiment. The redundant interference vectors of all paragraphs in the text corpus are arranged in paragraph order to obtain the redundancy distribution matrix of each text corpus. If the lengths of the redundant interference vectors of different paragraphs are not equal, the mean filling method is used to fill the redundant interference vectors to make the lengths of the redundant interference vectors of all paragraphs equal after filling.
[0034] Wherein, each row vector in the redundancy distribution matrix reflects the redundancy change characteristics of each paragraph in the text corpus. In order to more accurately measure the suspiciousness degree of all words in the entire text material, the redundancy distribution matrix of each text corpus is used as the SVD singular value decomposition algorithm (Singular Value Decomposition). The singular values of each row vector in the redundancy distribution matrix are obtained by the SVD singular value decomposition algorithm, and the singular values of all row vectors in the redundancy distribution matrix are normalized by the normalization exponential function. The normalization result of the singular values of each row vector is used as the singularity weight of each row vector in the redundancy distribution matrix. The greater the singularity weight is, the higher the credibility of the corresponding paragraph affected by the redundant information interference is. The SVD singular value decomposition algorithm is a known technology, and the specific process will not be repeated.
[0035] Based on the above analysis, the lexical suspiciousness of each text corpus is calculated: ; In the formula, is the lexical suspiciousness of the i-th text corpus, is the number of row vectors in the redundancy distribution matrix of the i-th text corpus, is the singularity weight of the i-th row vector in the redundancy distribution matrix of the i-th text corpus, is the mean value of the i-th row vector in the redundancy distribution matrix of the i-th text corpus.
[0036] The vocabulary suspiciousness reflects the suspiciousness degree of the vocabulary in all paragraphs in the text corpus. The greater the vocabulary suspiciousness, the lower the credibility of all the vocabulary in the text corpus, and the less suitable it is to select a larger context window. Otherwise, more suspicious vocabulary information is likely to be introduced, resulting in higher semantic ambiguity during training of the large language model, thereby causing the model hallucination of the trained large language model to be too high. Therefore, the greater the vocabulary suspiciousness, the more suitable it is to select a smaller context window, thereby fully and accurately capturing the semantic relationship between context words in the local area.
[0037] In order to pre-train the large model by using text sequences with different context window sizes for different text corpora, thereby taking into account the complexity of large model inference and the accuracy of large language model pre-training, the context window size suitable for each text corpus is more accurately determined by the vocabulary suspiciousness of each text corpus and the paragraph tightness of each text corpus. Therefore, the Jaccard similarity coefficient between the paragraph vocabulary sets of any two paragraphs in each text corpus is calculated, and the average of all Jaccard similarity coefficients is used as the paragraph similarity of each text corpus. The smaller the paragraph similarity, the lower the vocabulary similarity between different paragraphs in the text corpus, and the worse the tightness of the context connection in the text corpus, and the more suitable it is to select a smaller context window, thereby fully capturing the semantic relationship between context words in the local area.
[0038] Based on the above analysis, the window adaptation value of each text corpus in the text corpus library in the field of intelligent manufacturing is calculated: ; In the formula, is the window adaptation value of the i-th text corpus, is a floor function, is a preset initial value of the context window size, and in the present embodiment, the initial value is 512 tokens, is an exponential function with a natural constant as the base number, is the paragraph similarity of the i-th text corpus, is the maximum value of the preset context window size, and in the present embodiment, the maximum value is 2048 tokens.
[0039] The vocabulary suspiciousness and the paragraph tightness are used to more accurately determine the context window size suitable for each text corpus, thereby avoiding the problem of setting the context window too large or too small, so that the complexity of large model inference and the accuracy of large language model pre-training can be taken into account at the same time, thereby solving the problem of model hallucination of the pre-trained large language model.
[0040] The fourth step is to obtain a training data set of the LLM large model pre-training through the window adaptation value of each text corpus, so as to pre-train the LLM large model.
[0041] In order to more effectively pre-train the large language model, the window adaptation value of each text corpus is used as the size of the context window in each text corpus, and the text vectors of all context windows in each text corpus in the text corpus library are segmented, and the text vectors of all context windows after segmentation are used as the training data set.
[0042] Further, the training data set is used as the input of the LLM large model pre-training, the decoder of the Transformer model is used for the LLM large model pre-training, the Adam optimizer is used for the optimizer in the training process, the stochastic gradient descent (SGD) algorithm is used for gradient update, and the AR autoregressive model is used for unsupervised pre-training. After the training is completed, the pre-trained large language model is obtained.
[0043] Finally, the training data and validation data of a specific task are extracted from the text corpus in the field of intelligent manufacturing, the training data and validation data are segmented by using the jieba segmentation technology, the segmented training data and validation data are input into the pre-trained large language model, the large language model is fine-tuned, the stochastic gradient descent (SGD) algorithm is used for gradient update in the fine-tuning process, and the fine-tuned large language model is obtained after the fine-tuning is completed.
[0044] Among them, the segmentation processing, the LLM large model pre-training and the large language model fine-tuning are all known technologies, and the specific process will not be described again.
[0045] Based on the same inventive concept as the above method, the embodiments of the present application also provide a large model pre-training system for intelligent manufacturing, which comprises a memory, a processor and a computer program stored in the memory and running on the processor. The processor implements the steps of the large model pre-training method for intelligent manufacturing according to any one of the above embodiments when executing the computer program.
[0046] It can be understood that the above-mentioned embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. The above describes a specific embodiment of the present application. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous.
[0047] Each embodiment in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments.
[0048] The above merely provides the implementation of the present application, and is not intended to limit the scope of the present application. Any equivalent structure or equivalent flow transformation based on the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the protection scope of the present application.
Claims
1. A method for pre-training large models for intelligent manufacturing, characterized in that, Includes the following steps: Obtain a text corpus and perform word segmentation on each text in the corpus to obtain a vocabulary dataset for each text. For each paragraph in each text corpus, the semantic relationships between words in the paragraph vocabulary set are clustered to extract each paragraph's paragraph vocabulary set. The semantic ambiguity of each cluster is obtained based on the fluctuation and average level of the semantic relationships between words in the cluster. The low-frequency consistency of each cluster is obtained by the change in the frequency of each word in the paragraph vocabulary set of the paragraph. Then, the high-frequency redundancy interference of each cluster is obtained by combining the semantic ambiguity. Singular value decomposition is performed on the high-frequency redundancy interference degree of all clusters corresponding to each paragraph in each text corpus. By obtaining the singular values after decomposition and the average level of all high-frequency easy interference degree corresponding to each paragraph in each text corpus, the lexical suspiciousness of each text corpus is obtained. Combined with the similarity between the paragraph lexical sets of different paragraphs in each text corpus, the window fit value of each text corpus is obtained. The training dataset for pre-training the LLM large model is obtained by using the window fit value of each text corpus.
2. The large-scale model pre-training method for intelligent manufacturing as described in claim 1, characterized in that, The clusters for extracting the vocabulary set of each paragraph further include: From the vocabulary dataset of each text corpus, the paragraph vocabulary set of each paragraph of each text corpus is statistically determined. The normalized Google distance between any two words in the paragraph vocabulary set is calculated. The paragraph vocabulary set is then clustered using a clustering algorithm to obtain each cluster. The normalized Google distance between any two words is used as the distance metric of the clustering algorithm.
3. The large-scale model pre-training method for intelligent manufacturing as described in claim 2, characterized in that, The method for obtaining the semantic ambiguity of each cluster is as follows: the ratio of the dispersion of the normalized Google distance between any two different words in each cluster to the mean of all normalized Google distances is used as the semantic ambiguity of each cluster.
4. The large-scale model pre-training method for intelligent manufacturing as described in claim 1, characterized in that, The method for obtaining the low-frequency consistency of each cluster is as follows: The frequency of each word in each cluster within the paragraph vocabulary set is counted. The frequency distribution vector of each cluster is obtained by arranging the frequencies of all words in each cluster in ascending order. The mean of all elements in the frequency distribution vector of each cluster and the mean of all elements in the first difference vector of the frequency distribution vector are calculated respectively. The reciprocal of the sum of the two means is used as the low-frequency consistency of each cluster.
5. The large-scale model pre-training method for intelligent manufacturing as described in claim 1, characterized in that, The method for obtaining the high-frequency redundancy interference degree of each cluster is as follows: In the formula, Let be the high-frequency redundancy interference degree of the j-th cluster. Let be the semantic ambiguity of the j-th cluster. Let j be the low-frequency consistency of the j-th cluster. To avoid constants with a denominator of zero.
6. The large-scale model pre-training method for intelligent manufacturing as described in claim 1, characterized in that, The method for obtaining the lexical suspiciousness of each text corpus is as follows: ； In the formula, Let lexical suspiciousness be the value of the i-th text corpus. Let be the number of row vectors in the redundancy distribution matrix of the i-th text corpus. Let be the singularity weight of the i-th row vector in the redundancy distribution matrix of the i-th text corpus. Let be the mean of the i-th row vector in the redundancy distribution matrix of the i-th text corpus.
7. The large-scale model pre-training method for intelligent manufacturing as described in claim 6, characterized in that, Arrange the high-frequency redundancy interference degree of all clusters in each paragraph of each text corpus from small to large to form the redundancy interference vector of each paragraph. Then arrange the redundancy interference vectors of all paragraphs in paragraph order to obtain the redundancy distribution matrix of each text corpus. Singular value decomposition is performed on the redundant distribution matrix to extract the singular values of each row vector. The normalized result of the singular values of each row vector is used as the singularity weight of each row vector in the redundant distribution matrix.
8. The large-scale model pre-training method for intelligent manufacturing as described in claim 1, characterized in that, The method for obtaining the window fit value for each text corpus is as follows: In the formula, Let be the window fit value of the i-th text corpus. This is the floor function. The default initial value for the context window size. It is an exponential function with the natural constant as its base. This is the maximum value of the preset context window size. Let lexical suspiciousness be the value of the i-th text corpus. Let be the paragraph similarity of the i-th text corpus, where the mean of the Jaccard similarity coefficients between the paragraph vocabulary sets of any two paragraphs in the i-th text corpus is the paragraph similarity of the i-th text corpus.
9. The large-scale model pre-training method for intelligent manufacturing as described in claim 1, characterized in that, The method for obtaining the training dataset for the LLM large model pre-training is as follows: the window fit value of each text corpus is used as the size of the context window in each text corpus, the text vectors of all context windows in each text corpus are segmented, and the text vectors of all context windows after segmentation are used as the training dataset.
10. A large-scale model pre-training system for intelligent manufacturing, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the large model pre-training method for intelligent manufacturing as described in any one of claims 1-9.