Data processing method based on large language model and related equipment thereof
By introducing entity analysis and matrix processing technologies into large language models, the problem of poor semantic coherence in long text processing is solved, and the quality of content generation is significantly improved.
Patent Information
- Application Number
- CN202411492749.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-14
AI Technical Summary
When existing large language models process long text, it is difficult to maintain the consistency of context semantics, making it difficult to ensure the quality of content generation.
By introducing entity analysis, large language models are used to convert vectors, entity matrix processing and text matrix transformation of to-process text data, ensuring that context semantics are maintained when processing long text.
It effectively improves the quality of content generation of large language models in long text processing, ensuring that the generation results are closely related to the current topic and are semantically coherent.
Smart Images
Figure CN119443050A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of natural language processing, and in particular to a data processing method based on a large language model and related equipment. Background Art
[0002] Natural Language Processing (NLP) is an important field in the fields of computer science and artificial intelligence. Natural language processing studies various theories and methods that can achieve effective communication between humans and computers using natural language. For example, natural language processing technology can be used to process various corpus data, such as voice data and text data.
[0003] With the rapid development of artificial intelligence technology and hardware equipment, large language models (LLMs) have achieved remarkable results in the field of natural language processing. The existing large language model technology generally takes the Transformer framework as the core, and improves the content generation quality of large language models by optimizing pre-trained corpus, encoders, decoders and various fine-tuning tools. With the expansion of the application field of large language models, the number of scenarios that require large language models to analyze and understand large-volume tokens such as technical solutions, literary masterpieces, technical standards, and academic monographs is increasing. Correspondingly, the number of scenarios that require large language models to generate a large number of tokens is also increasing. However, the size of the predefined context window of the large language model itself is affected by multiple factors such as model architecture, implementation algorithm, and computing power resources, and the space for adjusting the window is relatively small. At present, the methods for solving the problem of long texts in large models have their own advantages and disadvantages. In practical applications, they all require a lot of tuning and testing, and the content quality is difficult to be well guaranteed. Summary of the invention
[0004] In view of this, the purpose of this application is to provide a data processing method based on a large language model and related devices, which can maintain the coherence of contextual semantics when processing long texts and improve the content generation quality of the large language model. The specific technical solutions are as follows:
[0005] In a first aspect, the present application provides a data processing method based on a large language model, the method comprising:
[0006] Get the text data to be processed;
[0007] Performing vector conversion on the text data to be processed by using a large language model to obtain a text vector and an entity vector corresponding to the text data to be processed;
[0008] Processing the entity vector using the large language model to obtain an entity matrix;
[0009] Processing the text vector based on the entity matrix using the large language model to obtain a text matrix;
[0010] The large language model is used to perform text conversion on the text matrix to obtain an output result corresponding to the text data to be processed.
[0011] In a possible implementation, the using of a large language model to perform vector conversion on the text data to be processed to obtain a text vector and an entity vector corresponding to the text data to be processed includes:
[0012] Using the large language model to perform entity extraction on the text data to be processed based on the entity data set to obtain an entity set;
[0013] Performing word segmentation processing on the text data to be processed based on the entity set to obtain a word segmentation set;
[0014] Performing vector conversion on the entity set to obtain the entity vector;
[0015] Perform vector conversion on the word segmentation set to obtain the text vector.
[0016] In a possible implementation, the entity vector is processed using the large language model to obtain an entity matrix, including:
[0017] Using the large language model to perform attention calculation on the entity vector to obtain an entity attention matrix;
[0018] Perform a linear transformation on the entity attention matrix to obtain the entity matrix.
[0019] In a possible implementation, the using the large language model to process the text vector based on the entity matrix to obtain a text matrix includes:
[0020] Using the large language model to perform block processing on the text vector based on the entity matrix to obtain a block vector;
[0021] Calculating a first similarity between each of the block vectors and each element in the entity matrix;
[0022] Screening the block vectors based on the first similarity to obtain a retained vector;
[0023] Performing attention calculation on the retained vector to obtain a text attention matrix;
[0024] Perform a linear transformation on the text attention matrix to obtain the text matrix.
[0025] In a possible implementation, the using the large language model to perform text conversion on the text matrix to obtain an output result corresponding to the text data to be processed includes:
[0026] Merging the elements in the text matrix using the large language model to obtain an output sequence;
[0027] The large language model is used to perform text conversion on the output sequence to obtain an output result corresponding to the text data to be processed.
[0028] In a second aspect, the present application further provides a data processing device based on a large language model, the device comprising:
[0029] A text acquisition module, used to acquire text data to be processed;
[0030] A vector conversion module, used for performing vector conversion on the text data to be processed by using a large language model to obtain a text vector and an entity vector corresponding to the text data to be processed;
[0031] An entity processing module, used to process the entity vector using the large language model to obtain an entity matrix;
[0032] A text processing module, used for processing the text vector based on the entity matrix using the large language model to obtain a text matrix;
[0033] The text conversion module is used to perform text conversion on the text matrix using the large language model to obtain an output result corresponding to the text data to be processed.
[0034] In a possible implementation, the vector conversion module includes:
[0035] An entity extraction unit, used to perform entity extraction on the text data to be processed based on the entity data set using the large language model to obtain an entity set;
[0036] A word segmentation processing unit, used for performing word segmentation processing on the text data to be processed based on the entity set to obtain a word segmentation set;
[0037] An entity conversion unit, used for performing vector conversion on the entity set to obtain the entity vector;
[0038] The word segmentation conversion unit is used to perform vector conversion on the word segmentation set to obtain the text vector.
[0039] In a possible implementation, the entity processing module includes:
[0040] An entity calculation unit, used to perform attention calculation on the entity vector using the large language model to obtain an entity attention matrix;
[0041] The entity transformation unit is used to perform a linear transformation on the entity attention matrix to obtain the entity matrix.
[0042] In a third aspect, the present application further provides a computer device, including: a memory and a processor;
[0043] Wherein, the memory is used to store computer programs;
[0044] The processor is used to execute the computer program in the memory to implement the method described in the first aspect or any one of the first aspects.
[0045] In a fourth aspect, the present application further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any one of the first aspects.
[0046] In the present application, the text data to be processed is obtained; the large language model is used to perform vector conversion on the text data to be processed, and the text vector and entity vector corresponding to the text data to be processed are obtained; the entity vector is processed by the large language model to obtain the entity matrix; the text vector is processed based on the entity matrix by the large language model to obtain the text matrix; the text matrix is converted to text by the large language model to obtain the output result corresponding to the text data to be processed. This application can improve the content generation quality of the large language model by introducing entity analysis to realize the analysis and processing of long text data and generate results. The use of the concept, data and related processing methods of entities can effectively ensure that the large language model maintains the coherence of contextual semantics when processing long texts, thereby ensuring that the large language model can stick to the current topic when generating results. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0048] Figure 1 A flow chart of a data processing method based on a large language model provided in an embodiment of the present application is shown;
[0049] Figure 2 A flowchart of a large language model training method provided in an embodiment of the present application is shown;
[0050] Figure 3 A schematic diagram of the structure of a data processing device based on a large language model provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0052] The emergence of large language models, which can be referred to as large models for short, has opened a new chapter for artificial intelligence. With its excellent semantic understanding ability and end-to-end content generation ability, large models have become a hot topic of concern for all parties in the industry and technology. All parties expect that the implementation of large models can solve the problems existing in the current industry scenarios, and realize the quality and efficiency improvement of production, management, services and other links through intelligence, and even innovate business models.
[0053] Existing large model technologies generally use the Transformer framework as the core, and improve the quality of content generation of large language models by optimizing pre-trained corpus, encoders, decoders, and various fine-tuning tools. With the expansion of the application fields of large models, the number of scenarios that require large models to analyze and understand large-volume tokens such as technical solutions, literary masterpieces, technical standards, and academic monographs is increasing. Correspondingly, the number of scenarios that require large models to generate a large number of tokens is also increasing. However, the size of the predefined context window of the large model itself is affected by multiple factors such as the model architecture, implementation algorithm, and computing power resources, and the space for adjusting the window is relatively small.
[0054] At present, there are many methods to solve the problem of large models and long texts, but each has its own advantages and disadvantages:
[0055] The problem of long text in large models can be solved by increasing the computing power supply of large models, so that large models can read and generate longer texts. However, the increase in text length of large models based on the traditional Transformer framework increases the computing power demand at a quadratic complexity, and the increase in text length will cause a sharp increase in computing power costs.
[0056] The idea of division-generalization is adopted to solve the problem of long text in large models. The long text is divided into blocks and then input into the large model, so that the content of each block is smaller than the context window threshold. The semantic processing capabilities of the large model such as summary analysis and simplified content are used to generate content of a specific size, and then the related content is merged to achieve content refinement to help the large model understand; when generating, the opposite method is adopted, first generating refined content, and then expanding the refined content in blocks. This method effectively avoids the algorithm and computing power pressure brought by one-time processing of long texts, but there is inevitable information loss in the processing process, and the intermediate process uses the large model for processing multiple times, which is easy to cause the expansion of problems such as model hallucination, and the content quality is difficult to be well guaranteed.
[0057] The problem of long text in large models is solved by optimizing the Transformer framework. For example, the Megalodon model launched by Mega and the University of Southern California optimizes the Attention calculation, introduces the gated attention mechanism of the gated recurrent unit (GRU) and the gated attention unit (GAU), and the exponential moving average method; the Infini-Transformer proposed by Google introduces local masked attention and long-term linear attention mechanisms in the attention mechanism to achieve limited memory usage and long text input support; in the field of large Chinese models, Kimi's Moonshot model uses the Retrieval-Augmented Generation (RAG) technology and external knowledge bases to achieve long text support capabilities. The above optimizations are the key to solving the problem of long texts, but the relevant technologies are still low-level and general, and a lot of tuning and testing are needed to achieve application in industry scenarios.
[0058] Therefore, the embodiment of the present application provides a data processing method based on a large language model, which introduces entity associations to assist in building long text semantic associations, reconstructs the traditional large language model, and ultimately realizes the large language model's ability to understand and generate long texts.
[0059] See also Figure 1 , shows a flow chart of a data processing method based on a large language model provided in an embodiment of the present application, and the embodiment of the present application at least includes the following steps:
[0060] S11, obtaining text data to be processed.
[0061] The text data to be processed in the embodiment of the present application may be long text data. After obtaining the text data to be processed, the text data to be processed may be input into a pre-trained large language model. The large language model may support different input forms such as question and answer, forms, and interfaces.
[0062] S12, using the large language model to perform vector conversion on the text data to be processed, to obtain a text vector and an entity vector corresponding to the text data to be processed.
[0063] The embedding layer of the large language model is used to convert the text data to vectors, that is, to vectorize the text data to be processed. Text vectorization is to represent text as a series of vectors that can express the semantics of the text, which is an important way to represent text.
[0064] For the input long text data, you can first use the word segmentation tool to perform word segmentation on the text data to be processed. In this stage of word segmentation processing, in addition to the basic word segmentation dictionary, you must also introduce the entity set in the text data to be processed as the word segmentation dictionary to ensure that the entities in the text data to be processed will not be broken down into multiple units and lose their meaning during the word segmentation process.
[0065] The entity set can be obtained from the text data to be processed, for example, by traversing the text data to be processed based on an entity data set related to the text data to be processed, or by extracting it from the text data to be processed using an entity extraction tool based on an entity data set related to the text data to be processed.
[0066] Entities in entity data sets refer to objects that exist in reality or in a specific industry field and can be described independently. They can be specific objects such as people, things, and objects, or they can be objects of a certain category.
[0067] Methods for obtaining entity data sets include but are not limited to: screening from industry materials such as encyclopedias and dictionaries; extracting from data such as knowledge graphs; using natural language processing tools to perform named entity recognition from industry technical materials, industry solutions, industry introduction materials and other industry-related texts; and manually sorting based on knowledge and experience. Then, the entity data set related to the text data to be processed and the text data to be processed belong to the same industry.
[0068] Based on this, using a large language model to perform vector conversion on the text data to be processed to obtain a text vector and an entity vector corresponding to the text data to be processed may include the following steps:
[0069] S121, using a large language model to extract entities from the text data to be processed based on the entity data set to obtain an entity set;
[0070] S122, using the large language model to perform word segmentation on the text data to be processed based on the entity set to obtain a word segmentation set;
[0071] S123, using the large language model to perform vector conversion on the entity set to obtain an entity vector;
[0072] S124, using a large language model to perform vector conversion on the word segmentation set to obtain a text vector.
[0073] The text vector can be expressed as: WCL = {wcl1,…wcl m}, the entity vector can be expressed as: WCEL = {wcel1,…wcel n}. m represents the number of words in the word set, and n represents the number of entities in the entity set. Each element in the text vector WCL is a word feature vector, and each element in the entity vector WCEL is an entity feature vector.
[0074] After obtaining the entity vector WCEL, the entity vector WCEL can also be supplemented. The association threshold α is pre-set, and the entity feature vector wcel in the entity vector WCEL i , if there is an entity feature vector wce in the entity training vector WCE j With the entity feature vector wcel i The entity association degree d i,j ≥α, then the entity feature vector wce j Add to the entity vector WCEL. The entity training vector WCE is obtained by vectorizing the entity data set related to the text data to be processed.
[0075] S13, using the large language model to process the entity vector to obtain an entity matrix.
[0076] The embodiment of the present application can use the attention mechanism of the large language model to process the entity vector to obtain the entity attention representation.
[0077] Using a large language model to process the entity vector to obtain an entity matrix can include the following steps:
[0078] S131, use the large language model to calculate the attention of the entity vector to obtain the entity attention matrix.
[0079] The entity attention layer of the large language model can pay attention to each wcel in the entity vector WCEL i To perform masking, first calculate the dot product or similarity score of the query matrix and all key matrices to obtain the non-masked matrix WCELQK T =WCELQ×WCELK T . After getting WCELQK T After that, the mask matrix (MASK matrix) is used to transform WCELQK T Mask to get MASKWCELQK T , and then use the softmax function to normalize these MASKWCELQKT , and get the attention weight distribution. The attention weight distribution is finally multiplied by the value matrix WCELV to get the output result WCELZ i . Use Concat() to combine multiple WCELZ i The concatenation is performed to obtain the entity attention matrix WCELZM. The specific process of converting the entity vector into the entity matrix will be explained in the subsequent large language model training.
[0080] Among them, WCELQ is the query matrix, WCELK is the key matrix, WCELV is the value matrix, WCELQ = WCEL × WQ, WCELK T =(WCEL×WK) T , WCELV=WCEL×WV. WQ, WK and WV are the corresponding matrices that convert the vector into the query (Query, Q) matrix, the key (Key, K) matrix and the value (Value, V) matrix. The MASK matrix is used to indicate the valid part of the data and mask the information after each word.
[0081] S132, using the large language model to perform a linear transformation on the entity attention matrix to obtain an entity matrix.
[0082] The entity attention matrix WCELZM is linearly transformed using the transformation matrix to obtain the final prediction result, which is the entity matrix WCELY. Each element in the entity matrix WCELY represents the prediction result of an entity, which can also be understood as the context vector of an entity. This vector integrates the information of all positions in the entity vector, but the degree of information contribution of different positions is determined by their attention relative to the query.
[0083] For each wcel i Mask processing is to process each wcel i A linear transformation is performed, so the dimension of the entity attention matrix WCELZM may be different from the dimension of the entity vector WCEL. Then the transformation matrix is used to convert the concatenated matrix back to the original dimension.
[0084] S14, using the large language model to process the text vector based on the entity matrix to obtain a text matrix.
[0085] The embodiment of the present application can use the attention mechanism of a large language model to process the text vector to obtain a text attention representation.
[0086] Using a large language model to process the text vector based on the entity matrix to obtain a text matrix may include the following steps:
[0087] S141, using the large language model to perform block processing on the text vector based on the entity matrix to obtain block vectors.
[0088] The basic attention layer of the large language model can process the text vector WCL in blocks based on the entity matrix WCELY before using the attention mechanism to process the text vector. While ensuring the entity relevance at the semantic level, on the one hand, by reducing the input content, the data pressure during model processing is reduced, and on the other hand, multiple data blocks can be processed in parallel to improve processing efficiency.
[0089] When dividing the blocks, the following methods may be used: dividing by the number of characters, vector size, etc.; dividing by chapters, paragraphs, sentences, etc. of natural language. The above are only examples, and the embodiments of the present application do not limit the specific block processing method.
[0090] S142, calculating the first similarity between each block vector and each element in the entity matrix.
[0091] For any wcely i ∈WCELY, using semantic similarity calculation method to analyze wcely i Each block with text vector WCL wclb r Similarity(wcely i ,wclb r ). The specific formula is as follows:
[0092]
[0093] In the above formula, y is expressed as wcely i Any entity in d y For block wclb r The entity association between entity y and other entities in WCL, K is the total number of entities in WCL, El k represents the sum of entity associations between the kth entity and other entities in WCL, ELN k Indicates block wclb r The entity association degree between the kth entity and other entities in wclb r Represents the rth block vector of the text vector WCL.
[0094] Leveraging Similarity(wcely i ,wclb r ) method, the correlation between each block content in the input text and each entity in the input text, as well as the overall correlation between each block content in the input text and the entity in the input text, that is, the first similarity, can be obtained. The formula is as follows:
[0095]
[0096] S143, screening the block vectors based on the first similarity to obtain a retained vector.
[0097] The embodiment of the present application can further filter the blocks of the text vector WCL according to the first similarity, including but not limited to: for wclbs r Blocks wclb that are less than or equal to the similarity threshold r After the block vectors are screened, the remaining block vectors are used as the reserved vectors.
[0098] S144, perform attention calculation on the retained vector to obtain a text attention matrix.
[0099] S145, perform linear transformation on the text attention matrix to obtain a text matrix.
[0100] S144 and S145 can refer to S131 and S132. The difference between the two is that the input vectors are different, which will not be repeated here.
[0101] In one implementation, the first similarity can be used as an added item to the three matrices WQ, WK, and WV, that is, the first similarity is added to each element value in the original matrix to form new WQ, WK, and WV, and the new WQ, WK, and WV are used for attention calculation.
[0102] S15, using the large language model to perform text conversion on the text matrix to obtain an output result corresponding to the text data to be processed.
[0103] Because the output result of the large language model is text data, after obtaining the text matrix, the embodiment of the present application also needs to perform text conversion on the text matrix to finally obtain the output result corresponding to the text data to be processed.
[0104] Specifically, using a large language model to perform text conversion on a text matrix to obtain an output result corresponding to the text data to be processed may include the following steps:
[0105] S151, using the large language model to merge the elements in the text matrix to obtain an output sequence.
[0106] When generating the text matrix, the text vector WCL is processed in blocks, so each element in the text matrix can be understood as each block vector. The embodiment of the present application can merge each block vector in the order of the text data to be processed to obtain an output sequence, or can merge each block vector in the order of the text data to be processed, and intercept a certain range of context at the joint of two adjacent block vectors to form a new block vector, repeat the S14 process, and add the result between two adjacent block vectors to finally form a complete output sequence.
[0107] Based on this, the elements in the text matrix are merged using a large language model to obtain an output sequence, which can include the following steps:
[0108] Get the adjacent block vector in the text matrix, where the adjacent block vector is the vector where two block vectors are spliced;
[0109] Calculate the second similarity between each adjacent block vector and each element in the entity matrix;
[0110] Screening adjacent block vectors based on the second similarity to obtain adjacent reserved vectors;
[0111] Perform attention calculation on the adjacent retained vectors to obtain the adjacent attention matrix;
[0112] Perform a linear transformation on the adjacent attention matrix to obtain the adjacent text matrix;
[0113] Merge the elements in the text matrix with the elements in the adjacent text matrix to obtain the output sequence.
[0114] The embodiment of the present application intercepts a certain range of context at the joint of two adjacent block vectors to form a new block vector, which can make the output sequence more complete, make the text data restored from the output sequence more fluent, and make the context connection more natural.
[0115] S152, using the large language model to perform text conversion on the output sequence to obtain an output result corresponding to the text data to be processed.
[0116] The large language model is used to convert the output sequence into text based on the information during text vectorization and output it.
[0117] In an embodiment of the present application, the text data to be processed is obtained; the large language model is used to perform vector conversion on the text data to be processed to obtain a text vector and an entity vector corresponding to the text data to be processed; the entity vector is processed using the large language model to obtain an entity matrix; the text vector is processed based on the entity matrix using the large language model to obtain a text matrix; the text matrix is converted to text using the large language model to obtain an output result corresponding to the text data to be processed. The embodiment of the present application can improve the content generation quality of the large language model by introducing entity analysis to achieve analysis and processing of long text data and result generation. The use of the concept, data and related processing methods of entities can effectively ensure that the large language model maintains the coherence of contextual semantics when processing long texts, thereby ensuring that the large language model can closely follow the current topic when generating results.
[0118] By training the large language model, the large language model can be equipped with the function of processing long text data, and can realize the word segmentation, vectorization and entity relationship extraction of the input text. By using the reasoning service of the entity attention layer of the large language model, the entity association relationship in the long text can be constructed. By using the reasoning service of the basic attention layer of the large language model, the optimization and parallel generation of the block sequence can be realized based on the entity association relationship and sequence block. The large language model can provide the block merging function to realize the orderly merging and splicing optimization of the block sequence. It can also provide the mapping transformation function of the generated sequence to the text to realize the final text output.
[0119] Next, the process of training a large language model in the embodiment of the present application is introduced. Figure 2 , training a large language model can include the following steps:
[0120] S21, obtaining an entity data set.
[0121] The entities in the entity data set of the embodiment of the present application refer to objects that exist in reality or in a specific industry field and can be independently described. They can be specific objects such as people, things, and objects, or they can be objects of a certain category.
[0122] Methods for obtaining entity data include, but are not limited to: screening from industry materials such as encyclopedias and dictionaries; extracting from data such as knowledge graphs; using natural language processing tools to perform named entity recognition from industry-related texts such as industry technical materials, industry solutions, and industry introduction materials; and manually organizing based on knowledge and experience.
[0123] The entity data set can be expressed as: E = {e1,…e n}.
[0124] S22, vectorize the training text and entity datasets.
[0125] The training text described in natural language needs to be vectorized to map the text into numerical vectors. This process can be processed using a variety of word embedding tools such as Word2Vec, GloVe (Global Vectors for Word Representation), FastText, BERT (Bidirectional Encoder Representation from Transformers), etc. The self-attention mechanism of Transformer usually includes two parts: word vector and position vector. Generally, the two can be combined to obtain a vector matrix as the input content of the attention layer. Therefore, the embodiment of the present application can also use it as an optional method in the vectorization process.
[0126] Before vectorizing the training text, you can first use the word segmentation tool to segment the training text. In the word segmentation process, in addition to the basic word segmentation dictionary, you can also introduce the entity dataset as the word segmentation dictionary to ensure that the entity will not be disassembled into multiple units and lose its meaning during the model processing. You can also extract entities from the training text based on the entity dataset to obtain a training entity set.
[0127] The data sources of the training text may include, but are not limited to: open source pre-training corpora, commercial pre-training corpora, and custom pre-training corpora.
[0128] The vectorized training text can be represented as the training text vector WC = {wc1,…wc m}, the vectorized training entity set can be represented as the training entity vector WCE = {wce1,…,wce m Each element in the training text vector WC is a word segmentation feature vector, and each element in the training entity vector WCE is an entity feature vector.
[0129] In the training process, the embodiment of the present application vectorizes the training text and entity data set, so that the large language model can provide vectorization processing function of text data and has the ability of text vectorization.
[0130] S23, construct entity semantic associations.
[0131] In the training text vector WC, there is actually a preliminary probabilistic reflection of the semantic relationship between each unit word in the training text, but it is still unable to accurately describe the semantic relationship between each entity in the training text.
[0132] The embodiment of the present application may adopt a similarity-based entity relationship construction method, for each pair of entities (e i ,e j ), the joint probability can be expressed as:
[0133]
[0134] For every pair of entities (e i ,e j ), the empirical distribution can be expressed as:
[0135]
[0136] Among them, p i,j (e i ,e j ) represents the joint probability, that is, entity e i ,e j In the word segmentation feature vector wc i ,wc j The probability of the distribution in represents the empirical distribution statistic; U represents the joint cumulative distribution statistic of the training entity set; u i,j Represents entity e i ,e j In the word segmentation feature vector wc i ,wc j Random distribution statistics in .
[0137] The relationship between any two entities can be described as:
[0138]
[0139] Wherein, d() represents the distance between two elements. In the embodiment of the present application, the distance can be calculated by using methods such as Euclidean distance, cosine distance, KL discreteness, etc.; d i,，j Can be regarded as entity e i ,e j The semantic similarity between any two entities can be established.
[0140] In the training process, the embodiment of the present application constructs entity semantic associations, so that the large language model can provide an entity association extraction function based on semantic similarity analysis and has the ability to calculate entity associations.
[0141] S24, attention layer training based on entity semantic association.
[0142] The embodiment of the present application adds an entity attention layer and a basic attention layer to the original large language model to form the large language model of the embodiment of the present application. During the training process of the large language model, the entity attention layer and the basic attention layer are mainly trained.
[0143] The embodiment of the present application can use weight matrices WQ, WK, and WV to transform the input vector a to obtain a query matrix Q, a key matrix K, and a value matrix V.
[0144] Q=a×WQ, K=a×WK, V=a×WV;
[0145] The values of WQ, WK, and WV may be random numbers or other values, but it is necessary to ensure that WQ, WK, and WV do not use the same value.
[0146] For the training text vector WC, the embodiment of the present application can use the weight matrices WQ, WK, and WV to train each word segmentation feature vector wc i Transform and obtain the query matrix Q i , key matrix K i , value matrix V i .
[0147] Q i =wc i ×WQ,K i =wc i ×WK,V i =wc i ×WV.
[0148] For training entity vector WCE, the embodiment of the present application can use weight matrices WQ, WK, and WV to train each entity feature vector wec i Convert and get the query matrix WCEQ i , key matrix WCEK i , value matrix WCEV i .
[0149] WCEQ i =wec i ×WQ,WCEK i =wec i ×WK,WCEV i =wec i ×WV.
[0150] The embodiment of the present application can use the Self-Attention() method to construct a self-attention layer, which can be expressed as:
[0151]
[0152] Multi-Head Attention(Q,K,V)=Concat(ZS1,…,ZS h )W t ;
[0153] Among them, a Self-Attention (Q, K, V) represents a ZS, d k represents the dimension of the attention layer, W trepresents the transformation matrix, h represents the number of input vectors, softmax() is the activation function, and concat() is used to connect two or more arrays.
[0154] Multiple ZS are concatenated together through concat() to form an h×d k Using W t Convert the concatenated matrix back to the original dimension d k The linear transformation matrix. After the linear transformation, Multi-Head Attention() can get the output result ZM. For the input vector and Self-Attention(), Multi-Head Attention(), the dimensions are consistent.
[0155] The embodiment of the present application can construct an entity attention layer and a basic attention layer in the original large language model through the above construction process. The original large language model can adopt Transformer or BERT, ERNIE and other similar models, and can also use residual links and fully connected layers to converge the model.
[0156] For the training text vector WC, the Self-Attention() and Multi-Head Attention() methods are used for processing, and the processed result WCZM can be obtained.
[0157] For the training entity vector WCE, the Self-Attention() and Multi-Head Attention() methods are used for processing, and the processed result WCEZM can be obtained.
[0158] It should be noted that during the conversion process between Self-Attention() and Multi-Head Attention(), after multiple rounds of linear transformation and inner product processing, the correspondence between the entity training vector and the training text vector also changes synchronously.
[0159] S25, model parameter tuning.
[0160] In the training process of S24, the parameters of the large language model are tuned through multiple rounds of repeated training.
[0161] During the training process of the large language model, the large language model can manage and process the training text and entity data sets, provide vectorized processing functions for text data, provide entity relationship extraction functions based on semantic similarity analysis, and provide support for hierarchical attention model training based on entity relationships. It can realize the training and relationship mapping of the entity attention layer and the basic attention layer. In addition, it also provides parameter tuning capability support.
[0162] Next, a data processing device based on a large language model provided in an embodiment of the present application is introduced. The data processing device based on a large language model introduced below and the data processing method based on a large language model introduced above can refer to each other.
[0163] See also Figure 3 , shows a schematic diagram of the structure of a data processing device based on a large language model provided in an embodiment of the present application, the device comprising:
[0164] The text acquisition module 301 is used to acquire the text data to be processed;
[0165] A vector conversion module 302 is used to perform vector conversion on the text data to be processed using a large language model to obtain a text vector and an entity vector corresponding to the text data to be processed;
[0166] An entity processing module 303 is used to process the entity vector using the large language model to obtain an entity matrix;
[0167] A text processing module 304, configured to process the text vector based on the entity matrix using the large language model to obtain a text matrix;
[0168] The text conversion module 305 is used to perform text conversion on the text matrix using the large language model to obtain an output result corresponding to the text data to be processed.
[0169] In the embodiment of the present application, the vector conversion module includes:
[0170] An entity extraction unit, used to perform entity extraction on the text data to be processed based on the entity data set using the large language model to obtain an entity set;
[0171] A word segmentation processing unit, used for performing word segmentation processing on the text data to be processed based on the entity set to obtain a word segmentation set;
[0172] An entity conversion unit, used for performing vector conversion on the entity set to obtain the entity vector;
[0173] The word segmentation conversion unit is used to perform vector conversion on the word segmentation set to obtain the text vector.
[0174] In the embodiment of the present application, the entity processing module includes:
[0175] An entity calculation unit, used to perform attention calculation on the entity vector using the large language model to obtain an entity attention matrix;
[0176] The entity transformation unit is used to perform a linear transformation on the entity attention matrix to obtain the entity matrix.
[0177] In the embodiment of the present application, the text processing module includes:
[0178] A block processing unit, used for performing block processing on the text vector based on the entity matrix using the large language model to obtain a block vector;
[0179] A similarity calculation unit, used for calculating a first similarity between each of the block vectors and each element in the entity matrix;
[0180] A screening unit, configured to screen the block vectors based on the first similarity to obtain a retained vector;
[0181] A text calculation unit, used for performing attention calculation on the retained vector to obtain a text attention matrix;
[0182] A text transformation unit is used to perform a linear transformation on the text attention matrix to obtain the text matrix.
[0183] In the embodiment of the present application, the text conversion module 305 includes:
[0184] A merging unit, used to merge the elements in the text matrix using the large language model to obtain an output sequence;
[0185] A text conversion unit is used to perform text conversion on the output sequence using the large language model to obtain an output result corresponding to the text data to be processed.
[0186] The embodiment of the present application also provides a computer device, including: a memory and a processor;
[0187] Wherein, the memory is used to store computer programs;
[0188] The processor is used to execute the computer program in the memory to implement the method described in the above method embodiment.
[0189] The embodiment of the present application also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the method described in the above method embodiment.
[0190] In an embodiment of the present application, a text acquisition module acquires text data to be processed; a vector conversion module uses a large language model to perform vector conversion on the text data to be processed, and obtains a text vector and an entity vector corresponding to the text data to be processed; an entity processing module uses a large language model to process the entity vector and obtain an entity matrix; a text processing module uses a large language model to process the text vector based on the entity matrix and obtain a text matrix; a text conversion module uses a large language model to perform text conversion on the text matrix and obtain an output result corresponding to the text data to be processed. An embodiment of the present application can improve the content generation quality of a large language model by introducing entity analysis to realize the analysis and processing of long text data and generate results. Using the concept, data and related processing methods of entities, it can effectively ensure that the large language model maintains the coherence of contextual semantics when processing long texts, thereby ensuring that the large language model can closely follow the current topic when generating results.
[0191] It should be noted that the same or similar parts between the various embodiments can be referred to each other. As for the device embodiment and the system embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0192] For the above-mentioned embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0194] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
[0195] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A data processing method based on a large language model, characterized in that: The method comprises: Get the text data to be processed; Performing vector conversion on the text data to be processed by using a large language model to obtain a text vector and an entity vector corresponding to the text data to be processed; Processing the entity vector using the large language model to obtain an entity matrix; Processing the text vector based on the entity matrix using the large language model to obtain a text matrix; The large language model is used to perform text conversion on the text matrix to obtain an output result corresponding to the text data to be processed.
2. The method according to claim 1, characterized in that: The step of performing vector conversion on the text data to be processed by using a large language model to obtain a text vector and an entity vector corresponding to the text data to be processed includes: Using the large language model to perform entity extraction on the text data to be processed based on the entity data set to obtain an entity set; Performing word segmentation processing on the text data to be processed based on the entity set to obtain a word segmentation set; Performing vector conversion on the entity set to obtain the entity vector; Perform vector conversion on the word segmentation set to obtain the text vector.
3. The method according to claim 1, characterized in that The entity vector is processed by using the large language model to obtain an entity matrix, including: Using the large language model to perform attention calculation on the entity vector to obtain an entity attention matrix; Perform a linear transformation on the entity attention matrix to obtain the entity matrix.
4. The method according to claim 1, characterized in that: The step of processing the text vector based on the entity matrix using the large language model to obtain a text matrix includes: Using the large language model to perform block processing on the text vector based on the entity matrix to obtain a block vector; Calculating a first similarity between each of the block vectors and each element in the entity matrix; Screening the block vectors based on the first similarity to obtain a retained vector; Performing attention calculation on the retained vector to obtain a text attention matrix; Perform a linear transformation on the text attention matrix to obtain the text matrix.
5. The method according to claim 1, characterized in that The step of performing text conversion on the text matrix using the large language model to obtain an output result corresponding to the text data to be processed includes: Merging the elements in the text matrix using the large language model to obtain an output sequence; The large language model is used to perform text conversion on the output sequence to obtain an output result corresponding to the text data to be processed.
6. A data processing device based on a large language model, characterized in that: The device comprises: A text acquisition module, used to acquire text data to be processed; A vector conversion module, used for performing vector conversion on the text data to be processed by using a large language model to obtain a text vector and an entity vector corresponding to the text data to be processed; An entity processing module, used to process the entity vector using the large language model to obtain an entity matrix; A text processing module, used for processing the text vector based on the entity matrix using the large language model to obtain a text matrix; The text conversion module is used to perform text conversion on the text matrix using the large language model to obtain an output result corresponding to the text data to be processed.
7. The device according to claim 6, characterized in that The vector conversion module comprises: An entity extraction unit, used to perform entity extraction on the text data to be processed based on the entity data set using the large language model to obtain an entity set; A word segmentation processing unit, used for performing word segmentation processing on the text data to be processed based on the entity set to obtain a word segmentation set; An entity conversion unit, used for performing vector conversion on the entity set to obtain the entity vector; The word segmentation conversion unit is used to perform vector conversion on the word segmentation set to obtain the text vector.
8. The device according to claim 6, characterized in that The entity processing module includes: An entity calculation unit, used to perform attention calculation on the entity vector using the large language model to obtain an entity attention matrix; The entity transformation unit is used to perform a linear transformation on the entity attention matrix to obtain the entity matrix.
9. A computer device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is configured to execute the computer program in the memory to implement the method according to any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that: The device stores instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 5.