Transform-based artificial intelligence model, system and device
By combining the GPT2 generator and CPU computation in the transformer model, an efficient artificial intelligence model is realized on low-computing-power devices, solving the problem of high resource consumption, supporting a variety of natural language processing tasks, and applicable to a variety of hardware platforms.
Patent Information
- Application Number
- CN202410980127.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
AI Technical Summary
The training process of existing generative language models requires a large amount of computing resources and is costly, making it difficult for ordinary users to develop and use them on their own.
Design an artificial intelligence model based on transformer, combining a transformer decoder and a GPT2 generator, utilizing the CPU for self-attention and feedforward neural network calculations, adapting to miniaturized devices, and integrating modules for text correction, article writing assistance, contextual understanding, and image and text generation.
It improves the computational efficiency of the model on the CPU, reduces hardware and resource requirements, supports a variety of natural language processing tasks, is suitable for a variety of hardware platforms, and has high accuracy and interactivity.
Smart Images

Figure CN118917360A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of data processing and artificial intelligence, and in particular to an improved transformer artificial intelligence model capable of using a CPU for calculations. Background Art
[0002] At present, there are a large number of generative language models on the market, such as GPT, LLAMA and other models. The training process of these large models involves a large amount of computing and storage resources, and large-scale computing is required to obtain good model effects. The training process requires thousands of integrated GPUs and other hardware devices as well as power, space and other resources. The consumption of these resources will lead to high costs and waste.
[0003] For ordinary users, developing such a large-scale dialogue generation model requires high technical barriers and expertise, as well as high resource costs. Even ordinary companies find it difficult to develop and use it on their own. Summary of the invention
[0004] To this end, it is necessary to provide a technical solution that can simplify and miniaturize artificial intelligence models, so that the artificial intelligence learning models can be transformed to be more suitable for operation on low-computing power devices.
[0005] To achieve the above-mentioned objectives, the inventors provide a transformer-based artificial intelligence model, comprising: a transformer decoder, the transformer decoder comprising several layers of first processing layers and several layers of second processing layers, the first processing layer comprising a multi-head self-attention layer and a first position-aware feedforward network layer; the second processing layer comprising a masked multi-head self-attention layer, a multi-head attention layer and a second position-aware feedforward network layer, the output of the first processing layer is connected to the input of the multi-head attention layer; the output end of the transformer decoder is used to connect to a GPT2 generator, the GPT2 generator comprising a first normalization layer, a second multi-head self-attention layer, a fully connected layer, a linear layer and a softmax layer; the output of the GPT2 generator is a probability output.
[0006] In some embodiments of the present application, the GPT2 generator uses the RMSNorm normalization function and the activation function of the GPT2 generator is SwiGLU.
[0007] In some other embodiments of the present application, a position encoder is further included, and the position encoder is arranged at the input end of the transformer decoder.
[0008] In some embodiments of the present application, a text error correction module, an article writing assistance module, a context connection module, and a graphic and text generation module are also included.
[0009] In a specific embodiment of the present application, the text correction module is used to use the text correction task prompt words and text error samples as input, and use the modified target samples of the correct text as output for fine-tuning training.
[0010] In a specific embodiment of the present application, the article writing assistance module is used to receive user writing requirements, parse the user writing requirements, and generate auxiliary articles.
[0011] In a specific embodiment of the present application, the contextual contact module is used to record the user's historical conversations. When the user brings up the current conversation, the historical conversations are called or the associative words extracted from the historical conversations are retrieved to generate a reply field to the user.
[0012] In a specific embodiment of the present application, the image and text generation module is used to classify each paragraph of the generated text when generating text, crawl the image information of the corresponding category of each paragraph, insert the image information into the corresponding paragraph of the generated text, and generate image and text information.
[0013] A transformer-based artificial intelligence system includes the artificial intelligence model as described above.
[0014] A transformer-based artificial intelligence device comprises a computer in which the above-mentioned artificial intelligence system runs, and the device also comprises an external input device.
[0015] Different from the existing technology, the above technical solution can calculate each neural network layer of the fused model independently, because the input of each neuron only depends on the output of the previous layer, which makes the CPU fully utilize multiple cores for self-attention and feedforward neural network calculations, thereby improving the computing efficiency of the model on the CPU. The system is suitable for various hardware platforms, making the system more universal. The system supports a variety of natural language processing tasks.
[0016] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and shall not be considered as limitations of the present application.
[0018] In the drawings of the specification:
[0019] Figure 1 This is a schematic diagram of a transformer-based artificial intelligence model described in a specific implementation method;
[0020] Figure 2 It is a structural schematic diagram of the GPT2 generator described in the specific implementation method;
[0021] Figure 3 This is a schematic diagram of a text error correction module according to a specific implementation example 1;
[0022] Figure 4 A schematic diagram of an auxiliary module for writing articles as described in a specific implementation example 1;
[0023] Figure 5 A schematic diagram of a contextual connection module according to a specific implementation example 1;
[0024] Figure 6 This is a schematic diagram of a graphic and text generation module according to a specific implementation example 1;
[0025] Figure 7 A schematic diagram of a transformer-based artificial intelligence model device as described in a specific implementation method. DETAILED DESCRIPTION
[0026] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.
[0027] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.
[0028] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.
[0029] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.
[0030] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.
[0031] Without further limitations, in this application, the words "include", "comprises", "has" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.
[0032] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.
[0033] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.
[0034] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.
[0035] See also Figures 1 to 7 , the embodiment shown in the figure is a transformer-based artificial intelligence model, including: a transformer decoder, the transformer decoder includes several layers of first processing layers and several layers of second processing layers, the first processing layer includes a multi-head self-attention layer and a first position-aware feedforward network layer; the second processing layer includes a masked multi-head self-attention layer, a multi-head attention layer and a second position-aware feedforward network layer, and the output of the first processing layer is connected to the input of the multi-head attention layer; the output end of the transformer decoder is used to connect to the GPT2 generator, the GPT2 generator includes a first normalization layer, a second multi-head self-attention layer, a fully connected layer, a linear layer and a softmax layer; the output of the GPT2 generator is a probability output.
[0036] Among them, the artificial intelligence model can be a model that helps analyze and identify the content of text and pictures with the help of computer computing power. The processing layer is the sum of several defined data transmission relationships, and the processing layer can include several sub-function layers. In some specific embodiments, the multi-head self-attention layer (Multi-Head Attention) is an extended form of the attention mechanism, which can extract information more effectively when processing sequence data. In the standard attention mechanism, a weighted context vector is calculated to represent the information of the input sequence. In multi-head attention, multiple sets of attention weights are used, each set of weights can learn different semantic information, and each set of weights will generate a context vector. The feedforward network layer (freeforward) is a unidirectional multi-layer network structure. Information starts from the input layer and is transmitted layer by layer in one direction, that is, unidirectional transmission, until the output layer ends. Feedforward means that the propagation direction refers to the forward direction. The normalization layer (Normalizationlayer) allows each sample feature to have the same mean and variance in the channel dimension, thereby improving the generalization ability of the model.
[0037] This application adjusts the output of the transformer model and superimposes the GPT2 generator model to further extract the features of the text, which can better generate a predictive text model. Each neural network layer of the fused model can be calculated independently, because the input of each neuron only depends on the output of the previous layer, which allows the CPU to make full use of multiple cores for self-attention and feedforward neural network calculations, thereby improving the computational efficiency of the model on the CPU. The specific performance is also related to factors such as the number of CPU cores, frequency, and the implementation method of parallel computing.
[0038] In some embodiments of the present application, the GPT2 generator uses the RMSNorm normalization function, and the activation function of the GPT2 generator is SwiGLU. Through the above changes, the GPT2 generator model is connected to the output end of the transformer decoder layer, and the RMSNorm normalization function and activation function used in the model are replaced with SwiGLU. The normalization function and activation function are commonly used in some existing large models such as the LLAMA model, which cannot be loaded and used on ordinary hardware. Here, the function replacement in the model is used, and in the miniaturized artificial intelligence model, the text processing work can be better performed.
[0039] In other embodiments of the present application, a position encoder is also included, and the position encoder is arranged at the input end of the transformer decoder. The position encoder (Embedding) layer is intended to use Word2Vec and EmbeddingsWord2Vec algorithms to perform dimensionality reduction on the matrix, which is actually to represent the semantic information of words in the form of word vectors by learning text, that is, through an embedding space, semantically similar words are very close in the space. Encoding can facilitate the analysis of text materials and achieve a better technical effect of semantic processing.
[0040] In some embodiments of the present application, a text correction module, an article writing auxiliary module, a contextual connection module, and a graphic generation module are also included. The system has higher accuracy and intelligence, and can quickly obtain the information required by the user. The system can use the transformer model and the GPT2 model at the same time, which improves the accuracy of the system and the effects of question and answer and dialogue. Compared with other question and answer systems, the scale of this system is relatively small. With the integration of a large amount of training data and multiple models, it can still provide relatively reasonable and fast question and answer and dialogue services. According to our needs, we can do vertical task training and fine-tuning, such as text correction, article generation and other tasks, and we can input our data for fine-tuning based on the model.
[0041] This application Figure 3In the illustrated embodiment, the text error correction module 300 is used to use the text error correction task prompt words and text error samples as input, and the target samples of the correct text after modification as output for fine-tuning training. In some specific embodiments, the fused model is fine-tuned by the text error correction samples so that the model supports the text error correction function. A large number of text error correction data sets are used, and the data set format is: samples containing text errors and target samples of the correct text after modification. The training of the model uses the text error correction task prompt words + text error samples as input, and the output is the target sample of the correct text after modification for fine-tuning training. In order to avoid overfitting in the fine-tuning process, data enhancement, regularization technology and other methods are used to alleviate the overfitting problem.
[0042] This application Figure 4 In the specific embodiment shown, the article writing assistance module 400 is used to receive user writing requirements, parse the user writing requirements, and generate auxiliary articles. Through semantic recognition of the user input of the need for article writing, such as inputting "write an article about artificial intelligence", the system can generate a corresponding article based on the learned corpus. The above scheme can enable the system to support general article writing assistance functions. Users can input corresponding article writing requirements. The more detailed the generated article is, the more it meets the user's needs, which can help users assist in writing.
[0043] In the specific embodiment of the present application, Figure 5 As shown, the contextual connection module 500 is used to record the user's historical conversations. When the user brings up the current conversation, the historical conversation is called or the associative words extracted from the historical conversation are retrieved to generate a reply field to the user. The historical conversation includes the content of the user input that has been generated before the conversation, and the content of the artificial intelligence reply to the user. The system will store the historical conversation and parse the historical conversation in advance to obtain several segmented words in the historical conversation, and use the segmented results of the historical conversation as associative words. This enables a more accurate understanding of the semantics of the conversation currently proposed by the user when the user brings up the current conversation.
[0044] In the specific embodiment of the present application, Figure 6 As shown, the image and text generation module 600 is used to classify each paragraph of the generated text, crawl the image information of the corresponding category of each paragraph, insert the image information into the corresponding paragraph of the generated text, and generate image and text information. The fused model has the specific text generation capability. On the basis of the generated text, each paragraph is classified, and the category of the text is pre-set, such as finance, education, government, sports, etc., and the category described in each paragraph is determined. The pictures of the corresponding category are crawled by web crawler technology, and inserted into the paragraph, and finally the image and text information is generated.
[0045] In such Figure 7 In the illustrated embodiment, a transformer-based artificial intelligence system 7 is further disclosed, including the artificial intelligence model as described above. The artificial intelligence system 7 can integrate one or more of the text error correction module 300, the article writing auxiliary module 400, the context connection module 500, and the graphic generation module 600 to achieve the functions of the corresponding modules.
[0046] A transformer-based artificial intelligence device comprises a computer in which the above-mentioned artificial intelligence system runs, and the device also comprises an external input device.
[0047] This solution can generate a T5 model based on transformer to train a large amount of data, and integrate the GPT2 model for continuous conversation, which makes the system have stronger semantic understanding and expression capabilities. 2) The advantage of the model in this patent is that it uses a small model to achieve the function of assisting people in writing. It is suitable for multiple operating systems such as Windows, Linux, and trusted computing platforms. Ordinary CPU chips can perform inference calculations, which makes the system more scalable and universal. 3) The system has the ability of continuous conversation, that is, it can conduct multiple rounds of conversations, allowing users to discuss problems more deeply, improving the interactivity and practicality of the system. 4) This system also supports the diversity of natural language processing functions, supports part-of-speech analysis, entity recognition, relationship extraction, event extraction, keyword extraction, text error correction, summary generation and other natural language processing tasks, and can be applied to multiple application fields, such as intelligent customer service, search engines, intelligent assisted writing, machine translation, etc.
[0048] It should be noted that, although the above embodiments have been described in this article, the patent protection scope of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included in the patent protection scope of the present invention.
[0049] Those skilled in the art should understand that the above embodiments can be provided as methods, devices, or computer program products. These embodiments can be in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. All or part of the steps in the methods involved in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a storage medium readable by a computer device for executing all or part of the steps described in the above embodiments. The computer device includes, but is not limited to: personal computers, servers, general-purpose computers, special-purpose computers, network devices, embedded devices, programmable devices, smart mobile terminals, smart home devices, wearable smart devices, in-vehicle smart devices, etc.; the storage medium includes, but is not limited to: RAM, ROM, disk, tape, CD, flash memory, U disk, mobile hard disk, memory card, memory stick, network server storage, network cloud storage, etc.
[0050] The above embodiments are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a computer device to generate a machine, so that the instructions executed by the processor of the computer device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0051] These computer program instructions may also be stored in a computer device readable memory capable of directing a computer device to operate in a specific manner, so that the instructions stored in the computer device readable memory produce a manufactured product including an instruction device, the instruction device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0052] These computer program instructions can also be loaded onto a computer device so that a series of operating steps are executed on the computer device to produce a computer-implemented process, whereby the instructions executed on the computer device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0053] Although the above embodiments have been described, once those skilled in the art know the basic creative concepts, they can make additional changes and modifications to these embodiments. Therefore, the above description is only an embodiment of the present invention and does not limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A transformer-based artificial intelligence model, characterized in that: include: transformer decoder, the transformer decoder comprising a plurality of first processing layers and a plurality of second processing layers, the first processing layers comprising a multi-head self-attention layer and a first position-aware feedforward network layer; the second processing layers comprising a masked multi-head self-attention layer, a multi-head attention layer, and a second position-aware feedforward network layer, the output of the first processing layer being connected to the input of the multi-head attention layer; The output end of the transformer decoder is used to connect to the GPT2 generator, which includes a first normalization layer, a second multi-head self-attention layer, a fully connected layer, a linear layer and a softmax layer; the output of the GPT2 generator is a probability output.
2. The transformer-based artificial intelligence model according to claim 1, characterized in that: The GPT2 generator uses the RMSNorm normalization function and the activation function of the GPT2 generator is SwiGLU.
3. The transformer-based artificial intelligence model according to claim 1, characterized in that: It also includes a position encoder, which is arranged at the input end of the transformer decoder.
4. The transformer-based artificial intelligence model according to claim 1, characterized in that: It also includes a text correction module, an article writing assistance module, a context connection module, and a graphic and text generation module.
5. The transformer-based artificial intelligence model according to claim 4, characterized in that: The text error correction module is used to take the text error correction task prompt words and text error samples as input, and take the modified target samples of the correct text as output for fine-tuning training.
6. The transformer-based artificial intelligence model according to claim 4, characterized in that: The article writing auxiliary module is used to receive user writing requirements, analyze the user writing requirements, and generate auxiliary articles.
7. The transformer-based artificial intelligence model according to claim 4, characterized in that: The contextual connection module is used to record the user's historical conversations. When the user brings up the current conversation, the historical conversations are called or the associated words extracted from the historical conversations are retrieved to generate a reply field to the user.
8. The transformer-based artificial intelligence model according to claim 4, characterized in that: The image-text generation module is used to classify each paragraph of the generated text when generating text, crawl the picture information of the corresponding category of each paragraph, insert the picture information into the corresponding paragraph of the generated text, and generate image-text information.
9. A transformer-based artificial intelligence system, characterized in that: Comprising an artificial intelligence model as described in any one of claims 1-8.
10. A transformer-based artificial intelligence device, characterized in that: The device comprises a computer, wherein the artificial intelligence system as claimed in claim 9 is running on the computer, and the device further comprises an external input device.
