Text processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a text processing method and device, electronic equipment and a storage medium. The invention discloses a text processing method, which comprises the following steps of: splitting a trained large language model to obtain a plurality of fragment models; respectively converting each fragment model to obtain a plurality of quantum neural network (QNN) models; and sending the QNN models to a terminal device, so that the terminal device obtains a text processing result based on the to-be-processed text and each QNN model through a neural network processor NPU. Therefore, the large-scale language model is sliced and converted, so that the terminal equipment can load and run the sliced and converted QNN models through the NPU to realize text processing, and the large-scale language model of a remote server does not need to be called for text processing; the problem that a large language model cannot be called without a network is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a text processing method, device, electronic device and storage medium. Background Art

[0002] In text processing application scenarios, text processing is usually performed based on a trained large language model (LLM). LLM is a deep learning model with a large number of parameters. The LLM model is designed to understand and generate human language and can perform tasks such as text summarization, translation, and text analysis.

[0003] Since the large language model is relatively large, in the prior art, the large language model is usually deployed in a remote server first. Then, the terminal device (such as a mobile phone) calls the large language model in the remote server to process the text to be processed.

[0004] However, using this method, if there is no network, the terminal device cannot call the large language model at any time. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for text processing, so as to solve the problem that large language models cannot be called at any time when performing text processing.

[0006] On the one hand, an embodiment of the present application provides a text processing method, which is applied to a terminal device, including:

[0007] Receive multiple quantum neural network QNN models sent by the server; each QNN model is obtained by splitting the trained large language model into multiple shard models and converting each shard model separately;

[0008] Through the neural network processor NPU, the text processing results are obtained based on the text to be processed and each QNN model.

[0009] In one implementation, the large language model includes a prefix model and a cache model, wherein the prefix model and the cache model are obtained by splitting multiple transformation modules in the large language model; the prefix model is used to process each word segmentation of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series; based on the text to be processed and each QNN model, a text processing result is obtained, including:

[0010] Based on the text to be processed and the QNN model corresponding to the prefix model, the initial text output is obtained;

[0011] Based on the initial text output and the QNN model corresponding to the cache model, the text processing result is obtained.

[0012] In one implementation, obtaining an initial text output based on the text to be processed and the QNN model corresponding to the prefix model includes:

[0013] If the text length of the text to be processed is not greater than the first set length, data padding is performed on the text to be processed to obtain a first padding text;

[0014] Generate a first attention mechanism mask according to the filling data in the first filling text;

[0015] The first filled text and the first attention mechanism mask are input into the QNN model corresponding to the prefix model to obtain the initial text output.

[0016] In one embodiment, the prefix model and the cache model both include multiple transformation modules; the initial text output includes a first text linear output and first module query information and its corresponding first module output value; the first module query information and its corresponding first module output value are output by the transformation module in the prefix model;

[0017] The text processing result includes the second text linear output and the second module query information and its corresponding second module output value; the second module query information and its corresponding second module output value are output by the transformation module in the cache model.

[0018] In one implementation, obtaining a text processing result based on the initial text output and the QNN model corresponding to the cache model includes:

[0019] If the text length of the first text linear output is not greater than the second set length, padding the first text linear output to obtain a second padding text;

[0020] Generate a second attention mechanism mask according to the filling data in the second filling text;

[0021] The second filling text, the second attention mechanism mask, and each first module query information and its corresponding first module output value are input into the QNN model corresponding to the cache model to obtain the text processing result.

[0022] In one embodiment, the method further comprises:

[0023] During the operation of each QNN model, if there is a pending value to be normalized, the pending value is scaled to obtain a scaled value;

[0024] The scaled value is normalized to obtain a normalized result.

[0025] On the one hand, an embodiment of the present application provides a text processing method, which is applied to a server and includes:

[0026] Split the trained large language model to obtain multiple shard models;

[0027] Convert each shard model separately to obtain multiple quantum neural network QNN models;

[0028] The QNN model is sent to the terminal device, so that the terminal device obtains the text processing result based on the text to be processed and each QNN model through the neural network processor NPU.

[0029] In one implementation, the trained large language model is split to obtain multiple fragment models, including:

[0030] Each transformation module in the large language model is split to obtain a prefix model and a cache model containing multiple transformation modules; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series;

[0031] The prefix model and the cache model are split respectively to obtain multiple shard models of the prefix model and multiple shard models of the cache model.

[0032] In one implementation, each slice model is converted separately to obtain multiple QNN models, including:

[0033] Convert the formats of each shard model to obtain the Open Neural Network Exchange ONNX format model corresponding to each shard model;

[0034] Convert each ONNX format model separately to obtain multiple QNN models.

[0035] In one embodiment, the method further comprises:

[0036] Compile the deep learning forward prediction framework MACE;

[0037] Through the compilation results, each ONNX format model is converted separately to obtain multiple MACE models;

[0038] Each MACE model is sent to the terminal device, so that the terminal device runs each QNN model based on each MACE model.

[0039] In one implementation, the number of shards of the shard model is determined based on the size of the large language model and the available memory space of the terminal device.

[0040] On the one hand, an embodiment of the present application provides a text processing device, which is applied to a terminal device, including:

[0041] A receiving unit is used to receive multiple quantum neural network QNN models sent by a server; each QNN model is obtained by the server splitting a trained large language model into multiple shard models and converting each shard model separately;

[0042] The obtaining unit is used to obtain the text processing result based on the text to be processed and each QNN model through the neural network processor NPU.

[0043] In one implementation, the large language model includes a prefix model and a cache model; the prefix model and the cache model are obtained by splitting multiple transformation modules in the large language model; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series; the obtaining unit is used to:

[0044] Based on the text to be processed and the QNN model corresponding to the prefix model, the initial text output is obtained; based on the initial text output and the QNN model corresponding to the cache model, the text processing result is obtained.

[0045] In one embodiment, the obtaining unit is used to:

[0046] If the text length of the text to be processed is not greater than the first set length, data padding is performed on the text to be processed to obtain a first padding text;

[0047] Generate a first attention mechanism mask according to the filling data in the first filling text;

[0048] The first filled text and the first attention mechanism mask are input into the QNN model corresponding to the prefix model to obtain the initial text output.

[0049] In one embodiment, the prefix model and the cache model both include multiple transformation modules; the initial text output includes a first text linear output and first module query information and its corresponding first module output value; the first module query information and its corresponding first module output value are output by the transformation module in the prefix model;

[0050] The text processing result includes the second text linear output and the second module query information and its corresponding second module output value; the second module query information and its corresponding second module output value are output by the transformation module in the cache model.

[0051] In one embodiment, the obtaining unit is used to:

[0052] If the text length of the first text linear output is not greater than the second set length, padding the first text linear output to obtain a second padding text;

[0053] Generate a second attention mechanism mask according to the filling data in the second filling text;

[0054] The second filling text, the second attention mechanism mask, and each first module query information and its corresponding first module output value are input into the QNN model corresponding to the cache model to obtain the text processing result.

[0055] In one implementation, the obtaining unit is further configured to:

[0056] During the operation of each QNN model, if there is a pending value to be normalized, the pending value is scaled to obtain a scaled value;

[0057] The scaled value is normalized to obtain a normalized result.

[0058] On the one hand, an embodiment of the present application provides a text processing device, which is applied to a server and includes:

[0059] A sharding unit, used to split the trained large language model to obtain multiple sharding models;

[0060] A conversion unit, used to convert each slice model separately to obtain multiple quantum neural network QNN models;

[0061] The sending unit is used to send the QNN model to the terminal device, so that the terminal device obtains the text processing result through the neural network processor NPU based on the text to be processed and each QNN model.

[0062] In one implementation, the sharding unit is used to:

[0063] Each transformation module in the large language model is split to obtain a prefix model and a cache model containing multiple transformation modules; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series;

[0064] The prefix model and the cache model are split respectively to obtain multiple shard models of the prefix model and multiple shard models of the cache model.

[0065] In one embodiment, the conversion unit is used to:

[0066] Convert the formats of each shard model to obtain the Open Neural Network Exchange ONNX format model corresponding to each shard model;

[0067] Convert each ONNX format model separately to obtain multiple QNN models.

[0068] In one implementation, the conversion unit is further configured to:

[0069] Compile the deep learning forward prediction framework MACE;

[0070] Through the compilation results, each ONNX format model is converted separately to obtain multiple MACE models;

[0071] Each MACE model is sent to the terminal device, so that the terminal device runs each QNN model based on each MACE model.

[0072] In one implementation, the number of shards of the shard model is determined based on the size of the large language model and the available memory space of the terminal device.

[0073] On the one hand, an embodiment of the present application provides an electronic device, including:

[0074] Processor; and

[0075] A memory stores computer instructions, wherein the computer instructions are used to cause a processor to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned text processing.

[0076] On the one hand, an embodiment of the present application provides a storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned text processing.

[0077] The text processing method in the embodiment of the present application includes splitting the trained large language model to obtain multiple slice models; converting each slice model separately to obtain multiple quantum neural network QNN models; sending the QNN model to the terminal device, so that the terminal device obtains the text processing result based on the text to be processed and each QNN model through the neural network processor NPU. In this way, the large language model is sliced ​​and converted, so that the terminal device can load and run the sliced ​​and converted QNN models through the NPU to realize text processing, without calling the large language model of the remote server for text processing, which solves the problem that the large language model cannot be called without a network. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0079] Figure 1 It is a schematic diagram of the architecture of a text processing system in an embodiment of the present application.

[0080] Figure 2 It is a flowchart of a text processing method in an embodiment of the present application.

[0081] Figure 3 This is an example diagram of a prefix model in an embodiment of the present application.

[0082] Figure 4 This is an example diagram of a cache model in an embodiment of the present application.

[0083] Figure 5 It is a structural block diagram of a text processing device in an embodiment of the present application.

[0084] Figure 6 It is a structural block diagram of another text processing device in an embodiment of the present application.

[0085] Figure 7 It is a structural schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0086] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described implementation methods are part of the implementation methods of the present application, rather than all of the implementation methods. Based on the implementation methods in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, the technical features involved in the different implementation methods of the present application described below can be combined with each other as long as they do not conflict with each other.

[0087] First, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0088] Terminal device: can be a mobile terminal, a fixed terminal or a portable terminal, such as a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system device, a personal navigation device, a personal digital assistant, an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the terminal device can support any type of interface for the user (such as a wearable device), etc.

[0089] Server: It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0090] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0091] The technical concept of the embodiments of the present application is described below.

[0092] Traditionally, when processing text using large language models, one of the following methods is usually used:

[0093] Method 1: First deploy the large language model in the remote server, and then the terminal device calls the large language model in the remote server for text processing.

[0094] However, with this approach, if there is no network, large language models cannot be called at any time, the cost of operating and maintaining remote servers is high, and it is difficult to ensure the security of user data.

[0095] Method 2: Deploy a large language model in the central processing unit (CPU) or graphics processing unit (GPU) of the terminal device, and perform text processing through the large language model in the terminal device.

[0096] However, using this method will occupy a large amount of CPU or GPU resources in the terminal device, affect the use of other applications in the terminal device, and generally make it difficult to fully utilize the AI ​​computing power in the terminal device.

[0097] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a method, device, electronic device and storage medium for text processing, aiming to solve the problem that large language models cannot be called at any time when performing text processing.

[0098] See also Figure 1FIG. 1 is a schematic diagram of the structure of a text processing system. The system includes a server and a terminal device.

[0099] The server is used to split the trained large language model to obtain multiple shard models; convert each shard model separately to obtain multiple quantum neural network (QNN) models; send the QNN model to the terminal device; each shard model contains multiple network layers of the large language model.

[0100] The terminal device is used to obtain text processing results based on the text to be processed and each QNN model through a neural network processor (Neural network Processing Unit, NPU).

[0101] In one implementation, the number of shards of the shard model is determined based on the size of the large language model and the available memory space of the terminal device.

[0102] In this way, by sharding the model, the memory limit problem that exists when the terminal device directly loads a large model is avoided. By loading and running each QNN model through the NPU, the powerful AI computing power of the NPU can be fully utilized without occupying the CPU or GPU resources of the terminal device, avoiding the problem of device heating when the GPU runs large models.

[0103] In one embodiment, the server is used to: split each transformation module in a large language model to obtain a prefix model and a cache model containing multiple transformation modules; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series; the prefix model and the cache model are split respectively to obtain multiple shard models of the prefix model and multiple shard models of the cache model.

[0104] As an example, the text to be processed is "How is the weather today?", the prefix model first segments "How is the weather today?" to obtain the segmented words: "today", "weather", "how", "like", and then processes each segmented word in parallel to obtain the output value corresponding to each segmented word (such as the state vector of the segmented word); the cache model processes each output value of the prefix model in turn (i.e., serial processing) to obtain the text processing result.

[0105] If a large language model is used directly for text processing, the large language model needs to perform complete process processing for each word of the text to be processed in turn, which will consume a lot of time. In the embodiment of the present application, the model part for parallel processing is split into a prefix model, and text processing is performed through the prefix model and the cache model respectively, thereby improving processing efficiency.

[0106] In one implementation, the server is used to: convert the format of each shard model to obtain an Open Neural Network Exchange (ONNX) format model corresponding to each shard model; and convert each ONNX format model to obtain multiple QNN models.

[0107] In this way, each shard model can be exported by ONNX format conversion.

[0108] In one implementation, the server is further used to: compile a deep learning forward prediction framework (Mobile AI Compute Engine, MACE); convert each ONNX format model respectively according to the compilation result to obtain multiple MACE models; and send each MACE model to a terminal device;

[0109] The terminal device is used to run each QNN model based on each MACE model.

[0110] In this way, the QNN model can be run in the terminal device relying on the MACE framework.

[0111] In one implementation, the terminal device is used to: obtain an initial text output based on the text to be processed and a QNN model corresponding to a prefix model; and obtain a text processing result based on the initial text output and a QNN model corresponding to a cache model.

[0112] In one implementation, the terminal device is used to:

[0113] If the text length of the text to be processed is not greater than the first set length, data padding is performed on the text to be processed to obtain a first padding text;

[0114] Generate a first attention mechanism mask according to the filling data in the first filling text;

[0115] The first filled text and the first attention mechanism mask are input into the QNN model corresponding to the prefix model to obtain the initial text output.

[0116] In one embodiment, the prefix model and the cache model both include multiple transformation modules; the initial text output includes a first text linear output and first module query information and its corresponding first module output value; the first module query information and its corresponding first module output value are output by the transformation module in the prefix model;

[0117] The text processing result includes the second text linear output and the second module query information and its corresponding second module output value; the second module query information and its corresponding second module output value are output by the transformation module in the cache model.

[0118] In one implementation, the terminal device is used to:

[0119] If the text length of the first text linear output is not greater than the second set length, padding the first text linear output to obtain a second padding text;

[0120] Generate a second attention mechanism mask according to the filling data in the second filling text;

[0121] The second filling text, the second attention mechanism mask, and each first module query information and its corresponding first module output value are input into the QNN model corresponding to the cache model to obtain the text processing result.

[0122] In this way, efficient decoding is achieved under fixed input length constraints, improving text processing efficiency.

[0123] In one implementation, the terminal device is used to:

[0124] During the operation of each QNN model, if there are pending values ​​to be normalized, the pending values ​​are scaled to obtain scaled values; and the scaled values ​​are normalized to obtain normalized results.

[0125] In this way, the values ​​to be processed are equivalently scaled to reduce the probability of operator overflow, thereby ensuring the availability and correctness of large language models in terminal devices.

[0126] See also Figure 2 , which is a flowchart of a text processing method in an embodiment of the present application, is applied to Figure 1 The text processing system in Figure 1 and Figure 2 The method is described below. The specific implementation process of the method is as follows:

[0127] Step 200: The server splits the trained large language model to obtain multiple shard models.

[0128] In one implementation, when executing step 200, the following steps may be used:

[0129] S2001: Split the large language model to obtain a prefix model and a cache model.

[0130] In one implementation, each transformation module in a large language model is split to obtain a prefix model and a cache model containing multiple transformation modules; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series. Both the prefix model and the cache model include multiple transformation modules.

[0131] S2002: Split the prefix model and the cache model respectively to obtain multiple shard models of the prefix model and multiple shard models of the cache model.

[0132] In one implementation, based on the network layer, the prefix model and the cache model are split separately to obtain a plurality of shard models, each of which includes a plurality of network layers.

[0133] The number of shards of the sharded model may be determined based on the size of the large language model and the available memory space of the terminal device.

[0134] In one implementation, a large language model includes an input layer, N (e.g., N is 10) transformer blocks, and an output layer; the large language model is divided into M (e.g., M is 10) slice models; wherein the input layer and the output layer are sliced ​​into separate slice models, and the N transformer blocks are evenly divided into K (e.g., K is 8) slice models. Wherein, N, M, and K are all positive integers.

[0135] Furthermore, the input and output of each shard model are determined according to the logical order of each shard model, so that the text processing result after the shard models are connected in series is consistent with the text processing result of the unsegmented large language model.

[0136] In the embodiment of the present application, a large language model is split up using the available memory space of a terminal device, thereby solving the problem that a terminal device has difficulty loading a large model.

[0137] Step 201: The server converts each slice model separately to obtain multiple QNN models.

[0138] In one implementation, when executing step 201, the following steps may be used:

[0139] S2011: Convert the formats of the slice models to obtain ONNX format model files corresponding to the slice models.

[0140] Among them, ONNX is an open format used to represent deep learning models. The so-called openness means that ONNX defines a set of standard formats that are independent of the environment and platform to enhance the interactivity of various AI models.

[0141] In this way, each shard model can be exported as a model file in ONNX format to facilitate subsequent device transfer of the model.

[0142] S2012: Convert each ONNX format model separately to obtain multiple QNN models.

[0143] In one implementation, each ONNX format model is converted separately through a QNN software development kit (SDK) to obtain multiple QNN models.

[0144] In this way, a QNN model that can be directly run on the terminal device can be obtained.

[0145] Step 202: The server sends the QNN model to the terminal device.

[0146] Furthermore, since the operation of the QNN model usually needs to rely on MACE, when executing step 202, the following steps may also be adopted:

[0147] S2021: Compile MACE; based on the compilation results, convert each ONNX format model separately to obtain multiple MACE models.

[0148] S2022: Send each MACE model to the terminal device.

[0149] Step 203: The terminal device obtains a text processing result through the NPU based on the text to be processed and each QNN model.

[0150] Specifically, the terminal device receives multiple QNN models sent by the server, and runs each QNN model based on each MACE model through the NPU.

[0151] Among them, each QNN model is obtained by the server splitting the trained large language model into multiple shard models and converting each shard model separately.

[0152] In an embodiment of the present application, each QNN model can be deployed to the NPU of the terminal device, and then each QNN model can be run based on each MACE model through the NPU, thereby not occupying CPU or GPU resources, reducing the impact of occupying CPU resources on the system application of the terminal device, avoiding the problem of GPU heating, and making full use of the AI ​​computing power of the terminal device. While ensuring sufficient processing efficiency, it avoids the problem of excessive occupation of system resources of the terminal device.

[0153] It should be noted that each QNN model usually only supports fixed-length input and output. In order to make text processing more reasonable and efficient and reduce the computational consumption of text processing, when executing step 200, the large language model is first split into a prefix model and a cache model using a type-based export method based on the computational characteristics of different stages in the large language model; in the text processing process of step 203, the text processing can be performed based on the QNN model corresponding to the prefix model to obtain the initial text output, and the QNN model corresponding to the cache model can be used to perform subsequent processing based on the initial text output.

[0154] To this end, in one implementation, when executing step 203, the following steps may be adopted:

[0155] S2031: Obtaining initial text output based on the text to be processed and the QNN model corresponding to the prefix model.

[0156] In one implementation, when executing S2031, the following steps may be adopted:

[0157] S2031-1: If the text length of the text to be processed is not greater than a first set length, data padding is performed on the text to be processed to obtain a first padding text.

[0158] In one implementation, the user inputs the text to be processed according to an input prompt (e.g., write a poem). The terminal device encodes the text to be processed through a word segmenter and a word list and converts it into a word segmentation identifier (token id). As an example, the text to be processed is "How is the weather today?"

[0159] According to the word segmentation identifier, the text length corresponding to the text to be processed is determined. If the text length is not longer than a first set length, the position that does not reach the first set length is filled with specified data.

[0160] In actual applications, the designated data can be set according to the actual application scenario, for example, it can be 0, and no restriction is imposed. The first set length can also be set according to the actual application scenario, and no restriction is imposed here.

[0161] It should be noted that the first set length is a fixed input length set for the QNN model. If the text length of the text to be processed is the first set length, the padding data is empty, and the text to be processed is the first padding text; if the text length of the text to be processed is less than the first set length, the specified data is padded at the position that does not reach the first set length. Furthermore, if the first set length is higher than the first set length, it means that the text to be processed cannot be processed, and the text to be processed is discarded.

[0162] S2031-2: Generate a first attention mechanism mask based on the filling data in the first filling text.

[0163] In one implementation, the length of the first attention mechanism mask (attention_mask) is set to a first set length, and the data at each position of the first attention mechanism mask is corresponded one-to-one with the data at each position of the first filling text. For each position in the first filling text, the following steps are performed: if a position in the first filling text is not filled (that is, the data at this position is the data in the text to be processed rather than the filling data), then the data at the corresponding position of the first attention mechanism mask is 1; if the data at this position is the filling data, then the data at the corresponding position of the first attention mechanism mask is 0.

[0164] S2031-3: Input the first filling text and the first attention mechanism mask into the QNN model corresponding to the prefix model to obtain the initial text output.

[0165] Among them, the initial text output includes the first text linear output and the first module query information and its corresponding first module output value; the first module query information and its corresponding first module output value are output by the transformation module in the prefix model.

[0166] See also Figure 3 The following is an example diagram of a prefix model. Figure 3 In the method, the prefix model includes a first input layer, multiple transformation modules and a first output layer.

[0167] Among them, the first input layer is Embedding, and the first output layer is a fully connected layer (linear). The input of the first input layer is the first filled text inputs1 and the first attention mechanism mask attention_mask1. The first output layer outputs the first text linear output outputs1, and each transformation module outputs the corresponding first module query information and the first module output value, including: presents query11 and presents value11, presents query12 and presents value12, presents query13 and presents value13.

[0168] S2032: Obtain a text processing result based on the initial text output and the QNN model corresponding to the cache model.

[0169] In one implementation, when executing S2032, the following steps may be adopted:

[0170] S2032-1: If the text length of the first text linear output is not greater than the second set length, padding is performed on the first text linear output to obtain a second padding text.

[0171] In actual applications, the second set length can be set according to the actual application scenario, for example, it can be 1, and there is no limitation here.

[0172] Specifically, the first text linear output may be filled based on the same principle as that of filling the text to be processed, which will not be described in detail here.

[0173] S2032-2: Generate a second attention mechanism mask based on the filling data in the second filling text.

[0174] Specifically, the second attention mechanism mask can be generated based on the same principle as that of generating the first attention mechanism mask, which will not be elaborated here.

[0175] S2032-3: Input the second filling text, the second attention mechanism mask, and each first module query information and its corresponding first module output value into the QNN model corresponding to the cache model to obtain the text processing result.

[0176] The text processing result includes the second text linear output and the second module query information and its corresponding second module output value; the second module query information and its corresponding second module output value are output by the transformation module in the cache model.

[0177] See also Figure 4 The figure shows an example diagram of a cache model. Figure 4 In the embodiment, the cache model includes a second input layer, a plurality of transformation modules and a second output layer.

[0178] Among them, the second input layer is Embedding, and the second output layer is linear. The input of the second input layer is the second filling text inputs2 and the second attention mechanism mask attention_mask2. The second text linear output outputs2 is output through the second output layer, and the corresponding second module query information and second module output value are output through each transformation module, including: presents query21 and presents value21, presents query22 and presentsvalue22, presents query23 and presents value23.

[0179] Furthermore, it is also possible to determine whether the second text linear output is an end token. If it is determined that the second text linear output is not ended, the following steps are executed in a loop until it is determined that the second text linear output is an end token:

[0180] The obtained second text linear output, second module query information and its corresponding second module output value are respectively used as inputs of each QNN model corresponding to the cache model, so as to obtain a new second text linear output and second module query information and its corresponding second module output value.

[0181] Furthermore, since the calculation accuracy of the terminal device is usually Fp16, this will cause overflow in the calculation of some operators used for normalization in the QNN model. For this reason, in the embodiment of the present application, an equivalent replacement method is adopted to equivalently scale the processed values ​​to reduce the probability of operator overflow, thereby ensuring the availability and correctness of large language models in terminal devices.

[0182] In one implementation, during the operation of each QNN model, if there is a processed value to be normalized, the processed value is scaled to obtain a scaled value; and the scaled value is normalized to obtain a normalized result. It should be noted that the processed value x is an array, including multiple elements x i , i is the sequence number of each element in the value to be processed.

[0183] In the traditional way, when normalizing the value x to be processed, the following formula is usually used:

[0184]

[0185] Among them, F norm is the normalized result, x is the value to be processed, and n is the number of values ​​to be processed.

[0186] In the embodiment of the present application, considering that directly performing a quadratic operation on x will generate a large amount of Fp16 overflow, each x is first scaled according to the scaling factor d (for example, d is 50) to obtain a scaled value y=x / d, and then the scaled value y is normalized to obtain a normalized result. Specifically, the following formula can be used:

[0187]

[0188] In this way, the probability of overflow when performing quadratic operations will be greatly reduced, and F norm consistency of results.

[0189] In an embodiment of the present application, each QNN model generated after slicing and conversion of a large language model is sent to a terminal device, so that it can be deployed in the terminal device without being deployed in a remote server, saving the operation and maintenance costs of the remote server, reducing dependence on the remote server, and performing text processing without a network. Furthermore, the terminal device can load and run each QNN model through the NPU, and can make full use of the powerful AI computing power of the NPU without occupying the CPU or GPU resources of the terminal device, thereby avoiding the problem of device heating when the GPU runs a large model; further, by exporting the model in slices, the problem of memory overlimit when the terminal device directly loads a large model is avoided, efficient decoding under a fixed input length limit is achieved, and text processing efficiency is improved.

[0190] Based on the same inventive concept, a text processing device is also provided in the embodiment of the present application. Since the principle of solving the problem by the above device and equipment is similar to that of a text processing method, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present application does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as terminal devices and servers, etc., and the present application will not repeat them.

[0191] See also Figure 5 , which is a structural block diagram of another apparatus for Chinese text processing implemented in this application. In some embodiments, the apparatus for text processing exemplified in this application includes:

[0192] The receiving unit 501 is used to receive multiple quantum neural network QNN models sent by the server; each QNN model is obtained by the server splitting the trained large language model into multiple fragment models and converting each fragment model respectively;

[0193] The obtaining unit 502 is used to obtain the text processing result based on the text to be processed and each QNN model through the neural network processor NPU.

[0194] In one implementation, the large language model includes a prefix model and a cache model; the prefix model and the cache model are obtained by splitting multiple transformation modules in the large language model; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series; the obtaining unit 502 is used to:

[0195] Based on the text to be processed and the QNN model corresponding to the prefix model, the initial text output is obtained;

[0196] Based on the initial text output and the QNN model corresponding to the cache model, the text processing result is obtained.

[0197] In one implementation, the obtaining unit 502 is used to:

[0198] If the text length of the text to be processed is not greater than the first set length, data padding is performed on the text to be processed to obtain a first padding text;

[0199] Generate a first attention mechanism mask according to the filling data in the first filling text;

[0200] The first filled text and the first attention mechanism mask are input into the QNN model corresponding to the prefix model to obtain the initial text output.

[0201] In one embodiment, the prefix model and the cache model both include multiple transformation modules; the initial text output includes a first text linear output and first module query information and its corresponding first module output value; the first module query information and its corresponding first module output value are output by the transformation module in the prefix model;

[0202] The text processing result includes the second text linear output and the second module query information and its corresponding second module output value; the second module query information and its corresponding second module output value are output by the transformation module in the cache model.

[0203] In one implementation, the obtaining unit 502 is used to:

[0204] If the text length of the first text linear output is not greater than the second set length, padding the first text linear output to obtain a second padding text;

[0205] Generate a second attention mechanism mask according to the filling data in the second filling text;

[0206] The second filling text, the second attention mechanism mask, and each first module query information and its corresponding first module output value are input into the QNN model corresponding to the cache model to obtain the text processing result.

[0207] In one implementation, the obtaining unit 502 is further configured to:

[0208] During the operation of each QNN model, if there is a pending value to be normalized, the pending value is scaled to obtain a scaled value;

[0209] The scaled value is normalized to obtain a normalized result.

[0210] See also Figure 6 , which is a structural block diagram of another apparatus for Chinese text processing in an embodiment of the present application. In some embodiments, the apparatus for text processing in the example of the present application includes:

[0211] A sharding unit 601 is used to split the trained large language model to obtain multiple sharding models;

[0212] A conversion unit 602 is used to convert each slice model separately to obtain multiple quantum neural network QNN models;

[0213] The sending unit 603 is used to send the QNN model to the terminal device, so that the terminal device obtains the text processing result through the neural network processor NPU based on the text to be processed and each QNN model.

[0214] In one implementation, the slicing unit 601 is used to:

[0215] Each transformation module in the large language model is split to obtain a prefix model and a cache model containing multiple transformation modules; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series;

[0216] The prefix model and the cache model are split respectively to obtain multiple shard models of the prefix model and multiple shard models of the cache model.

[0217] In one implementation, the conversion unit 602 is used to:

[0218] Convert the formats of each shard model to obtain the Open Neural Network Exchange ONNX format model corresponding to each shard model;

[0219] Convert each ONNX format model separately to obtain multiple QNN models.

[0220] In one implementation, the conversion unit 602 is further configured to:

[0221] Compile the deep learning forward prediction framework MACE;

[0222] Through the compilation results, each ONNX format model is converted separately to obtain multiple MACE models;

[0223] Each MACE model is sent to the terminal device, so that the terminal device runs each QNN model based on each MACE model.

[0224] In one implementation, the number of shards of the shard model is determined based on the size of the large language model and the available memory space of the terminal device.

[0225] In an embodiment of the present application, an electronic device is provided, including:

[0226] Processor; and

[0227] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0228] In an embodiment of the present application, a storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments. Figure 7 FIG. 7 is a schematic diagram showing the structure of an electronic device 7000. Figure 7 As shown, the electronic device 7000 includes: a processor 7010 and a memory 7020 , and optionally, may also include a power supply 7030 , a display unit 7040 , and an input unit 7050 .

[0229] The processor 7010 is the control center of the electronic device 7000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 7000 by running or executing software programs and / or data stored in the memory 7020, thereby monitoring the electronic device 7000 as a whole.

[0230] In the embodiment of the present application, the processor 7010 executes the various steps in the above embodiment when calling the computer program stored in the memory 7020.

[0231] Optionally, the processor 7010 may include one or more processing units; preferably, the processor 7010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 7010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0232] The memory 7020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 7000, etc. In addition, the memory 7020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0233] The electronic device 7000 also includes a power source 7030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 7010 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0234] The display unit 7040 may be used to display information input by a user or information provided to a user, various menus of the electronic device 7000, and in the embodiment of the present application, is mainly used to display the display interface of each application in the electronic device 7000 and objects such as text and pictures displayed in the display interface. The display unit 7040 may include a display panel 7041. The display panel 7041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), and the like.

[0235] The input unit 7050 may be used to receive information such as numbers or characters input by the user. The input unit 7050 may include a touch panel 7051 and other input devices 7052. The touch panel 7051, also known as a touch screen, may collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 7051).

[0236] Specifically, the touch panel 7051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 7010, and receive and execute the command sent by the processor 7010. In addition, the touch panel 7051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 7052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0237] Of course, the touch panel 7051 can cover the display panel 7041. When the touch panel 7051 detects a touch operation on or near it, it is transmitted to the processor 7010 to determine the type of touch event. Then the processor 7010 provides corresponding visual output on the display panel 7041 according to the type of touch event. Figure 7 In the embodiment, the touch panel 7051 and the display panel 7041 are used as two independent components to realize the input and output functions of the electronic device 7000, but in some embodiments, the touch panel 7051 and the display panel 7041 can be integrated to realize the input and output functions of the electronic device 7000.

[0238] The electronic device 7000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 7000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present application, Figure 7It is not shown and will not be described in detail.

[0239] Those skilled in the art will understand that Figure 7 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0240] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0241] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the embodiments. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A method for text processing, characterized in that: Applied to terminal equipment, including: Receiving multiple quantum neural network (QNN) models sent by a server; each QNN model is obtained by the server splitting a trained large language model into multiple slice models and converting each slice model respectively; each slice model includes multiple network layers of the large language model; Through the neural network processor NPU, the text processing results are obtained based on the text to be processed and each QNN model.

2. The method according to claim 1, characterized in that The large language model includes a prefix model and a cache model; the prefix model and the cache model are obtained by splitting a plurality of transformation modules in the large language model; the prefix model is used for parallel processing of each word segment of the text to be processed; The cache model is used to serially process multiple outputs of the prefix model; The text processing result is obtained based on the text to be processed and each QNN model, including: Obtaining an initial text output based on the text to be processed and the QNN model corresponding to the prefix model; The text processing result is obtained based on the initial text output and the QNN model corresponding to the cache model.

3. The method according to claim 2, characterized in that The obtaining of the initial text output based on the text to be processed and the QNN model corresponding to the prefix model includes: If the text length of the to-be-processed text is not greater than a first set length, data padding is performed on the to-be-processed text to obtain a first padded text; Generate a first attention mechanism mask according to the filling data in the first filling text; The first filling text and the first attention mechanism mask are input into the QNN model corresponding to the prefix model to obtain the initial output of the text.

4. The method according to claim 3, characterized in that The text initial output includes a first text linear output and first module query information and its corresponding first module output value; the first module query information and its corresponding first module output value are output by a transformation module in the prefix model; The text processing result includes a second text linear output and second module query information and its corresponding second module output value; The second module query information and its corresponding second module output value are output by the transformation module in the cache model.

5. The method according to claim 4, characterized in that The obtaining of the text processing result based on the initial text output and the QNN model corresponding to the cache model includes: If the text length of the first text linear output is not greater than a second set length, padding the first text linear output to obtain a second padding text; Generate a second attention mechanism mask according to the filling data in the second filling text; The second filling text, the second attention mechanism mask, and each first module query information and its corresponding first module output value are input into the QNN model corresponding to the cache model to obtain the text processing result.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: During the operation of each QNN model, if there is a processed value to be normalized, the processed value is scaled to obtain a scaled value; The scaling value is normalized to obtain a normalized result.

7. A method for text processing, characterized in that: Applied to a server, the method comprises: Splitting the trained large language model to obtain multiple shard models; each shard model includes multiple network layers of the large language model; Convert each shard model separately to obtain multiple quantum neural network QNN models; The QNN model is sent to the terminal device, so that the terminal device obtains the text processing result based on the text to be processed and each QNN model through the neural network processor NPU.

8. The method according to claim 7, characterized in that The trained large language model is split to obtain multiple fragment models, including: Each transformation module in the large language model is split to obtain a prefix model and a cache model containing multiple transformation modules; the prefix model is used to process each word segment of the text to be processed in parallel; the cache model is used to process multiple outputs of the prefix model in series; The prefix model and the cache model are split respectively to obtain multiple shard models of the prefix model and multiple shard models of the cache model.

9. The method according to claim 7, characterized in that: The slice models are converted separately to obtain multiple QNN models, including: Convert the formats of each shard model to obtain the Open Neural Network Exchange ONNX format model corresponding to each shard model; Convert each ONNX format model separately to obtain multiple QNN models.

10. The method according to claim 9, characterized in that The method further comprises: Compile the deep learning forward prediction framework MACE; Through the compilation results, each ONNX format model is converted separately to obtain multiple MACE models; Each MACE model is sent to the terminal device, so that the terminal device runs each QNN model based on each MACE model.

11. The method according to any one of claims 7 to 10, characterized in that: The number of shards of the shard model is determined according to the size of the large language model and the available memory space of the terminal device.

12. A text processing device, characterized in that: Applied to a terminal device, the device comprises: A receiving unit, used for receiving multiple quantum neural network QNN models sent by a server; each QNN model is obtained by the server splitting a trained large language model into multiple shard models and converting each shard model respectively; each shard model includes multiple network layers of the large language model; The obtaining unit is used to obtain the text processing result based on the text to be processed and each QNN model through the neural network processor NPU.

13. A text processing device, characterized in that: Applied to a server, the device comprises: A sharding unit, used for splitting the trained large language model to obtain multiple sharding models; each sharding model includes multiple network layers of the large language model; A conversion unit, used to convert each slice model separately to obtain multiple quantum neural network QNN models; The sending unit is used to send the QNN model to the terminal device, so that the terminal device obtains the text processing result based on the text to be processed and each QNN model through the neural network processor NPU.

14. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1-11.

15. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1-11.

Images