Methods for training large generative language models and for image processing tasks

Abstract

The present disclosure provides a method for training a generative large language model and for processing image tasks, which relates to the fields of artificial intelligence technology such as computer vision, image processing, and large models, and can be applied to scenarios such as optical character recognition. The method includes: obtaining a first generative large language model provided with a self-attention network layer and a feedforward neural network layer obtained by training using a general text training set; adding a cross-attention network layer between the self-attention network layer and the feedforward neural network layer whose parameters are frozen, and initializing the parameters of the cross-attention network layer to obtain a second generative large language model; using the image features of the acquired sample image and the prompt words used to indicate the task type as actual inputs, and using the text sequence matching the actual inputs as the expected outputs, and training the second generative large language model in a fine-tuning manner to obtain a third generative large language model.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as computer vision, image processing, and large models, and can be applied to scenarios such as optical character recognition, and in particular to a method for training a generative large language model, a method for processing image tasks, and corresponding devices, electronic devices, computer-readable storage media, and computer program products. Background Art

[0002] Optical Character Recognition (OCR) technology is widely used in various industries in society, such as education, medical care, finance, etc. Applications such as card and ticket recognition, automatic document entry, and photo search have greatly improved the intelligence and production efficiency of traditional industries and facilitated people's daily learning and life.

[0003] The Large Language Model (LLM, which is essentially a generative model, also referred to as a generative large language model) has also demonstrated powerful NLP (Natural Language Processing) understanding and generation capabilities.

[0004] How to apply the conventional generative large language model, which is only used to process text-related tasks, to process image tasks related to optical character recognition, is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] The embodiments of the present disclosure provide a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for training a generative large language model and for processing image tasks.

[0006] In a first aspect, an embodiment of the present disclosure proposes a method for training a generative large language model, comprising: obtaining a first generative large language model trained using a general text training set; wherein the first generative large language model is used to return an output text for an input text, and a self-attention network layer and a feedforward neural network layer are provided in the first generative large language model; freezing the parameters of the self-attention network layer and the feedforward neural network layer, and adding a cross-attention network layer between the self-attention network layer and the feedforward neural network layer, and initializing the parameters of the cross-attention network layer to obtain a second generative large language model; using the image features of the acquired sample image and the prompt words used to indicate the task type as actual input, and using the text sequence matching the actual input as the expected output, to train the second generative large language model in a fine-tuning manner to obtain a third generative large language model.

[0007] In a second aspect, an embodiment of the present disclosure proposes a device for training a generative large language model, comprising: a text task processing model acquisition unit, configured to acquire a first generative large language model trained using a general text training set; wherein the first generative large language model is used to return an output text for an input text, and a self-attention network layer and a feedforward neural network layer are provided in the first generative large language model; a network layer addition processing unit, configured to freeze the parameters of the self-attention network layer and the feedforward neural network layer, and to add a cross-attention network layer between the self-attention network layer and the feedforward neural network layer, and to initialize the parameters of the cross-attention network layer, to obtain a second generative large language model; an image task processing model training unit, configured to use the image features of the acquired sample image and the prompt words used to indicate the task type as actual input, and the text sequence matching the actual input as the expected output, to train the second generative large language model in a fine-tuning manner, to obtain a third generative large language model.

[0008] In a third aspect, an embodiment of the present disclosure proposes a method for processing an image task, including: obtaining an image to be processed contained in the task to be processed and a target prompt word representing the task type; inputting the image to be processed and the target prompt word into a preset third generative large language model to obtain a text sequence output by the third generative large language model; wherein the third generative large language model is obtained according to the method for training a generative large language model as described in the first aspect; and returning the task processing result consisting of the text sequence.

[0009] In a fourth aspect, an embodiment of the present disclosure proposes a device for processing an image task, comprising: a task acquisition unit to be processed, configured to acquire an image to be processed contained in the task to be processed and a target prompt word representing the task type; a model calling unit, configured to input the image to be processed and the target prompt word into a preset third generative large language model to obtain a text sequence output by the third generative large language model; wherein the third generative large language model is obtained according to the device for training a generative large language model as described in the second aspect; and a task processing result returning unit, configured to return a task processing result consisting of a text sequence.

[0010] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the method for training a generative large language model as described in the first aspect or the method for processing image tasks as described in the third aspect.

[0011] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the method for training a generative large language model as described in the first aspect or the method for processing image tasks as described in the third aspect when executed.

[0012] In a seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the method for training a generative large language model as described in the first aspect or the steps of the method for processing image tasks as described in the third aspect.

[0013] The scheme for training a generative large language model and the scheme for processing image tasks provided by the embodiments of the present disclosure are based on the first generative large language model that already has strong text data processing capabilities. By fixing the parameters of its self-attention network layer and feedforward neural network layer, and adding a cross-attention network layer for determining the association between cross-modal data (i.e., the association between text data and image data), and combining a small number of samples consisting of sample images, prompt words indicating task types, and matching text sequences in the field of image processing, a third generative large language model that can accurately process images is obtained at low cost and high efficiency by only fine-tuning the parameters of the added cross-attention network layer. Then, with the help of the third generative large language model, a better processing effect on image processing tasks related to optical character recognition technology is achieved.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0016] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0017] Figure 2 A flowchart of a method for training a generative large language model provided in an embodiment of the present disclosure;

[0018] Figure 3 A flowchart of a method for generating actual input provided by an embodiment of the present disclosure;

[0019] Figure 4 A flowchart of a method for processing an image task provided by an embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of a structural flow for training a second generative large language model to obtain a third generative large language model provided by an embodiment of the present disclosure;

[0021] Figure 6 A structural block diagram of a device for training a generative large language model provided in an embodiment of the present disclosure;

[0022] Figure 7 A structural block diagram of a device for processing image tasks provided by an embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram of the structure of an electronic device suitable for executing a method for training a generative large language model and / or a method for processing an image task provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0025] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for training a generative large language model and for processing image tasks of the present application can be applied.

[0027] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various applications for realizing information communication between the terminal devices 101, 102, 103 and the server 105 can be installed, such as sample collection applications, model training applications, image task processing applications, etc.

[0029] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0030] The server 105 can provide various services through various built-in applications. Taking the image task processing application that can provide image processing services to users as an example, the server 105 can achieve the following effects when running the image task processing application: first, the task to be processed transmitted by the terminal devices 101, 102, and 103 is received through the network 104, and the task to be processed includes the image to be processed and the target prompt word representing the task type; then, the image to be processed and the target prompt word are input into the preset third generative large language model to obtain the text sequence output by the third generative large language model; finally, the text sequence is returned to the terminal devices 101, 102, and 103 through the network 104 as the task processing result.

[0031] Among them, the third generative large language model can be trained by a model training application built into the server 105 for training a generative large language model according to the following steps: first, a first generative large language model trained using a general text training set is obtained, the first generative large language model is used to return an output text for an input text, and a self-attention network layer and a feedforward neural network layer are provided in the first generative large language model; then, the parameters of the self-attention network layer and the feedforward neural network layer are frozen, and a cross-attention network layer is added between the self-attention network layer and the feedforward neural network layer, and the parameters of the cross-attention network layer are initialized to obtain a second generative large language model; finally, the image features of the obtained sample image and the prompt words used to indicate the task type are used as actual inputs, and a text sequence matching the actual input is used as the expected output, and the second generative large language model is trained in a fine-tuning manner to obtain a third generative large language model.

[0032] Since it takes more computing resources and stronger computing power to train the third generative large language model, the methods for training the generative large language model provided in the subsequent embodiments of the present application are generally performed by the server 105 with stronger computing power and more computing resources. Accordingly, the device for training the generative large language model is generally also set in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the various operations assigned to the server 105 by the class application for training the generative large language model installed thereon, and then output the same result as the server 105. Correspondingly, the device for training the generative large language model can also be set in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0033] Of course, the server used to train the third generative large language model can be different from the server that calls the trained third generative large language model for use. In particular, the third generative large language model trained by the server 105 can also be used to obtain a lightweight third generative large language model suitable for being placed in the terminal devices 101, 102, and 103 by model distillation, that is, it can be flexibly selected to use the lightweight third generative large language model in the terminal devices 101, 102, and 103 according to the recognition accuracy of actual needs, or to choose to use the more complex third generative large language model in the server 105.

[0034] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0035] Please refer to Figure 2 , Figure 2 A flowchart of a method for training a generative large language model provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0036] Step 201: Obtain a first generative large language model trained using a general text training set;

[0037] This step is intended to be performed by the execution subject of the method for training a generative large language model (e.g. Figure 1 The server 105 shown in the figure obtains a first generative large language model trained using a general text training set, where the general text training set is composed of training samples consisting of input texts and output texts corresponding to multiple pure text processing tasks. Therefore, the first generative large language model trained based on this also only has the text task processing capability of returning corresponding text processing results for the input text data.

[0038] Since the first generative large language model is only used to process text data, it is a model that only processes unimodal data, and only has a self-attention network layer (Self-Attention) and a feedforward neural network layer (Feed Forward) in the model.

[0039] Among them, Self-Attention is a self-attention mechanism that encodes the relationship between different positions in the input sequence into feature representation. It is widely used in natural language processing tasks, such as machine translation, text summarization, and semantic understanding. The input of attention includes query, key, and value. The attention distribution is obtained by calculating the similarity between the query and the key, and then the attention distribution is weighted and summed with the corresponding value to generate the final representation. Self-attention can capture long-distance dependencies in the input sequence and provide more global information modeling capabilities.

[0040] Feed Forward is a basic structure in neural networks. It consists of multiple fully connected layers. Each layer is connected using a nonlinear activation function (such as ReLU, whose full English name is: Linear rectification function, Chinese name: linear rectification function). It maps the input to a new representation space through a series of linear transformations and nonlinear activation functions. Feedforward neural networks are widely used in many deep learning models, such as convolutional neural networks and multi-layer perceptrons. Its function is to introduce nonlinear modeling capabilities and extract high-level feature representations of the input for classification and regression of downstream tasks.

[0041] Typically, the first generative large language model includes multiple network modules, and each network module is provided with a self-attention network layer and a feedforward neural network layer connected in sequence, so as to jointly use the functions of these two different types of network layers to improve the model performance.

[0042] Step 202: Freeze the parameters of the self-attention network layer and the feedforward neural network layer, add a cross-attention network layer between the self-attention network layer and the feedforward neural network layer, and initialize the parameters of the cross-attention network layer to obtain a second generative large language model;

[0043] On the basis of step 201, this step aims to first freeze the network layer parameters of the self-attention network layer and the feedforward neural network layer by the above-mentioned execution entity, and further add a cross-attention network layer (Cross-Attention) between the self-attention network layer and the feedforward neural network layer, and initialize the parameters of the cross-attention network layer, and finally name the currently obtained generative large language model as the second generative large language model.

[0044] Cross Attention is an attention mechanism used in multimodal or multimodal tasks, which is used to establish associations between different input sequences or feature maps by weightedly aggregating one sequence or feature map (query) with another sequence or feature map (key and value) across domains. Cross attention allows the model to interact and transfer information between different inputs so that relevant information can be selected and integrated when needed. A possible application includes attention transfer between source and target language sequences in machine translation, and attention transfer between image feature maps and generated caption sequences in image caption generation.

[0045] The reason why this embodiment adds a cross-attention network layer to the first generative large language model used only for processing text tasks is that in the image task processing scenario that the present disclosure intends to apply, it is used to establish a cross-modal association between text data and image data. In practice, the cross-attention calculation process of the cross-attention network layer can be as follows:

[0046] First, the matching degree between the query and the key is determined by calculating the similarity (attention score) between the query and the key; then, the values ​​are weighted and aggregated using the attention score to produce a weighted sum representation that combines the information of the query and the key; finally, the weighted sum representation can be used for downstream tasks, such as generating translation results or image captions. The advantage of cross-attention is that it can establish long-range dependencies between input sequences or feature maps, thereby helping the model capture global information between inputs. By allowing the model to weight keys and values ​​differently according to the needs of the query, cross-attention can efficiently select and integrate relevant information in the input sequence or feature map, improving the performance of the model.

[0047] Step 203: The image features of the acquired sample image and the prompt words used to indicate the task type are taken as actual inputs, and the text sequence matching the actual inputs is taken as expected outputs, and the second generative large language model is trained in a fine-tuning manner to obtain a third generative large language model.

[0048] On the basis of step 202, this step aims to train the second generative large language model in a fine-tuning manner by the above-mentioned execution subject using the image features of the sample image and the prompt words used to indicate the task type as the actual input and the text sequence matching the actual input as the expected output, so as to obtain the third generative large language model. That is, the actual input and the matching text sequence will constitute the fine-tuning training samples for fine-tuning the second generative large language model, and due to the difference in acquisition difficulty, the number of samples of the fine-tuning training samples constructed by the actual input and the expected output will usually be much smaller than the number of samples contained in the general text training set (for example, the difference in order of magnitude is ten thousand times, one hundred thousand times or even one million times).

[0049] The image features are generated by encoding the input sample image using an image encoder added to the input layer of the third generative large language model. Specifically, since the first generative large language model and the second generative large language model are not actually used to process image data, even if there are encoders in them, they should only be conventional text encoders used only to encode input text. Therefore, the image encoder can be additionally added on the basis of the text encoder, or the original text encoder can be replaced with an image encoder used to encode image data during the fine-tuning training process when the text encoder is not needed to encode the text.

[0050] In addition, compared to training a generative large language model for image task processing from scratch, fine-tuning training is an additional training based on a model that has been pre-trained on a large-scale dataset. The pre-trained model has learned rich semantic and contextual information and has good general feature extraction capabilities. Therefore, by fine-tuning these models, we can use the knowledge they have learned on large-scale data to provide better initial parameters, thereby accelerating the convergence speed of the model and improving the performance of the model. At the same time, for fine-tuning training samples that are difficult to collect, fine-tuning training can flexibly adapt to the characteristics of new tasks by making minor adjustments based on the pre-trained model. This transfer learning method can achieve better performance on limited data samples and reduce the need for large-scale labeled data. In addition, fine-tuning training can provide a better initial model, which is easier to achieve higher performance in actual tasks. The pre-trained model has been fully trained on rich data, and can provide a more general feature representation and parameter initialization, which reduces the overfitting problem of the model in the initial stage, and also provides a better starting point to solve new tasks.

[0051] The method for training a generative large language model provided by the embodiment of the present disclosure is based on a first generative large language model that already has a strong text data processing capability. By fixing the parameters of its self-attention network layer and feedforward neural network layer, and adding a cross-attention network layer for determining the association between cross-modal data (i.e., the association between text data and image data), and combining a small number of samples consisting of sample images, prompt words indicating task types, and matching text sequences in the field of image processing, a third generative large language model that can accurately process images is obtained at low cost and high efficiency by only fine-tuning the parameters of the added cross-attention network layer.

[0052] OCR technology can specifically include text detection, text recognition, information extraction, table information extraction, structured information extraction, specific area information extraction and paraphrase and other sub-techniques. Since the development of OCR technology, each task has been developing along different technical routes. Different tasks have different network designs, and there is nothing in common between them. Considering that under normal circumstances, different models are used for independent modeling for different tasks, which leads to the need to train multiple models and design multiple solutions, the R&D cost is high, and each model is completely independent, the data cannot be used efficiently, and there is a lack of coordination and mutual promotion between tasks.

[0053] To solve this problem, we can also unify the modeling of different image processing sub-tasks so that the models can be unified among multiple tasks, thereby making full use of the data of different tasks and promoting the effects of different tasks. An implementation method including but not limited to:

[0054] At the same time, the second generative large language model is trained in a multi-task unified learning manner, using image features and prompt words for indicating different task types as actual input and text sequences matching the actual input as expected output. The different types of tasks are all tasks related to optical character recognition technology, including at least one of character recognition, text extraction, table content extraction, and structured information extraction.

[0055] The technology used in this solution is Multi-Task Learning (MTL), which aims to improve the performance and generalization ability of the model by learning multiple related tasks at the same time. Its basic principles can be described as follows: 1) Task relevance: The core idea of ​​MTL is to train multiple related tasks together, and there is a certain degree of relevance or shared information between these tasks. By leveraging the relevance between tasks, knowledge can be better shared and transferred, thereby improving the learning effect of each task; 2) Shared representation learning: In MTL, the model handles different tasks by sharing the underlying representation learning. The underlying representation refers to the abstract feature representation learned by the model in the first few layers, which has a certain degree of sharing between different tasks. By sharing the underlying representation, the model can simultaneously learn the relevant features shared by multiple tasks, so that each task can benefit from the learning of other tasks; 3) Model structure design: The model structure of MTL usually consists of two parts: a shared layer and a specific task layer. The shared layer is the underlying part of the model, which is used to learn shared representations. The specific task layer is a dedicated layer designed for each task, which is used to learn task-specific representations and predictions. Information transfer and sharing can be achieved between the shared layer and the task layer by sharing parameters or sharing part of the parameters; 4) Objective function: The objective function of MTL is the joint loss function of multiple tasks, which combines the loss functions of multiple tasks. Usually a weighted loss function is used to assign different weights to each task. These weights can be determined based on the importance of the task, or based on the difficulty of the task or the amount of data; 5) Learning strategy: Different learning strategies can be used in the training process of MTL. Common strategies include alternating optimization, joint optimization, and joint training. Alternating optimization refers to optimizing each task alternately. Joint optimization refers to optimizing the parameters of all tasks at the same time. Joint training refers to jointly training models of multiple tasks on the entire dataset.

[0056] That is, through multi-task unified learning, the model can jointly learn and share knowledge from related tasks, thereby improving the performance and generalization ability of each task. This joint learning method can reduce data requirements, reduce the risk of overfitting, and encourage the model to learn common features and abstract representations, achieving better results when processing related tasks.

[0057] Please refer to Figure 3 , Figure 3 A flowchart of a method for generating actual input provided in an embodiment of the present disclosure, that is, for Figure 2 The process 203 shown uses actual input to provide a specific generation method, wherein the process 300 includes the following steps:

[0058] Step 301: vectorize the image features of the sample image to obtain an image feature vector;

[0059] Step 302: vectorizing the prompt word input together with the sample image to obtain a prompt word vector;

[0060] Step 303: concatenate the image feature vector and the prompt word vector, and use the obtained concatenated vector as actual input.

[0061] That is, this embodiment performs vectorization processing on the image features and the prompt words respectively, and concatenates the vectors resulting from the vectorization processing, thereby using the concatenated vector as the actual input.

[0062] Furthermore, in order to unify the input length, when the length of the splicing vector actually obtained is less than the preset length, a zero padding operation may be performed at a preset position of the splicing vector. The preset position should be a position where the zero padding operation does not affect the model's extraction of image features and prompt word information, such as the very end or the beginning of the splicing vector.

[0063] On the basis of any of the above embodiments, this embodiment can further use a preset verification sample set to verify the third generative large language model with unfrozen parameters of the self-attention network layer and the feedforward neural network layer; wherein the verification sample in the verification sample set is also composed of an input sample containing a sample image and a prompt word and an output sample consisting of a matched text sequence, and the verification sample can be a sample of the same type but different from the fine-tuning sample; finally, the parameters of each network layer in the third generative large language model are corrected according to the verification results.

[0064] It should be noted that the "thawing" and "freezing" of parameters of a certain functional layer or network layer in the above embodiment and this embodiment is a common technique in deep learning, which is used to control whether to allow the weights and parameters of a specific layer to be updated during the training process. This concept usually plays an important role in transfer learning, fine-tuning, and optimization training processes.

[0065] The following will explain these two concepts in detail:

[0066] 1. Freezing parameters:

[0067] In a neural network, each layer contains some trainable parameters such as weights and biases, which are updated through backpropagation during training to minimize the loss function. When the parameters of a layer are frozen, it means that the parameters of the layer are no longer updated during training, that is, its weights remain unchanged. This is often used to retain the feature extraction capabilities that have been learned in the pre-trained model. The frozen layer will not be affected by new data and loss functions, so the features learned on the original task can be retained in subsequent tasks without being over-adjusted.

[0068] 2. Unfreezing parameters:

[0069] Unfreezing refers to re-enabling parameter updates in previously frozen layers. When fine-tuning or training on a new task is needed, some previously frozen layers can be unfrozen to allow them to adjust based on the new data. The purpose of this is to gradually adapt to the new data distribution and problem to be solved in the new task.

[0070] Simply put, the concept of freezing and unfreezing parameters is to control the weights and parameters of network layers in deep learning in order to retain previously trained knowledge in specific situations or to make adaptive adjustments in new tasks. This technique is used to improve the performance and generalization ability of the model when transferring learning, fine-tuning the model, and training complex models.

[0071] That is, in the previous training process to obtain the third generative large language model, only the parameters of the added cross-attention network layer are iterated and optimized. After the preliminary training is completed, the parameters of all network layers are revised as a whole by using verification samples, so that the parameters of the revised self-attention network layer and feedforward neural network layer can be more suitable for processing image tasks.

[0072] The above embodiments explain how to train the third generative large language model from various aspects. In order to highlight the effect of the trained third generative large language model from the actual usage scenario as much as possible, the present disclosure also specifically provides a solution for using the trained third generative large language model to solve practical problems. A method for processing an image task includes the following steps:

[0073] Step 401: Acquire the image to be processed contained in the task to be processed and the target prompt word representing the task type;

[0074] This step is intended to be performed by an execution subject of the method for processing an image task (e.g. Figure 1 The server 105 shown in the figure receives an incoming message from a terminal device (eg Figure 1 The terminal devices 101, 102, 103 shown in the figure have tasks to be processed, and the tasks to be processed include images to be processed and target prompt words representing the task types.

[0075] Step 402: inputting the image to be processed and the target prompt word into a preset third generative large language model to obtain a text sequence output by the third generative large language model;

[0076] Step 403: Return the task processing result represented by the text sequence

[0077] Based on the above-mentioned solution for training a generative large language model, this embodiment achieves better processing effects on image processing tasks related to optical character recognition technology by using the third generative large language model.

[0078] To deepen understanding, the present disclosure also provides a specific implementation scheme in combination with a specific application scenario, which is mainly divided into the following main parts:

[0079] 1. Unified input and output:

[0080] This embodiment first unifies the input and output of all tasks into the form of input picture, input prompt (prompt word), and output result sequence.

[0081] For example, for the detection task, the input image is the image to be detected, the input prompt is "output the position of the text in the image", and the output result is x1,y1,x2,y2,x3,y3,x4,y4x1,y1,x2,y2,x3,y3,x4,y4 <eos>Sequence. Among them is a separator, used to separate different texts. <eos>The end of the sequence.

[0082] For example, for the recognition task, the input image is a detected image, the input prompt is "output recognition result", and the output result is hello word <eos>;

[0083] For example, for the information extraction task, the output image is a picture (such as a train ticket), the input prompt is "output the departure station of the train", and the output result is "Beijing West <eos>”;

[0084] 2. Model structure

[0085] 1. Encoder

[0086] This embodiment uses a convolutional neural network as an example to implement feature encoding of image data (also called image feature extraction). In theory, a backbone network of any structure can be used in this solution.

[0087] 2.Decoder

[0088] This example uses Transformer Decoder as the decoding module. The prompt and image features are used as the input of the Decoder, and the predicted results are input serially. Taking the train ticket information extraction task mentioned above as an example, the image features and "how much is the train ticker <eoq>" is input into the network as the actual input. At time t0, the network outputs 8. Then at time t1, the network inputs the image features and "how much is the trainticker <eoq>8”, the output at time t1 is 2, and so on until the network outputs <eos>termination.

[0089] 3. Fine-tuning solution based on pre-trained large model

[0090] This embodiment fine-tunes the model based on the pre-trained large model to give the model a strong generalization ability. Specifically, based on the pre-trained large model (only the Self-Attention part and the Feed Forward part), Cross-Attention is added to allow the text prompt and image feature to interact to ensure that the network outputs the information in the picture required by the text prompt. Only the cross attention layer is trained to retain the generalization ability of the pre-trained large model.

[0091] This embodiment not only proposes a multi-task unified modeling solution for a generative large language model, but also simplifies the model training process and improves R&D efficiency through fine-tuning training, which is conducive to improving user experience.

[0092] Further references Figure 6 and Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for training a generative large language model and an embodiment of a device for processing an image task. The embodiment of the device for training a generative large language model and the embodiment of the device for processing an image task are respectively provided. Figure 2 Corresponding to the method embodiment for training a generative large language model shown in FIG. 1 , the apparatus embodiment for processing an image task is similar to Figure 4 The above device can be applied to various electronic devices.

[0093] like Figure 6 As shown, the apparatus 600 for training a generative large language model in this embodiment may include: a text task processing model acquisition unit 601, a network layer addition processing unit 602, and an image task processing model training unit 603. The text task processing model acquisition unit 601 is configured to acquire a first generative large language model trained using a general text training set; the first generative large language model is used to return an output text for an input text, and a self-attention network layer and a feedforward neural network layer are provided in the first generative large language model; the network layer addition processing unit 602 is configured to freeze the parameters of the self-attention network layer and the feedforward neural network layer, and add a cross-attention network layer between the self-attention network layer and the feedforward neural network layer, and initialize the parameters of the cross-attention network layer to obtain a second generative large language model; the image task processing model training unit 603 is configured to use the image features of the acquired sample image and the prompt words for indicating the task type as actual inputs, and use the text sequence matching the actual inputs as expected outputs, and train the second generative large language model in a fine-tuning manner to obtain a third generative large language model.

[0094] In the present embodiment, in the apparatus 600 for training a generative large language model, the specific processing of the text task processing model acquisition unit 601, the network layer additional processing unit 602, and the image task processing model training unit 603 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0095] In some optional implementations of this embodiment, the apparatus 600 for training a generative large language model may further include:

[0096] The multi-task unified learning unit is configured to simultaneously take the image features and the prompt words for indicating different task types as actual input and the text sequence matching the actual input as expected output, and train the second generative large language model in a multi-task unified learning manner.

[0097] In some optional implementations of this embodiment, different types of tasks are tasks related to optical character recognition technology, including: at least one of: character recognition, text extraction, table content extraction, and structured information extraction.

[0098] In some optional implementations of this embodiment, the first generative large language model includes multiple network modules, and each network module is provided with a self-attention network layer and a feedforward neural network layer connected in sequence.

[0099] In some optional implementations of this embodiment, the image features are generated by encoding the input sample image using an image encoder added to the input layer of the third generative large language model.

[0100] In some optional implementations of this embodiment, the number of fine-tuning training samples constructed from actual input and expected output is much smaller than the number of samples included in the general text training set.

[0101] In some optional implementations of this embodiment, the apparatus 600 for training a generative large language model may further include:

[0102] The actual input generation unit is configured to generate an actual input, and the actual input generation unit can be further configured to:

[0103] Vectorize the image features of the sample image to obtain an image feature vector;

[0104] Vectorize the prompt word input together with the sample image to obtain the prompt word vector;

[0105] Concatenate the image feature vector and the prompt word vector, and use the resulting concatenated vector as the actual input.

[0106] In some optional implementations of this embodiment, the apparatus 600 for training a generative large language model may further include:

[0107] The length padding unit is configured to perform a zero padding operation at a preset position of the splicing vector in response to the length of the splicing vector being less than a preset length.

[0108] In some optional implementations of this embodiment, the apparatus 600 for training a generative large language model may further include:

[0109] A validity verification unit is configured to verify the third generative large language model with parameters of the self-attention network layer and the feedforward neural network layer unfrozen using a preset verification sample set; wherein the verification samples in the verification sample set are composed of input samples including sample images and prompt words and output samples consisting of matched text sequences;

[0110] The parameter fine-tuning unit is configured to correct the parameters of each network layer in the third generative large language model according to the verification result.

[0111] like Figure 7 As shown, the apparatus 700 for processing image tasks of this embodiment may include: a task acquisition unit 701, a model calling unit 702, and a task processing result returning unit 703. The task acquisition unit 701 is configured to acquire the image to be processed and the target prompt word representing the task type contained in the task to be processed; the model calling unit 702 is configured to input the image to be processed and the target prompt word into a preset third generative large language model to obtain a text sequence output by the third generative large language model; wherein the third generative large language model is obtained according to the apparatus 600 for training a generative large language model; and the task processing result returning unit 703 is configured to return the task processing result as a text sequence.

[0112] In this embodiment, in the device 700 for processing image tasks: the specific processing of the task acquisition unit 701, the model calling unit 702, and the task processing result return unit 703 and the technical effects brought about by them can respectively correspond to the relevant descriptions in the method embodiment, and will not be repeated here.

[0113] This embodiment exists as an apparatus embodiment corresponding to the above method embodiment. The apparatus for training a generative large language model and the apparatus for processing an image task provided by this embodiment are based on the first generative large language model that already has a strong text data processing capability. By fixing the parameters of its self-attention network layer and feedforward neural network layer, and adding a cross-attention network layer for determining the association between cross-modal data (i.e., the association between text data and image data), and combining a small number of samples consisting of sample images, prompt words indicating task types, and matching text sequences in the field of image processing, a third generative large language model capable of accurately processing images is obtained at low cost and high efficiency by only fine-tuning the parameters of the added cross-attention network layer. Then, with the help of the third generative large language model, a better processing effect on image processing tasks related to optical character recognition technology is achieved.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method for training a generative large language model and / or the method for processing image tasks described in any of the above embodiments when executing.

[0115] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for training a generative large language model and / or the method for processing image tasks described in any of the above embodiments when executed.

[0116] The embodiments of the present disclosure provide a computer program product, which, when executed by a processor, can implement the steps of the method for training a generative large language model and / or the steps of the method for processing an image task described in any of the above embodiments.

[0117] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0118] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a method for training a generative large language model and / or a method for processing an image task. For example, in some embodiments, the method for training a generative large language model and / or a method for processing an image task may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for training a generative large language model and / or the method for processing an image task described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other suitable manner (e.g., by means of firmware) to execute the method for training a generative large language model and / or the method for processing an image task.

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0126] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0127] The technical solution of the disclosed embodiment is based on the first generative large language model that already has strong text data processing capabilities. By fixing the parameters of its self-attention network layer and feedforward neural network layer, and adding a cross-attention network layer for determining the association between cross-modal data (i.e., the association between text data and image data), and combining a small number of samples consisting of sample images, prompt words indicating task types, and matching text sequences in the field of image processing, a third generative large language model that can accurately process images is obtained at low cost and high efficiency by only fine-tuning the parameters of the added cross-attention network layer. Then, with the help of the third generative large language model, a better processing effect on image processing tasks related to optical character recognition technology is achieved.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.< / eos> < / eoq> < / eoq> < / eos> < / eos> < / eos> < / eos>

Claims

1. A method for training a generative large language model, comprising: Obtaining a first generative large language model trained using a general text training set; wherein the first generative large language model is used to return an output text for an input text, and the first generative large language model is provided with a self-attention network layer and a feedforward neural network layer; Freeze the parameters of the self-attention network layer and the feedforward neural network layer, add a cross-attention network layer between the self-attention network layer and the feedforward neural network layer, and initialize the parameters of the cross-attention network layer to obtain a second generative large language model; The image features of the acquired sample image and the prompt words for indicating the task type are taken as actual inputs, and the text sequence matching the actual inputs is taken as expected outputs, and the second generative large language model is trained in a fine-tuning manner to obtain a third generative large language model; different types of tasks are all tasks related to optical character recognition technology, including: at least one of character recognition, text extraction, table content extraction, and structured information extraction; the image features are generated by encoding the input sample image using an image encoder added to the input layer of the third generative large language model.

2. The method according to claim 1, further comprising: At the same time, the image features and prompt words for indicating different task types are used as actual input, and a text sequence matching the actual input is used as expected output, and the second generative large language model is trained in a multi-task unified learning manner.

3. The method according to claim 1, wherein: The first generative large language model includes multiple network modules, and each of the network modules is provided with a self-attention network layer and a feedforward neural network layer connected in sequence.

4. The method according to claim 1, wherein: The number of fine-tuning training samples constructed by the actual input and the expected output is much smaller than the number of samples included in the general text training set.

5. The method according to claim 1, wherein: The actual input is generated as follows: Vectorizing the image features of the sample image to obtain an image feature vector; Vectorizing the prompt word input together with the sample image to obtain a prompt word vector; The image feature vector and the prompt word vector are concatenated, and the obtained concatenated vector is used as the actual input.

6. The method according to claim 5, further comprising: In response to the length of the splicing vector being less than a preset length, a zero padding operation is performed at a preset position of the splicing vector.

7. The method according to any one of claims 1 to 6, further comprising: Using a preset verification sample set to verify the third generative large language model with parameters of the self-attention network layer and the feedforward neural network layer unfrozen; wherein the verification samples in the verification sample set are composed of input samples including sample images and prompt words and output samples consisting of matched text sequences; The parameters of each network layer in the third generative large language model are modified according to the verification result.

8. A method for processing an image task, comprising: Acquire the image to be processed contained in the task to be processed and the target prompt word representing the task type; Inputting the image to be processed and the target prompt word into a preset third generative large language model to obtain a text sequence output by the third generative large language model; wherein the third generative large language model is obtained according to the method for training a generative large language model according to any one of claims 1 to 7; Returns the text sequence as the result of the task processing.

9. A device for training a generative large language model, comprising: A text task processing model acquisition unit is configured to acquire a first generative large language model trained using a general text training set; wherein the first generative large language model is used to return an output text for an input text, and a self-attention network layer and a feedforward neural network layer are provided in the first generative large language model; A network layer additional processing unit is configured to freeze the parameters of the self-attention network layer and the feedforward neural network layer, and to add a cross-attention network layer between the self-attention network layer and the feedforward neural network layer, and to initialize the parameters of the cross-attention network layer to obtain a second generative large language model; The image task processing model training unit is configured to use the image features of the acquired sample image and the prompt words indicating the task type as actual inputs and the text sequence matching the actual inputs as expected outputs, and train the second generative large language model in a fine-tuning manner to obtain a third generative large language model; different types of tasks are all tasks related to optical character recognition technology, including: at least one of character recognition, text extraction, table content extraction, and structured information extraction; the image features are generated by encoding the input sample image using an image encoder added to the input layer of the third generative large language model.

10. The apparatus according to claim 9, further comprising: The multi-task unified learning unit is configured to simultaneously take the image features and the prompt words indicating different task types as actual input and the text sequence matching the actual input as expected output, and train the second generative large language model in a multi-task unified learning manner.

11. The device according to claim 9, wherein: The first generative large language model includes multiple network modules, and each of the network modules is provided with a self-attention network layer and a feedforward neural network layer connected in sequence.

12. The device according to claim 9, wherein: The number of fine-tuning training samples constructed by the actual input and the expected output is much smaller than the number of samples included in the general text training set.

13. The apparatus according to claim 9, further comprising: An actual input generating unit configured to generate an actual input, wherein the actual input generating unit is further configured to: Vectorizing the image features of the sample image to obtain an image feature vector; Vectorizing the prompt word input together with the sample image to obtain a prompt word vector; The image feature vector and the prompt word vector are concatenated, and the obtained concatenated vector is used as the actual input.

14. The apparatus according to claim 13, further comprising: The length padding unit is configured to perform a zero padding operation at a preset position of the splicing vector in response to the length of the splicing vector being less than a preset length.

15. The device according to any one of claims 9 to 14, further comprising: A validity verification unit is configured to verify the third generative large language model with parameters of the self-attention network layer and the feedforward neural network layer unfrozen using a preset verification sample set; wherein the verification samples in the verification sample set are composed of input samples including sample images and prompt words and output samples consisting of matched text sequences; A parameter fine-tuning unit is configured to correct the parameters of each network layer in the third generative large language model according to the verification result.

16. A device for processing an image task, comprising: A task-to-be-processed acquisition unit is configured to acquire an image to be processed contained in the task to be processed and a target prompt word representing a task type; A model calling unit is configured to input the image to be processed and the target prompt word into a preset third generative large language model to obtain a text sequence output by the third generative large language model; wherein the third generative large language model is obtained according to the apparatus for training a generative large language model according to any one of claims 9 to 15; The task processing result returning unit is configured to return the text sequence as the task processing result.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for training a generative large language model described in any one of claims 1-7 and / or the method for processing image tasks described in claim 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method for training a generative large language model as described in any one of claims 1 to 7 and / or the method for processing image tasks as described in claim 8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for training a generative large language model according to any one of claims 1 to 7 and / or the steps of the method for processing image tasks according to claim 8.

Images