Text generation method based on multi-task, terminal device and readable storage medium

Abstract

The application relates to the field of machine learning, and discloses a multi-task-based text generation method, a terminal device and a computer readable storage medium. The method comprises the following steps: receiving a multi-task text generation instruction, and acquiring a task type and an image corresponding to the multi-task text generation instruction; acquiring a prompt chain template corresponding to the task type; taking the image and the prompt chain template as input parameters of a preprocessing model, and controlling the preprocessing model to generate target text corresponding to the multi-task text generation instruction. The application solves the problem that, in the related technology of multi-task text generation, the use of a complex model leads to an unstable text recognition task execution process and cannot guarantee the accuracy of the task execution result each time, and achieves the effect of improving the accuracy of the text recognition task.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to multi-task-based text generation methods, terminal devices, and computer-readable storage media. Background Technology

[0002] Discriminative multi-task learning has a wide range of applications and is often used to handle complex visual tasks such as image classification, object detection, and image segmentation. Through multi-task learning, the network can share learned features from multiple related tasks, thereby improving the performance of each task.

[0003] However, discriminative multi-task learning applications typically use a unified backbone network, with different network heads attached to achieve different tasks, such as image classification, object detection, and image segmentation. But attaching a separate network head for each task significantly increases model complexity. Complex models are prone to instability during training, leading to inconsistent performance in actual use and making it impossible to guarantee the accuracy of results for each task. Summary of the Invention

[0004] This application provides a multi-task-based text generation method, terminal device, and computer-readable storage medium. It solves the problem in related technologies of multi-task text generation where the use of complex models leads to instability in the text recognition task execution process and makes it impossible to guarantee the accuracy of the results of each task execution, thereby improving the accuracy of text recognition tasks.

[0005] This application provides a multi-task-based text generation method, the method comprising:

[0006] Receive a multi-task text generation instruction and obtain the task type and image corresponding to the multi-task text generation instruction;

[0007] Obtain the prompt chain template corresponding to the task type;

[0008] The image and the prompt chain template are used as input parameters for the preprocessing model, and the preprocessing model is controlled to generate the target text corresponding to the multi-task text generation instruction.

[0009] Optionally, before the step of receiving the multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction, the method further includes:

[0010] Receive a multi-task training dataset and train the preprocessed model;

[0011] Obtain test data and test the trained preprocessed model;

[0012] The preprocessing model that has passed the test is deployed to a multi-task text generation platform, waiting to receive the multi-task text generation instructions.

[0013] Optionally, the step of receiving the multi-task training dataset and training the preprocessed model includes:

[0014] Receive a multi-task training dataset, the multi-task training dataset including images, labels corresponding to the images, and cue chain statements corresponding to the images;

[0015] The multi-task training dataset is input into the first pre-trained model to obtain the image-text similarity matrix;

[0016] The image-text similarity matrix is ​​permuted to obtain a one-dimensional vector, and the one-dimensional vector is used as the input of the second pre-trained model to obtain the descriptive statement of the image;

[0017] The description and the label are fused and corrected to form image cue chain data.

[0018] Optionally, the step of receiving a multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction includes:

[0019] Receive the multi-task text generation instruction, and obtain single task information based on the multi-task text generation instruction, wherein the single task information includes the task type corresponding to the single task and the image corresponding to the single task;

[0020] Based on the correlation between the individual task information, the tasks in the multi-task text generation instruction are sorted.

[0021] Output the task type and the image based on the sorting result.

[0022] Optionally, after the steps of receiving the multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction, the method further includes:

[0023] Determine whether the task type exists in the task type library;

[0024] If the task type exists, then the step of obtaining the prompt chain template corresponding to the task type is executed;

[0025] If the task type does not exist, an auxiliary prompt chain template is obtained, and the auxiliary prompt chain template and the image are used as the input parameters. The preprocessing model is then controlled to generate the target text corresponding to the multi-task text generation instruction.

[0026] Optionally, the auxiliary cue chain template includes a general cue chain template, and the step of obtaining the auxiliary cue chain template includes:

[0027] The historical target text generated by the preprocessing model is obtained, and the correlation between the historical figures and historical images corresponding to the historical target text is learned.

[0028] Migrate the correlation to the task type and the image to construct a general cue chain template.

[0029] Optionally, the auxiliary cue chain template includes a leading cue chain template, and the step of obtaining the auxiliary cue chain template includes:

[0030] Based on the task type, search for historical similar task types corresponding to the task type in the historical target text generated by the preprocessing model;

[0031] Obtain the historical prompt chain template corresponding to the historical similar task type, and construct the leading prompt chain template based on the historical prompt chain template.

[0032] Optionally, the step of using the image and the cue chain template as input parameters of the preprocessing model, and controlling the preprocessing model to generate the target text corresponding to the multi-task text generation instruction, includes:

[0033] The image and the prompt chain template are used as input parameters of the preprocessing model to identify the corresponding text description of the image;

[0034] The text description is filled into the prompt chain template to generate the target text.

[0035] In addition, to achieve the above objectives, embodiments of the present invention also provide a terminal device, including a memory, a processor, and a multitasking text generation program stored in the memory and executable on the processor. When the processor executes the multitasking text generation program, it implements the method described above.

[0036] In addition, to achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium storing a multitasking text generation program, which, when executed by a processor, implements the method described above.

[0037] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0038] Upon receiving a multi-task text generation instruction, the system acquires the corresponding task type and image, and then obtains the corresponding prompt chain template. The image and prompt chain template are used as input parameters for a preprocessing model, which then generates the target text corresponding to the multi-task text generation instruction. "Multi-task" refers to various types of tasks. For different types of text generation tasks, corresponding prompt chain templates can be obtained. The preprocessing model then parses the image, extracts the information required for the task type, and fills the extracted information into the corresponding positions in the prompt chain template to generate the target text. This means that diverse text generation tasks can be completed without using complex models. This solves the problem in related multi-task text generation technologies where using complex models leads to instability in the text recognition task execution process and fails to guarantee the accuracy of each task's result, thus improving the accuracy of text recognition tasks. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating an embodiment of the multi-task-based text generation method of this application;

[0040] Figure 2 This is a flowchart illustrating Embodiment 2 of the multi-task-based text generation method of this application;

[0041] Figure 3 This is a schematic diagram of the prompt chain text generation method based on multi-task in this application;

[0042] Figure 4 This is a flowchart illustrating Embodiment 3 of the multi-task-based text generation method of this application;

[0043] Figure 5 This is a schematic diagram of the terminal structure of the hardware operating environment involved in one embodiment of this application. Detailed Implementation

[0044] To achieve text generation tasks involving image classification, object detection, and image segmentation, related technologies typically require the construction of complex models. However, these complex models are unstable during training, leading to inconsistent performance in actual tasks and compromising the accuracy of results. To address this issue, this application proposes a multi-task-based text generation method. Upon receiving a multi-task text generation instruction, the method acquires the corresponding task type and image. Based on the task type, it obtains a corresponding prompt chain template. The prompt chain template and image are used as data parameters for a preprocessing model. After the preprocessing model extracts target data from the image, this data is filled into the corresponding positions of the prompt chain template, and the target text is output, completing the multi-task text generation task. This improves the accuracy of text recognition tasks.

[0045] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0046] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0047] Example 1

[0048] In this embodiment, a multi-task-based text generation method is provided.

[0049] Reference Figure 1 The multi-task-based text generation method in this embodiment includes the following steps:

[0050] Step S100: Receive a multi-task text generation instruction and obtain the task type and image corresponding to the multi-task text generation instruction;

[0051] In this embodiment, "multi-task" in the multi-task text generation instruction refers to multiple different types of tasks, such as image classification tasks, object detection tasks, or image segmentation tasks.

[0052] As an optional implementation, image classification refers to identifying content in an image according to a defined type, such as identifying color types or species in an image. Object detection refers to detecting specified content in an image, such as detecting whether an image contains a specific type of item or data. Image segmentation refers to segmenting an image according to a task definition, such as segmenting a ticket. In a multi-task text generation instruction, each task corresponds to at least one image, and each image carries an identifier for the corresponding task type. The task type can be determined based on the identifier.

[0053] For example, if the received multi-task text generation instruction is for a mixed-issue ticket scenario, the corresponding task types include image classification and image segmentation tasks. The entire page is segmented according to the image segmentation task to obtain various small regions. Then, according to the image classification task, the tickets in each small region are classified, such as invoices and itineraries.

[0054] As another optional implementation, for various types of text recognition tasks, it is usually necessary to sort the execution order of the various tasks to ensure the readability of the final generated target text. Specifically, upon receiving a multi-task text generation instruction, individual task information is obtained based on the instruction. This individual task information includes the task type and the corresponding image. The tasks in the multi-task text generation instruction are then sorted according to the relationships between the individual task information. Finally, the task type and the image are output based on the sorting result. Alternatively, tasks can be executed according to a pre-marked task execution order in the multi-task text generation instruction.

[0055] For example, in the scenario of mixed receipts, the image classification task and the image segmentation task can be determined based on the correlation between the task information of these two tasks. It can be determined that the image needs to be segmented first to generate multiple sub-images, and then these sub-images need to be classified. Therefore, the image segmentation task needs to be performed first, followed by the image classification task.

[0056] Step S200: Obtain the prompt chain template corresponding to the task type;

[0057] In this embodiment, the cue chain is equivalent to a branch of prompts in the field of natural language processing. It is a discrete cue learning method that simulates the thinking process by providing prompts in the input text (such as "Let's do it step by step"), thereby solving logical natural language question-and-answer problems. The cue chain templates correspond one-to-one with each task type, and the cue chain templates are pre-set, with one task potentially having multiple templates.

[0058] As an optional implementation, when setting prompt chain templates according to different task types, the prompt chain template can be calculated by comprehensively considering the statements generated by various task types in the past.

[0059] For example, the main annotations for the dataset of the object detection task are bounding boxes and object categories. These descriptive data are written into a sentence using natural language, such as "This image is a picture of, includes,,". The part without underscores is the cue chain template, while the part with underscores varies depending on the image.

[0060] Step S300: Use the image and the prompt chain template as input parameters of the preprocessing model, and control the preprocessing model to generate the target text corresponding to the multi-task text generation instruction.

[0061] In this embodiment, the preprocessing model is a trained model, and it is a single model consisting of a backbone network and a network head. The trained model can quickly and accurately identify target data in an image according to the task type.

[0062] As an optional implementation, by using the image and cue chain template as input parameters to the preprocessing model, the text description required for the task type in the image can be identified, i.e., the target data. Filling the cue chain template with the text description yields the target text.

[0063] For example, if the received multi-task text generation instruction includes image classification, object detection, and image segmentation tasks, the target text can be generated using the image, its labels, and a cue chain template, such as "This image is a picture of...". pets In region posx1,posy1 is a cat In region posx2,posy2 isa dog The border of cat is point1, point2, ..., pointn The border of dog is point1, point2, ..., pointn In this target text, the underlined part is the target data extracted from image classification, object detection, and image segmentation tasks.

[0064] In this embodiment, for multi-task text generation tasks, the cue chain templates and images corresponding to each task type are obtained and input into the preprocessing model. The preprocessing model calculates and identifies the target data in the image, fills the cue chain template with the target data, and outputs the target text, thus completing the multi-task text generation task. This achieves the completion of multi-task text generation tasks in a single model, thereby improving the accuracy of text recognition tasks.

[0065] Example 2

[0066] Based on Embodiment 1, another embodiment of this application is proposed, with reference to... Figure 2 Before the steps of receiving a multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction, the method further includes the following steps:

[0067] Step S010: Receive the multi-task training dataset and train the preprocessed model;

[0068] Step S020: Obtain test data and test the trained preprocessed model;

[0069] Step S030: Deploy the preprocessing model that has passed the test to the multi-task text generation platform and wait to receive the multi-task text generation instructions.

[0070] In this embodiment, before putting the preprocessing model into formal use, it is trained so that the preprocessing model can accurately identify the target data in the image according to the task type and fill the target data into the prompt chain template.

[0071] As an optional implementation, training the model requires acquiring a large training dataset, including datasets for tasks such as image classification, object detection, and image segmentation, as well as image-text pair datasets. After recognizing the datasets for tasks such as image classification, object detection, and image segmentation, the recognition results are matched with the corresponding content in the image-text pair dataset to test the preprocessed model. After passing the test, the preprocessed model is deployed to a multi-task text generation platform, awaiting multi-task text generation instructions.

[0072] For example, datasets for tasks such as image classification, object detection, and image segmentation are acquired. Then, cue chains are generated using existing images and image labels. The underlined parts in the cue chains represent the labels from the image classification, object detection, and image segmentation datasets. Logical relationships can be added to the generated cue chains to make them more grammatically correct. An additional sentence description model is then used to generate sentences from the images. These generated sentences are then used to replace some words in the previously generated cue chains, making them more coherent and logical. Based on the cue chains with added logical relationships and word replacements, the image text recognition function of the preprocessing model is continuously improved, enhancing the model's capabilities.

[0073] As another optional implementation, a multi-task training dataset is received, the multi-task training dataset including images, labels corresponding to the images, and cue chain statements corresponding to the images; the multi-task training dataset is input into a first pre-trained model to obtain an image-text similarity matrix; the image-text similarity matrix is ​​permuted to obtain a one-dimensional vector, and the one-dimensional vector is used as input to a second pre-trained model to obtain a description statement of the image; the description statement and the labels are fused and corrected to form image cue chain data.

[0074] For example, refer to Figure 3The corresponding cue chain statements for images in the multi-task training set are input into a first pre-trained model. After encoding the images, an image-text similarity matrix is ​​generated. This matrix is ​​then permuted to obtain a one-dimensional vector. This one-dimensional vector is input into a second pre-trained model, and after decoding, the image cue chain data is obtained. This model training method enables the preprocessing model to accurately fill the cue chain template with the target data extracted from the image.

[0075] In this embodiment, the preprocessing model is trained so that it can extract image content according to the corresponding task type and fill the extracted target data into the prompt chain template.

[0076] Example 3

[0077] Based on the above embodiments, another embodiment of this application is proposed, with reference to... Figure 4 After receiving the multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction, the method includes the following steps:

[0078] Step S400: Determine whether the task type exists in the task type library;

[0079] Step S500: If the task type exists, then execute the step of obtaining the prompt chain template corresponding to the task type;

[0080] Step S600: If the task type does not exist, obtain the auxiliary prompt chain template, use the auxiliary prompt chain template and the image as the input parameters, and control the preprocessing model to generate the target text corresponding to the multi-task text generation instruction.

[0081] In this embodiment, the multi-task text generation platform possesses strong generalization capabilities, enabling it to complete text recognition tasks even for data types not included in existing task types and task categories that have not undergone pre-training, without requiring retraining of the preprocessing model. The task type library contains various previously executed task types, each with a corresponding prompt chain template. Upon receiving the task type corresponding to the multi-task text generation instruction, the task type is matched against existing data in the task database. If a matching task type is found, the step of obtaining the prompt chain template corresponding to that task type is executed according to standard procedures. If no matching task type is found, an auxiliary prompt chain template is obtained to assist in generating the target text.

[0082] As an optional implementation, when the task type does not exist, historical target text generated by the preprocessing model is obtained, and the correlation between each historical figure type and historical image corresponding to the historical target text is learned; the correlation is then transferred to the task type and the image to construct a general cue chain template. This enables the construction of the general cue chain from scratch.

[0083] For example, to address task types that have not undergone pre-training, transfer learning can be used. This involves combining historical data with corresponding historical figures and images, along with machine learning algorithms such as classifiers, regression models, or deep learning models. By learning from historical data, the representation and characteristics of different types of historical figures and images in text can be discovered, and a mapping relationship can be established between them. This mapping relationship is then transferred to the task type and the image, and a general cue chain template corresponding to the task type and the image is constructed using a pre-defined machine learning algorithm. This general cue chain serves as the auxiliary cue chain. The general cue chain template is associated with this new task type and stored in a task type library for future use.

[0084] As another optional implementation, when the task type does not exist, it is also possible to search for historical similar task types corresponding to the task type in the historical target text generated by the preprocessing model; obtain the historical prompt chain template corresponding to the historical similar task type, and construct a leading prompt chain template based on the historical prompt chain template. The historical similar task types are determined according to the similarity values ​​of each task type.

[0085] For example, similarity values ​​can be determined by the task's objective or image features. If determined by the task's objective, tasks where the objective is to find a specific item in an image can be grouped into one type, and the similarity of tasks within this type can be calculated. Historical prompt chain templates corresponding to historically similar task types with similarity values ​​exceeding a preset similarity threshold are extracted. A leading prompt chain is constructed based on these historical prompt chain templates, serving as an auxiliary prompt chain. If determined by image features, image tag information can be extracted, and similarity can be calculated based on this image tag information. Image tag information can include usage scenarios, time, or other semantically labeled information.

[0086] In this embodiment, even when encountering data types not included in the existing task type library and task categories that have not been pre-trained, the text generation task can still be achieved through zero-learning and meta-learning functions.

[0087] Example 4

[0088] In this application embodiment, a multi-task-based text generation device is proposed.

[0089] Reference Figure 5 , Figure 5 This is a schematic diagram of the terminal structure of the hardware operating environment involved in one embodiment of this application.

[0090] like Figure 5 As shown, the control terminal may include: a processor 1001, such as a CPU, a network interface 1003, a memory 1004, and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The network interface 1003 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be high-speed RAM or stable non-volatile memory, such as a disk drive. Alternatively, the memory 1004 may be a storage device independent of the aforementioned processor 1001.

[0091] Those skilled in the art will understand that Figure 5 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0092] like Figure 5 As shown, the memory 1004, which serves as a computer storage medium, may include an operating system, a network communication module, and a multi-tasking text generation program.

[0093] exist Figure 5 In the hardware structure of the multitasking text generation device shown, the processor 1001 can call the multitasking text generation program stored in the memory 1004 and perform the following operations:

[0094] Receive a multi-task text generation instruction and obtain the task type and image corresponding to the multi-task text generation instruction;

[0095] Obtain the prompt chain template corresponding to the task type;

[0096] The image and the prompt chain template are used as input parameters for the preprocessing model, and the preprocessing model is controlled to generate the target text corresponding to the multi-task text generation instruction.

[0097] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0098] Receive a multi-task training dataset and train the preprocessed model;

[0099] Obtain test data and test the trained preprocessed model;

[0100] The preprocessing model that has passed the test is deployed to a multi-task text generation platform, waiting to receive the multi-task text generation instructions.

[0101] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0102] Receive a multi-task training dataset, the multi-task training dataset including images, labels corresponding to the images, and cue chain statements corresponding to the images;

[0103] The multi-task training dataset is input into the first pre-trained model to obtain the image-text similarity matrix;

[0104] The image-text similarity matrix is ​​permuted to obtain a one-dimensional vector, and the one-dimensional vector is used as the input of the second pre-trained model to obtain the descriptive statement of the image;

[0105] The description and the label are fused and corrected to form image cue chain data.

[0106] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0107] Receive the multi-task text generation instruction, and obtain single task information based on the multi-task text generation instruction, wherein the single task information includes the task type corresponding to the single task and the image corresponding to the single task;

[0108] Based on the correlation between the individual task information, the tasks in the multi-task text generation instruction are sorted.

[0109] Output the task type and the image based on the sorting result.

[0110] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0111] Determine whether the task type exists in the task type library;

[0112] If the task type exists, then the step of obtaining the prompt chain template corresponding to the task type is executed;

[0113] If the task type does not exist, an auxiliary prompt chain template is obtained, and the auxiliary prompt chain template and the image are used as the input parameters. The preprocessing model is then controlled to generate the target text corresponding to the multi-task text generation instruction.

[0114] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0115] The historical target text generated by the preprocessing model is obtained, and the correlation between the historical figures and historical images corresponding to the historical target text is learned.

[0116] Migrate the association to the task type and the image to build a generic cue chain template.

[0117] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0118] Based on the task type, search for historical similar task types corresponding to the task type in the historical target text generated by the preprocessing model;

[0119] Obtain the historical prompt chain template corresponding to the historical similar task type, and construct the leading prompt chain template based on the historical prompt chain template.

[0120] Optionally, the processor 1001 may invoke a multitasking-based text generation program stored in the memory 1004, and further perform the following operations:

[0121] The image and the prompt chain template are used as input parameters of the preprocessing model to identify the corresponding text description of the image;

[0122] The text description is filled into the prompt chain template to generate the target text.

[0123] In addition, to achieve the above objectives, embodiments of the present invention also provide a terminal device, including a memory, a processor, and a multitasking text generation program stored in the memory and executable on the processor. When the processor executes the multitasking text generation program, it implements the multitasking text generation method as described above.

[0124] Furthermore, to achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium storing a multi-task-based text generation program, which, when executed by a processor, implements the multi-task-based text generation method as described above.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0130] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0131] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A text generation method based on multi-task, characterized in that, The multi-task-based text generation method includes the following steps: Receive a multi-task text generation instruction and obtain the task type and image corresponding to the multi-task text generation instruction; Determine whether the task type exists in the task type library, wherein the task type library contains various previously executed task types, and each task type has a corresponding prompt chain template; If the task type does not exist, the historical target text generated by the preprocessing model is obtained, and the correlation between the historical figures and historical images corresponding to the historical target text is learned. Migrate the correlation to the task type and the image to construct a generic cue chain template; Based on the task type, search for historical similar task types corresponding to the task type in the historical target text generated by the preprocessing model; Obtain the historical prompt chain template corresponding to the historical similar task type, and construct the leading prompt chain template based on the historical prompt chain template. The historical similar task type is determined according to the similarity value of each task type. The similarity is judged by the target of the task or the features of the image. The auxiliary cue chain template and the image are used as input parameters, and the preprocessing model is controlled to generate the target text corresponding to the multi-task text generation instruction. The auxiliary cue chain template includes the general cue chain template and the leading cue chain template.

2. The text generation method based on multi-task as described in claim 1, characterized in that, Before the step of receiving the multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction, the method further includes: Receive a multi-task training dataset and train the preprocessed model; Obtain test data and test the trained preprocessed model; The preprocessing model that has passed the test is deployed to a multi-task text generation platform, waiting to receive the multi-task text generation instructions.

3. The text generation method based on multi-task as described in claim 2, characterized in that, The step of receiving the multi-task training dataset and training the preprocessed model includes: Receive a multi-task training dataset, the multi-task training dataset including images, labels corresponding to the images, and cue chain statements corresponding to the images; The multi-task training dataset is input into the first pre-trained model to obtain the image-text similarity matrix; The image-text similarity matrix is ​​permuted to obtain a one-dimensional vector, and the one-dimensional vector is used as the input of the second pre-trained model to obtain the descriptive statement of the image; The description and the label are fused and corrected to form image cue chain data.

4. The text generation method based on multi-task as described in claim 1, characterized in that, The steps of receiving a multi-task text generation instruction and obtaining the task type and image corresponding to the multi-task text generation instruction include: Receive the multi-task text generation instruction, and obtain single task information based on the multi-task text generation instruction, wherein the single task information includes the task type corresponding to the single task and the image corresponding to the single task; Based on the correlation between the individual task information, the tasks in the multi-task text generation instruction are sorted. Output the task type and the image based on the sorting result.

5. The text generation method based on multi-task as described in claim 1, characterized in that, The steps of using the image and the cue chain template as input parameters to the preprocessing model, and controlling the preprocessing model to generate the target text corresponding to the multi-task text generation instruction, include: The image and the prompt chain template are used as input parameters of the preprocessing model to identify the corresponding text description of the image; The text description is filled into the prompt chain template to generate the target text.

6. A terminal device, characterized in that, The method includes a memory, a processor, and a multitasking text generation program stored in the memory and executable on the processor, wherein when the processor executes the multitasking text generation program, it implements the method of any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multitasking text generation program, which, when executed by a processor, implements the method described in any one of claims 1-5.

Images