Model training and instruction generation method, device, electronic device and medium

Abstract

The present application provides a model training and instruction generation method, device, electronic device and medium, which are applied to the field of computer technology. The method includes obtaining the image data, required text data and target execution instructions of a sample task; inputting the image data and required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, and the OCR module is used to identify the current image data of the sample task to obtain the local text data and location information data of the sample task; based on the target execution instruction and the predicted execution instruction, the task execution instruction generation model is trained. In this method, the accuracy of the execution instructions generated by the task execution instruction generation model can be further improved through the historical image data and the local text data and location information data identified by the OCR module.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training and instruction generation method, device, electronic device and medium. Background Art

[0002] The automatic generation of task execution instructions is often used in intelligent agent-related fields (such as robots, automated programs, etc.). It uses perception, planning, reasoning and other methods to perceive the environment and generate corresponding execution instructions according to task requirements, thereby guiding subsequent actions to complete the established task goals. As a core part of general artificial intelligence technology, the automatic generation of task execution instructions is directly related to the accuracy of subsequent action execution, thus affecting the completion of the final task.

[0003] In the related technology, there are two implementation methods for automatic generation of task execution instructions: a method based on traditional reasoning and planning and a method based on deep learning models. Deep learning models, especially large models, have demonstrated strong understanding and reasoning capabilities through pre-training of multimodal data (such as images and text). However, the generation of instructions for these large models in specific business scenarios requires fine-tuning, and the training process is complex. At the same time, the fine-tuned model still has poor understanding of images in some proprietary fields, resulting in low accuracy of the generated instructions. Summary of the invention

[0004] The embodiments of the present application provide a model training and instruction generation method, device, electronic device and medium to solve the problem of low accuracy of instructions generated in existing instruction generation methods.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0007] Acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task;

[0008] Inputting the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction;

[0009] Based on the target execution instruction and the predicted execution instruction, the task execution instruction generation model is trained.

[0010] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0011] The step of inputting the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model includes:

[0012] Inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task;

[0013] Recognize the current image data of the sample task by the OCR module to obtain local text data and position information data of the sample task;

[0014] The text processing module performs feature encoding on the required text data of the sample task and the local text data and position information data of the sample task to obtain a text feature vector of the sample task;

[0015] The image feature vector of the sample task and the text feature vector of the sample task are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the predicted execution instruction output by the task execution instruction generation model.

[0016] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0017] The step of inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task includes:

[0018] Performing feature encoding on the current picture data of the sample task and the historical picture data of the sample task respectively by the picture encoding module to obtain a current picture feature vector of the sample task and a historical picture feature vector of the sample task;

[0019] The current image feature vector of the sample task and the historical image feature vector of the sample task are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the sample task.

[0020] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0021] The step of performing feature encoding on the required text data of the sample task and the local text data and position information data of the sample task by the text processing module to obtain a text feature vector of the sample task includes:

[0022] The text encoding module performs feature encoding on the requirement text data of the sample task and the local text data and position information data of the sample task respectively, to obtain the requirement text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task;

[0023] The attention alignment module aligns the required text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task to obtain the text feature vector of the sample task.

[0024] In a second aspect, an embodiment of the present application provides an instruction generation method, the method comprising:

[0025] Acquire image data of a task to be executed and text data of requirements for the task to be executed, wherein the image data of the task to be executed includes current image data of the task to be executed and historical image data of the task to be executed;

[0026] The image data of the task to be executed and the required text data of the task to be executed are input into a trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

[0027] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0028] The step of inputting the image data of the task to be executed and the required text data of the task to be executed into the trained task execution instruction generation model to obtain the execution instruction of the task to be executed includes:

[0029] Performing feature encoding on the image data of the task to be executed by the image processing module to obtain a feature vector of the image of the task to be executed;

[0030] Recognize the required text data of the task to be executed by the OCR module to obtain the local text data and location information data of the task to be executed;

[0031] The text processing module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain a text feature vector of the task to be executed;

[0032] The image feature vector of the task to be executed and the text feature vector of the task to be executed are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the execution instruction of the task to be executed.

[0033] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0034] The image processing module of the task execution instruction generation model performs feature encoding on the image data of the task to be executed to obtain the image feature vector of the task to be executed, including:

[0035] Performing feature encoding on the current picture data of the task to be executed and the historical picture data of the task to be executed by the picture encoding module to obtain a current picture feature vector of the task to be executed and a historical picture feature vector of the task to be executed;

[0036] The current image feature vector of the task to be executed and the historical image feature vector of the task to be executed are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the task to be executed.

[0037] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0038] The text processing module of the task execution instruction generation model performs feature encoding on the requirement text data of the task to be executed and the local text data and location information data of the task to be executed to obtain the text feature vector of the task to be executed, including:

[0039] The text encoding module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain the requirement text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed;

[0040] The attention alignment module aligns the required text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed to obtain the text feature vector of the task to be executed.

[0041] In a third aspect, an embodiment of the present application further provides a model training device, the device comprising:

[0042] A first acquisition module is used to acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task;

[0043] A first processing module is used to input the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction;

[0044] The first training module is used to train the task execution instruction generation model based on the target execution instruction and the predicted execution instruction.

[0045] In a fourth aspect, an embodiment of the present application further provides an instruction generating device, the device comprising:

[0046] A second acquisition module is used to acquire image data of a task to be executed and required text data of the task to be executed, wherein the image data of the task to be executed includes current image data of the task to be executed and historical image data of the task to be executed;

[0047] The second processing module is used to input the image data of the task to be executed and the required text data of the task to be executed into the trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

[0048] In a fifth aspect, an embodiment of the present application further provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above-mentioned model training method or instruction generation method when executed by the processor.

[0049] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned model training method or instruction generation method are implemented.

[0050] In a seventh aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect or the second aspect.

[0051] The model training method of the embodiment of the present application includes obtaining image data of a sample task, requirement text data of the sample task and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task; inputting the image data of the sample task and the requirement text data of the sample task into an initial task execution instruction generation model to obtain predicted execution instructions output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, and the OCR module is used to identify the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instructions; and training the task execution instruction generation model based on the target execution instructions and the predicted execution instructions. In this method, historical image data can provide contextual references to help the task execution instruction generation model further understand the current image data. The OCR module obtains local text data and location information data by identifying the current image data, which can provide more fine-grained operation guidance for the task execution instruction generation model. Both can improve the model's environmental perception ability and detailed operation level, thereby further improving the accuracy of the execution instructions generated by the task execution instruction generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1This is one of the flow charts of the model training method provided in the embodiment of the present application;

[0054] Figure 2 This is the second flowchart of the model training method provided in the embodiment of the present application;

[0055] Figure 3 is a flow chart of the instruction generation method provided in an embodiment of the present application;

[0056] Figure 4 is a structural diagram of a model training device provided in an embodiment of the present application;

[0057] Figure 5 is a structural diagram of an instruction generation device provided in an embodiment of the present application;

[0058] Figure 6 It is one of the structural diagrams of the electronic device provided in the embodiment of the present application;

[0059] Figure 7 This is the second structural diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] This application embodiment provides a model training method. Figure 1 , Figure 1 is a flow chart of the model training method provided in the embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0062] Step 101: Acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task;

[0063] In this step, the current image data of the sample task can be understood as the image acquired when the system is about to perform a certain action in a specific task environment. This image reflects the immediate visual information during the task execution; the historical image data of the sample task can be understood as the previously recorded image, which is used to provide contextual reference to help understand the current task and operation method; the requirement text data of the sample task can be understood as the task content that needs to be completed; the target execution instructions of the sample task can be understood as a series of steps that need to be performed to actually implement the sample task.

[0064] For example, in a business scenario where an intelligent robot needs to place items in a warehouse according to a new arrival order:

[0065] The current image data may be that the robot is in front of the shelf, and the manipulator has approached the target item;

[0066] Historical image data can be video screenshots of the robot's path and operating equipment when it was previously carrying other items;

[0067] The demand text data may be to take item 1 and item 2 to packaging area B;

[0068] The target execution instruction is to first move to shelf A to get item 1, then go to shelf C to get item 2, and finally put it into packaging area B.

[0069] Step 102: input the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction;

[0070] In this step, the task instruction generation model of the present application is different from the task instruction generation model in the related art in that the task instruction generation model of the present application includes an optical character recognition (OCR) module. The OCR module can recognize the current image data of the sample task to obtain the local text data and position information data of the sample task, thereby providing more fine-grained operation guidance and assisting in generating predictive execution instructions.

[0071] Continuing with the example of the above steps, by using the OCR module to recognize the current image data, we can obtain the identification text of "item 1" and the coordinate position of "item 1" in the current image, as well as the identification text of "item 2" and the coordinate position of "item 2" in the current image.

[0072] Step 103: Based on the target execution instruction and the predicted execution instruction, the task execution instruction generation model is trained.

[0073] In this step, during the training process, by comparing the predicted execution instructions output by the model with the actual target execution instructions, the model parameters can be adjusted to reduce differences and errors, thereby improving the accuracy and effectiveness of the model.

[0074] In one embodiment, image data of a sample task, required text data of the sample task, and target execution instructions of the sample task are obtained, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task; the image data of the sample task and the required text data of the sample task are input into an initial task execution instruction generation model to obtain predicted execution instructions output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, and the OCR module is used to identify the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instructions; the task execution instruction generation model is trained based on the target execution instructions and the predicted execution instructions.

[0075] In this implementation, historical image data can provide contextual references to help the task execution instruction generation model further understand the current image data. The OCR module obtains local text data and location information data by identifying the current image data, which can provide more fine-grained operation guidance for the task execution instruction generation model. Both of these can improve the model's environmental perception ability and detailed operation level, thereby further improving the accuracy of the execution instructions generated by the task execution instruction generation model.

[0076] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0077] The step of inputting the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model includes:

[0078] Inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task;

[0079] Recognize the current image data of the sample task by the OCR module to obtain local text data and position information data of the sample task;

[0080] The text processing module performs feature encoding on the required text data of the sample task and the local text data and position information data of the sample task to obtain a text feature vector of the sample task;

[0081] The image feature vector of the sample task and the text feature vector of the sample task are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the predicted execution instruction output by the task execution instruction generation model.

[0082] In one embodiment, see Figure 2 , the image data of the sample task is input into the task execution instruction generation model, and the image data of the sample task is feature encoded through the image processing module to obtain the image feature vector, which mathematically expresses the core content of the image, such as important image attributes such as shape, color, and texture. As the basis for subsequent calculations, the image feature vector can participate in the decision-making and prediction of the model. They refine the image data and convert it into an information form that the model can directly process.

[0083] The OCR module is used to recognize the current image data of the sample task. The OCR module scans the input current image data to find the text information contained therein. This process involves converting the optical signals in the image into digital characters. For example, if there are written slogans, words on labels, characters displayed on the screen, etc. in the image, OCR will automatically detect and record these contents. All the text recognized in the image is extracted to form the so-called "local text data", which may include text information related to the current task such as object labels and instruction manual fragments. In addition to recognizing text characters, the OCR module also records the position of each recognized text in the image, which refers to the coordinates or area information where the text is located. With the help of position information, the model can understand the position of each text data relative to the entire image, which is very important for generating execution instructions.

[0084] The text processing module performs detailed analysis and processing on the required text data of the sample task and the local text data and location information data of the sample task. After the encoding process, these text information are converted into a mathematical form, namely the "text feature vector". The feature encoding can capture the semantics, context, relationship and location information of the text, making it a format that the model can understand and further process.

[0085] The VL-Adapter module aligns the image feature vector of the sample task with the text feature vector of the sample task, unifying the image features and text features into a common representation. This process ensures that the information from the image and the information from the text can be seamlessly combined with each other, thus providing a basis for generating instructions. The role of the decoder is to convert the aligned and comprehensive feature representation into a specific output, namely the predicted execution instruction. It uses integrated visual and language information to generate precise task guidance steps.

[0086] In this implementation, by combining image data and text data, the model can obtain a more comprehensive understanding of the task, thereby further improving the accuracy of the instructions generated by the task instruction generation model.

[0087] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0088] The step of inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task includes:

[0089] Performing feature encoding on the current picture data of the sample task and the historical picture data of the sample task respectively by the picture encoding module to obtain a current picture feature vector of the sample task and a historical picture feature vector of the sample task;

[0090] The current image feature vector of the sample task and the historical image feature vector of the sample task are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the sample task.

[0091] In one embodiment, continue to refer to Figure 2 The image processing module includes an image encoding module, a self-attention module, and a projection module. The image encoding module performs feature encoding on the current image data C of the sample task and the historical image data 1, 2, ... N of the sample task, respectively, to obtain the current image feature vector I of the sample task C And the historical image feature vectors I1, I2, ... I N The current image feature vector I for the sample task C And the historical image feature vectors I1, I2, ... I N , after the weighted summation pooling strategy, it is input into the projection module for mapping to obtain the final image feature vector. The specific process can be seen in the following formula (1):

[0092]

[0093] Among them, β represents the weighting parameter, I n Represents the input image feature vector, I poolrepresents the output image feature vector, index C represents the index of the current image feature vector, and 1...N represents the index of the historical image feature vector. Since different historical images contribute differently to the understanding of current needs, a self-attention mechanism is introduced here. Through parameter learning, a greater weight is given to images with greater contribution, and vice versa. For details, see the following formulas (2) and (3):

[0094] α n =Vf(WI n +b)+k (2)

[0095]

[0096] Among them, V, W, b and k are model parameters obtained through model training, and f is the activation function.

[0097] β is the weighting parameter.

[0098] In this implementation, by introducing a self-attention mechanism to calculate the contribution weight of the historical image feature vector, it is possible to fully extract useful historical information, thereby guiding the model to generate more accurate instructions.

[0099] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0100] The step of performing feature encoding on the required text data of the sample task and the local text data and position information data of the sample task by the text processing module to obtain a text feature vector of the sample task includes:

[0101] The text encoding module performs feature encoding on the requirement text data of the sample task and the local text data and position information data of the sample task respectively, to obtain the requirement text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task;

[0102] The attention alignment module aligns the required text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task to obtain the text feature vector of the sample task.

[0103] In one embodiment, the text processing module includes a language encoding module and an attention alignment module. The text encoding module performs feature encoding on the required text data of the sample task and the local text data and position information data of the sample task, respectively, to obtain the required text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task. Figure 2, Query (Q) represents the required text feature vector, Key (K) and Value (V) are the local text encoding vector and the position encoding vector respectively. Then the attention alignment module is used to align the required text feature vector with the local text encoding vector and the position encoding vector to obtain the final text feature vector. For details, please refer to the following formula (4):

[0104]

[0105] In this implementation, by encoding the required text data and local text data and their location information separately, rich information can be extracted from different sources. This multi-source information fusion can more comprehensively capture task requirements and environmental characteristics. Attention alignment enables the model to effectively focus on the most relevant information fragments, which is conducive to the subsequent generation of more accurate instructions.

[0106] The present application embodiment provides a method for generating an instruction. Figure 3 , Figure 3 is a flow chart of the instruction generation method provided in an embodiment of the present application, such as Figure 3 As shown, the following steps are included:

[0107] Step 301: Acquire image data of a task to be executed and required text data of the task to be executed, wherein the image data of the task to be executed includes current image data of the task to be executed and historical image data of the task to be executed;

[0108] Step 302: input the image data of the task to be executed and the required text data of the task to be executed into a trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

[0109] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0110] The step of inputting the image data of the task to be executed and the required text data of the task to be executed into the trained task execution instruction generation model to obtain the execution instruction of the task to be executed includes:

[0111] Performing feature encoding on the image data of the task to be executed by the image processing module to obtain a feature vector of the image of the task to be executed;

[0112] Recognize the required text data of the task to be executed by the OCR module to obtain the local text data and location information data of the task to be executed;

[0113] The text processing module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain a text feature vector of the task to be executed;

[0114] The image feature vector of the task to be executed and the text feature vector of the task to be executed are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the execution instruction of the task to be executed.

[0115] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0116] The image processing module of the task execution instruction generation model performs feature encoding on the image data of the task to be executed to obtain the image feature vector of the task to be executed, including:

[0117] Performing feature encoding on the current picture data of the task to be executed and the historical picture data of the task to be executed by the picture encoding module to obtain a current picture feature vector of the task to be executed and a historical picture feature vector of the task to be executed;

[0118] The current image feature vector of the task to be executed and the historical image feature vector of the task to be executed are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the task to be executed.

[0119] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0120] The text processing module of the task execution instruction generation model performs feature encoding on the requirement text data of the task to be executed and the local text data and location information data of the task to be executed to obtain the text feature vector of the task to be executed, including:

[0121] The text encoding module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain the requirement text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed;

[0122] The attention alignment module aligns the required text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed to obtain the text feature vector of the task to be executed.

[0123] The instruction generation method of the embodiment of the present application can refer to the relevant description of the embodiment of predicting instruction generation in the model training method. In order to avoid repeated description, this embodiment will not be repeated, and the same beneficial effects can be achieved.

[0124] See also Figure 4 , Figure 4 is a structural diagram of a model training device provided by another embodiment of the present application. Figure 4 As shown, the model training device 400 includes:

[0125] A first acquisition module 401 is used to acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task;

[0126] A first processing module 402 is used to input the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction;

[0127] The first training module 403 is used to train the task execution instruction generation model based on the target execution instruction and the predicted execution instruction.

[0128] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0129] The first processing module comprises:

[0130] A first encoding unit, configured to input the image data of the sample task into the task execution instruction generation model, perform feature encoding on the image data of the sample task through the image processing module, and obtain an image feature vector of the sample task;

[0131] A first recognition unit, configured to recognize the current image data of the sample task through the OCR module to obtain local text data and position information data of the sample task;

[0132] A second encoding unit, configured to perform feature encoding on the required text data of the sample task and the local text data and position information data of the sample task through the text processing module to obtain a text feature vector of the sample task;

[0133] The first decoding unit is used to align the image feature vector of the sample task and the text feature vector of the sample task through the VL-Adapter module and then input them into the decoder module for decoding to obtain the predicted execution instruction output by the task execution instruction generation model.

[0134] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0135] The first encoding unit is specifically used for:

[0136] Performing feature encoding on the current picture data of the sample task and the historical picture data of the sample task respectively by the picture encoding module to obtain a current picture feature vector of the sample task and a historical picture feature vector of the sample task;

[0137] The current image feature vector of the sample task and the historical image feature vector of the sample task are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the sample task.

[0138] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0139] The second encoding unit is specifically used for:

[0140] The text encoding module performs feature encoding on the requirement text data of the sample task and the local text data and position information data of the sample task respectively, to obtain the requirement text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task;

[0141] The attention alignment module aligns the required text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task to obtain the text feature vector of the sample task.

[0142] See also Figure 5 , Figure 5 FIG. 1 is a structural diagram of an instruction generation device provided by another embodiment of the present application. Figure 5 As shown, the instruction generating device 500 includes:

[0143] The second acquisition module 501 is used to acquire the image data of the task to be executed and the required text data of the task to be executed, wherein the image data of the task to be executed includes the current image data of the task to be executed and the historical image data of the task to be executed;

[0144] The second processing module 502 is used to input the image data of the task to be executed and the required text data of the task to be executed into a trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

[0145] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0146] The second processing module comprises:

[0147] A third encoding unit, configured to perform feature encoding on the image data of the task to be executed through the image processing module to obtain a feature vector of the image of the task to be executed;

[0148] A second recognition unit is used to recognize the required text data of the task to be executed through the OCR module to obtain the local text data and position information data of the task to be executed;

[0149] a fourth encoding unit, configured to perform feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed through the text processing module to obtain a text feature vector of the task to be executed;

[0150] The second decoding unit is used to align the image feature vector of the task to be executed and the text feature vector of the task to be executed through the VL-Adapter module and then input them into the decoder module for decoding to obtain the execution instruction of the task to be executed.

[0151] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0152] The third encoding unit is specifically used for:

[0153] Performing feature encoding on the current picture data of the task to be executed and the historical picture data of the task to be executed by the picture encoding module to obtain a current picture feature vector of the task to be executed and a historical picture feature vector of the task to be executed;

[0154] The current image feature vector of the task to be executed and the historical image feature vector of the task to be executed are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the task to be executed.

[0155] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0156] The fourth encoding unit is specifically used for:

[0157] The text encoding module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain the requirement text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed;

[0158] The attention alignment module aligns the required text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed to obtain the text feature vector of the task to be executed.

[0159] The present application also provides an electronic device. Since the principle of solving the problem by the electronic device is similar to the model training method in the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated. Figure 6 As shown, the client of the embodiment of the present application includes: a processor 600, which is used to read the program in the memory 620 and execute the following process:

[0160] Acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task;

[0161] Inputting the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction;

[0162] Based on the target execution instruction and the predicted execution instruction, the task execution instruction generation model is trained.

[0163] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0164] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0165] Inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task;

[0166] Recognize the current image data of the sample task by the OCR module to obtain local text data and position information data of the sample task;

[0167] The text processing module performs feature encoding on the required text data of the sample task and the local text data and position information data of the sample task to obtain a text feature vector of the sample task;

[0168] The image feature vector of the sample task and the text feature vector of the sample task are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the predicted execution instruction output by the task execution instruction generation model.

[0169] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0170] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0171] Performing feature encoding on the current picture data of the sample task and the historical picture data of the sample task respectively by the picture encoding module to obtain a current picture feature vector of the sample task and a historical picture feature vector of the sample task;

[0172] The current image feature vector of the sample task and the historical image feature vector of the sample task are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the sample task.

[0173] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0174] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0175] The text encoding module performs feature encoding on the requirement text data of the sample task and the local text data and position information data of the sample task respectively, to obtain the requirement text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task;

[0176] The attention alignment module aligns the required text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task to obtain the text feature vector of the sample task.

[0177] The present application also provides an electronic device. Since the principle of solving the problem by the electronic device is similar to the instruction generation method in the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated. Figure 7 As shown, the client of the embodiment of the present application includes: a processor 700, which is used to read the program in the memory 720 and execute the following process:

[0178] Acquire image data of a task to be executed and text data of requirements for the task to be executed, wherein the image data of the task to be executed includes current image data of the task to be executed and historical image data of the task to be executed;

[0179] The image data of the task to be executed and the required text data of the task to be executed are input into a trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

[0180] Optionally, the task execution instruction generation model further includes an image processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module;

[0181] The processor 700 is used to read the program in the memory 720 and execute the following process:

[0182] Performing feature encoding on the image data of the task to be executed by the image processing module to obtain a feature vector of the image of the task to be executed;

[0183] Recognize the required text data of the task to be executed by the OCR module to obtain the local text data and location information data of the task to be executed;

[0184] The text processing module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain a text feature vector of the task to be executed;

[0185] The image feature vector of the task to be executed and the text feature vector of the task to be executed are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the execution instruction of the task to be executed.

[0186] Optionally, the image processing module includes an image encoding module, a self-attention module and a projection module;

[0187] The processor 700 is used to read the program in the memory 720 and execute the following process:

[0188] Performing feature encoding on the current picture data of the task to be executed and the historical picture data of the task to be executed by the picture encoding module to obtain a current picture feature vector of the task to be executed and a historical picture feature vector of the task to be executed;

[0189] The current image feature vector of the task to be executed and the historical image feature vector of the task to be executed are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the task to be executed.

[0190] Optionally, the text processing module includes a text encoding module and an attention alignment module;

[0191] The processor 700 is used to read the program in the memory 720 and execute the following process:

[0192] The text encoding module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain the requirement text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed;

[0193] The attention alignment module aligns the required text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed to obtain the text feature vector of the task to be executed.

[0194] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned model training method or instruction generation embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium, such as read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0195] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 or Figure 3 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0196] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0197] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0198] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task; Inputting the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction; Based on the target execution instruction and the predicted execution instruction, the task execution instruction generation model is trained.

2. The model training method according to claim 1, characterized in that: The task execution instruction generation model also includes a picture processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module; The step of inputting the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model includes: Inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task; Recognize the current image data of the sample task by the OCR module to obtain local text data and position information data of the sample task; Performing feature encoding on the required text data of the sample task and the local text data and position information data of the sample task through the text processing module to obtain a text feature vector of the sample task; The image feature vector of the sample task and the text feature vector of the sample task are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the predicted execution instruction output by the task execution instruction generation model.

3. The model training method according to claim 2, characterized in that: The image processing module includes an image encoding module, a self-attention module and a projection module; The step of inputting the image data of the sample task into the task execution instruction generation model, and performing feature encoding on the image data of the sample task through the image processing module to obtain the image feature vector of the sample task includes: Performing feature encoding on the current picture data of the sample task and the historical picture data of the sample task respectively by the picture encoding module to obtain a current picture feature vector of the sample task and a historical picture feature vector of the sample task; The current image feature vector of the sample task and the historical image feature vector of the sample task are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the sample task.

4. The model training method according to claim 3, characterized in that: The text processing module includes a text encoding module and an attention alignment module; The step of performing feature encoding on the required text data of the sample task and the local text data and position information data of the sample task by the text processing module to obtain a text feature vector of the sample task includes: The text encoding module performs feature encoding on the requirement text data of the sample task and the local text data and position information data of the sample task respectively, so as to obtain the requirement text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task; The attention alignment module aligns the required text feature vector of the sample task and the local text encoding vector and position encoding vector of the sample task to obtain the text feature vector of the sample task.

5. A method for generating an instruction, characterized in that: The method comprises: Acquire image data of a task to be executed and text data of requirements for the task to be executed, wherein the image data of the task to be executed includes current image data of the task to be executed and historical image data of the task to be executed; The image data of the task to be executed and the required text data of the task to be executed are input into a trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

6. The instruction generation method according to claim 5, characterized in that: The task execution instruction generation model also includes a picture processing module, a text processing module, a vision and language adapter VL-Adapter module and a decoder module; The step of inputting the image data of the task to be executed and the required text data of the task to be executed into the trained task execution instruction generation model to obtain the execution instruction of the task to be executed includes: Performing feature encoding on the image data of the task to be executed by the image processing module to obtain a feature vector of the image of the task to be executed; Recognize the required text data of the task to be executed by the OCR module to obtain the local text data and location information data of the task to be executed; The text processing module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain a text feature vector of the task to be executed; The image feature vector of the task to be executed and the text feature vector of the task to be executed are aligned through the VL-Adapter module and then input into the decoder module for decoding to obtain the execution instruction of the task to be executed.

7. The instruction generation method according to claim 6, characterized in that: The image processing module includes an image encoding module, a self-attention module and a projection module; The image processing module of the task execution instruction generation model performs feature encoding on the image data of the task to be executed to obtain the image feature vector of the task to be executed, including: Performing feature encoding on the current picture data of the task to be executed and the historical picture data of the task to be executed by the picture encoding module to obtain a current picture feature vector of the task to be executed and a historical picture feature vector of the task to be executed; The current image feature vector of the task to be executed and the historical image feature vector of the task to be executed are weighted pooled by the self-attention module and then input into the projection module for mapping to obtain the image feature vector of the task to be executed.

8. The instruction generation method according to claim 6, characterized in that: The text processing module includes a text encoding module and an attention alignment module; The text processing module of the task execution instruction generation model performs feature encoding on the requirement text data of the task to be executed and the local text data and location information data of the task to be executed to obtain the text feature vector of the task to be executed, including: The text encoding module performs feature encoding on the requirement text data of the task to be executed and the local text data and position information data of the task to be executed to obtain the requirement text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed; The attention alignment module aligns the required text feature vector of the task to be executed and the local text encoding vector and position encoding vector of the task to be executed to obtain the text feature vector of the task to be executed.

9. A model training device, characterized in that: The device comprises: A first acquisition module, used to acquire image data of a sample task, required text data of the sample task, and target execution instructions of the sample task, wherein the image data of the sample task includes current image data of the sample task and historical image data of the sample task; A first processing module is used to input the image data of the sample task and the required text data of the sample task into an initial task execution instruction generation model to obtain a predicted execution instruction output by the task execution instruction generation model, wherein the task execution instruction generation model includes an optical character recognition (OCR) module, the OCR module is used to recognize the current image data of the sample task to obtain local text data and position information data of the sample task, and the local text data and position information data of the sample task are used to assist in outputting the predicted execution instruction; The first training module is used to train the task execution instruction generation model based on the target execution instruction and the predicted execution instruction.

10. An instruction generating device, characterized in that: The device comprises: A second acquisition module is used to acquire image data of a task to be executed and required text data of the task to be executed, wherein the image data of the task to be executed includes current image data of the task to be executed and historical image data of the task to be executed; The second processing module is used to input the image data of the task to be executed and the required text data of the task to be executed into the trained task execution instruction generation model to obtain the execution instruction of the task to be executed. The task execution instruction generation model includes an OCR module. The OCR module is used to identify the current image data of the task to be executed to obtain the local text data and position information data of the task to be executed. The local text data and position information data of the task to be executed are used to assist in obtaining the execution instruction of the task to be executed.

11. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the model training method as described in any one of claims 1 to 4, or implements the steps of the instruction generation method as described in any one of claims 5 to 8.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model training method as described in any one of claims 1 to 4, or implements the steps of the instruction generation method as described in any one of claims 5 to 8 when the computer program is executed by the processor.

13. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the model training method as described in any one of claims 1 to 4, or the computer instructions, when executed by a processor, implement the steps of the instruction generation method as described in any one of claims 5 to 8.

Images