Visual language multi-task processing method and device, equipment and medium

Abstract

The invention discloses a visual language multi-task processing method and device, equipment and a medium. The method comprises the following steps: acquiring a to-be-processed image; performing image coding on the to-be-processed image to obtain image features; task demand information is received, a corresponding task instruction is constructed according to the task demand information and the image feature, the task instruction comprises the image feature, a task type and instruction content, and the task instruction is used for prompting the large language model to execute a corresponding visual language task on the image feature based on the task type and the instruction content; and inputting the task instruction into the large language model to obtain language description information of the image features under the visual language task. The corresponding task instruction is constructed based on the task requirement and the to-be-processed image to prompt the large language model to execute the corresponding visual language task, so that visual language multi-task processing can be realized under the same framework, and the visual language processing efficiency of various images and the adaptability to different task requirements under the vehicle insurance scene are improved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and is applied to auto insurance scenarios, and in particular to a visual language multi-tasking processing method, device, equipment and medium. Background Art

[0002] In the auto insurance scenario, the automatic inspection site usually involves taking photos of the car accident scene and the customer's ID information, while the manual inspection site usually involves photos of the surveyor's work. The collected images involve a lot of visual information, such as vehicle color, vehicle logo, model, damaged parts, etc., as well as important text information, such as license plate number, frame number, driver's license, and the logo text on the surveyor's uniform. By identifying these visual information, the processing speed and accuracy of the auto insurance inspection and loss assessment process can be significantly improved.

[0003] Traditional visual recognition systems are integrated with various specialized modules. For example, the automobile component damage segmentation (or detection) model is responsible for identifying the damaged parts, the license plate recognition model is responsible for detecting and identifying the license plate number, the vehicle model classification model is responsible for identifying the vehicle model, and the clothing classification model is responsible for identifying whether the surveyor is dressed in a standard manner, etc. In addition, for various task requirements, it is necessary to re-collect and annotate samples and train a new module from scratch, which makes the entire system multi-functional and complex, increasing the difficulty of deployment and maintenance, and making it difficult to adapt to the diverse needs of visual information recognition in auto insurance scenarios. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a visual language multi-tasking processing method, device, equipment and medium that can be applied to financial technology or other related fields. Its main purpose is to realize visual language multi-tasking processing in the same framework, improve processing efficiency and adaptability to different task requirements.

[0005] The technical solution of the present invention is as follows:

[0006] A first aspect of the present invention provides a visual language multi-task processing method, comprising:

[0007] Get the image to be processed;

[0008] Performing image coding on the image to be processed to obtain image features;

[0009] receiving task requirement information, and constructing corresponding task instructions according to the task requirement information and the image features, wherein the task instructions include the image features, the task type and the instruction content, and the task instructions are used to prompt the large language model to perform the corresponding visual language task on the image features based on the task type and the instruction content;

[0010] The task instruction is input into the large language model to obtain language description information of the image feature under the visual language task.

[0011] In one embodiment, performing image encoding on the image to be processed to obtain image features includes:

[0012] Encoding the image to be processed by a preset image encoder and outputting a corresponding feature vector;

[0013] The dimensions of all feature vectors are adjusted through the trained linear projection layer to generate image features of the image to be processed.

[0014] In one embodiment, encoding the image to be processed by a preset image encoder and outputting a corresponding feature vector includes:

[0015] Dividing the image to be processed into a plurality of image blocks of preset sizes, and performing embedding conversion on each image block to generate an image block embedding vector;

[0016] Perform position encoding on all image blocks to generate image block position vectors;

[0017] The sequence of the image block embedding vector and the image block position vector is input into a Transformer encoder for feature extraction, and a feature vector of each image block is output.

[0018] In one embodiment, the dimension adjustment of all feature vectors by the trained linear projection layer to generate image features of the image to be processed includes:

[0019] Connect several adjacent eigenvectors as a group, and convert all eigenvectors into several eigenvector groups;

[0020] The feature vector group is projected into a single embedding vector in the feature space of the large language model to generate image features of the image to be processed.

[0021] In one embodiment, the receiving task requirement information and constructing corresponding task instructions according to the task requirement information and the image features include:

[0022] Receiving task requirement information, performing task identification on the task requirement information, and obtaining a task type and instruction content of a visual language task to be performed, wherein the task type includes a visual question-answering task and an object detection task;

[0023] The image features, task types and instruction contents are added to the designated positions in the preset multi-task instruction template to construct corresponding task instructions.

[0024] In one embodiment, when the task type is a visual question answering task, inputting the task instruction into the large language model to obtain language description information of the image feature under the visual language task specifically includes:

[0025] The task instruction is input into the large language model, and the large language model is used to perform feature recognition on the image features under the context prompt of the instruction content to obtain a response text corresponding to the instruction content.

[0026] In one embodiment, when the task type is a target detection task, inputting the task instruction into the large language model to obtain language description information of the image feature under the visual language task specifically includes:

[0027] Inputting the task instruction into the large language model, using the large language model to perform target detection on the image features under the contextual prompt of the instruction content, and determining the position of the target rectangular frame;

[0028] The corresponding rectangular frame coordinate values ​​or position description text are output according to the position of the target rectangular frame.

[0029] A second aspect of the present invention provides a visual language multi-task processing device, comprising:

[0030] An acquisition module, used for acquiring an image to be processed;

[0031] An image processing module, used for performing image coding on the image to be processed to obtain image features;

[0032] An instruction construction module, used to receive task requirement information, and construct corresponding task instructions according to the task requirement information and the image features, wherein the task instructions include the image features, the task type and the instruction content, and the task instructions are used to prompt the large language model to perform the corresponding visual language task on the image features based on the task type and the instruction content;

[0033] The task processing module is used to input the task instruction into the large language model to obtain the language description information of the image feature under the visual language task.

[0034] A third aspect of the present invention provides a computer device, comprising at least one processor; and

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned visual language multi-tasking processing method.

[0037] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned visual language multi-tasking processing method.

[0038] Beneficial effects: The present invention discloses a visual language multi-task processing method, device, equipment and medium. Compared with the prior art, the embodiment of the present invention obtains an image to be processed; performs image encoding on the image to be processed to obtain image features; receives task requirement information, and constructs corresponding task instructions according to the task requirement information and image features. The task instructions include image features, task types and instruction contents. The task instructions are used to prompt the large language model to perform corresponding visual language tasks on the image features based on the task type and instruction contents; the task instructions are input into the large language model to obtain the language description information of the image features under the visual language task. By constructing corresponding task instructions based on task requirements and the image to be processed to prompt the large language model to perform corresponding visual language tasks, visual language multi-task processing in the auto insurance scenario can be realized under the same framework, simplifying the differences in processing models between different visual language tasks while also reducing maintenance costs, so that different tasks can interact with the model in the same way, improving the visual language processing efficiency of various types of images in the auto insurance scenario, and also being conducive to adapting to changing task requirements and improving the response speed and accuracy to new tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the scheme in the present invention, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flow chart of a visual language multi-task processing method provided by an embodiment of the present invention;

[0041] Figure 2 A flow chart of step S101 in the visual language multi-task processing method provided by an embodiment of the present invention;

[0042] Figure 3 A flow chart of step S201 in the visual language multi-task processing method provided by an embodiment of the present invention;

[0043] Figure 4 A flow chart of step S202 in the visual language multi-task processing method provided by an embodiment of the present invention;

[0044] Figure 5 A flow chart of step S103 in the visual language multi-task processing method provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of the functional modules of a visual language multi-task processing device provided by an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0048] In the auto insurance scenario, the automatic inspection site usually involves taking photos of the car accident scene and the customer's ID information, while the manual inspection site usually involves photos of the surveyor's work. The collected images involve a lot of visual information, such as vehicle color, vehicle logo, model, damaged parts, etc., as well as important text information, such as license plate number, frame number, driver's license, and the logo text on the surveyor's uniform. By identifying these visual information, the processing speed and accuracy of the auto insurance inspection and loss assessment process can be significantly improved.

[0049] Traditional visual recognition systems are integrated with various specialized modules. For example, the automobile component damage segmentation (or detection) model is responsible for identifying the damaged parts, the license plate recognition model is responsible for detecting and identifying the license plate number, the vehicle model classification model is responsible for identifying the vehicle model, and the clothing classification model is responsible for identifying whether the surveyor is dressed in a standard manner, etc. In addition, for various task requirements, it is necessary to re-collect and annotate samples and train a new module from scratch, which makes the entire system multi-functional and complex, increasing the difficulty of deployment and maintenance, and making it difficult to adapt to the diverse needs of visual information recognition in auto insurance scenarios.

[0050] In order to solve the above problems, the present invention proposes a visual language multi-task processing method, see Figure 1 , Figure 1 The flowchart of an embodiment of the visual language multi-task processing method provided by the present invention. The system architecture of the visual language multi-task processing method provided by this embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0051] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0052] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0053] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0054] It should be noted that the visual language multi-task processing method provided by the embodiment of the present invention can generally be executed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the visual language multi-task processing device provided by the embodiment of the present invention can also be set in the first terminal device, the second terminal device or the third terminal device.

[0055] Alternatively, the visual language multi-task processing method provided by the embodiment of the present invention may also be generally executed by a server. Accordingly, the visual language multi-task processing device provided by the embodiment of the present invention may generally be arranged in a server. The visual language multi-task processing method provided by the embodiment of the present invention may also be executed by a server or server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Accordingly, the visual language multi-task processing device provided by the embodiment of the present invention may also be arranged in a server or server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.

[0056] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0057] like Figure 1 As shown, the visual language multi-task processing method provided in this embodiment specifically includes the following steps:

[0058] S101: Obtain an image to be processed.

[0059] In this embodiment, in the auto insurance scenario, especially the inspection and damage assessment scenario, the various visual semantic understanding functions in the scene are realized through the visual language model that is pre-trained and deployed in the production environment. The surveyor can improve the efficiency of inspection and damage assessment by collecting the images to be processed and uploading them to the visual language model to perform the corresponding visual language tasks.

[0060] Specific images to be processed can be, for example, vehicle damage photos, that is, images of damaged parts of the vehicle caused by the accident, such as scratches, dents, and broken parts, which can be used to assess the degree of damage and repair costs; vehicle full-view photos, that is, images of the overall appearance of the vehicle collected from different angles, which can be used to identify vehicle model, color and other information, and as a reference for damage assessment; accident scene environment photos, that is, the location and surrounding environment of the accident, which can be used to analyze the cause of the accident and determine responsibility; document photos, that is, photos of the owner's driver's license, vehicle registration and other documents, which can be used to verify the owner's identity and the legality of the vehicle; surveyor work photos, that is, images of the surveyor's clothing and equipment used, which can be used to verify the standardization of the survey process. In the process of auto insurance survey, the corresponding images to be processed are collected based on different process requirements and uploaded to the uniformly deployed visual language model, making the auto insurance survey process simpler and more efficient.

[0061] S102: Perform image encoding on the image to be processed to obtain image features.

[0062] In this embodiment, the visual language model first performs image encoding processing on the acquired image to be processed, thereby converting the input image to be processed into corresponding image features. The image features are specifically a set of feature vectors, which are highly compressed representations of the original image and contain key visual information in the image. For example, hierarchical structural information such as the edge, texture, and shape of the image can be captured, laying the foundation for subsequent processing and analysis.

[0063] In one embodiment, Figure 2 As shown, step S102 includes:

[0064] S201, encoding the image to be processed by a preset image encoder, and outputting a corresponding feature vector;

[0065] S202: Dimensionally adjust all feature vectors through a trained linear projection layer to generate image features of the image to be processed.

[0066] In this embodiment, the model architecture of the pre-deployed visual language model includes an image encoder, a linear projection layer, and a large language model, wherein the linear projection layer and the large language model are trainable, and the parameters of the image encoder remain frozen, so that when training the multi-tasking visual language model, you can focus on adjusting the parameters of the linear projection layer and the large language model to make them more in line with the task requirements in the auto insurance scenario.

[0067] After the visual language model is trained and deployed, the input image to be processed is first encoded through the image encoder to output the corresponding feature vector, which is a feature vector that can reflect the global and local features of the image. Since the embedding space of the feature vector output by the image encoder may be inconsistent with the feature space of the large language model, all feature vectors are further dimensionalized through the trained linear projection layer after encoding, and the encoded feature vectors are mapped from the high-dimensional image space to the low-dimensional feature space to generate image features for the image to be processed, so that it not only meets the input requirements of the large language model, but also retains the most useful features for the task while reducing resource consumption, thereby improving the generalization ability of the model.

[0068] In one embodiment, Figure 3 As shown, step S201 includes:

[0069] S301, dividing the image to be processed into a plurality of image blocks of preset sizes, and performing embedding conversion on each image block to generate an image block embedding vector;

[0070] S302, performing position encoding on all image blocks to generate image block position vectors;

[0071] S303: Input the sequence of the image block embedding vector and the image block position vector into a Transformer encoder for feature extraction, and output a feature vector for each image block.

[0072] In this embodiment, features of the image are extracted through an image encoder based on the Transformer architecture. Transformer is a language model based on the Seq2Seq structure, and its architecture includes an encoder and a decoder. The self-attention mechanism is used to capture the dependencies between different positions in the input sequence, so that all positions in the sequence can be processed simultaneously, and the global information is modeled without the need to process each element sequentially like the traditional RNN, thereby greatly improving the computational efficiency and being able to solve the long-distance dependency problem in the RNN.

[0073] Specifically, the input image to be processed is first divided into a series of fixed-size image blocks, and these image blocks are regarded as elements in a sequence, so that these sequences can be processed by the Transformer model to capture the global dependencies between image blocks. Each image block after segmentation is embedded and transformed to generate a series of image block embedding vectors, which capture the local features of the image block; at the same time, since the Transformer architecture itself does not have the ability to process the position information of elements in the sequence, all image blocks are also position-encoded to generate image block position vectors to retain the spatial position information of each image block; then the sequence containing the image block embedding vector and the image block position vector is input into the Transformer encoder, and feature extraction is performed through the multi-head self-attention mechanism and the fully connected feedforward neural network to capture the relationship between different image block embedding vectors, and then extract and refine the input embedding vector to generate a more advanced and abstract feature representation, and obtain a feature vector that can reflect the complex pattern and structure of the image block, which serves as the basis for various subsequent visual language tasks.

[0074] In one embodiment, Figure 4 As shown, step S202 includes:

[0075] S401, connecting a plurality of adjacent feature vectors as a group, and converting all feature vectors into a plurality of feature vector groups;

[0076] S402: Project the feature vector group into a single embedding vector in the feature space of the large language model to generate image features of the image to be processed.

[0077] In this embodiment, in order to improve the model processing efficiency and the matching degree of the feature space, the feature vector output by the image encoder is dimensionally adjusted through the trained linear projection layer. Specifically, several adjacent feature vectors are connected as a group, and all feature vectors are converted into several feature vector groups. For example, every 4 adjacent feature vectors in the embedding space are spliced ​​or connected in a certain order (usually a time or space order) to form a longer feature vector group. By connecting adjacent feature vectors, the visual information of the local area can be integrated, which helps to capture the local pattern and structure in the image; then, the feature vector group connected in the embedding space is projected into a single embedding vector in the feature space of the large language model, that is, the original feature vector is mapped to a feature space suitable for processing by the large language model, so that the visual information can be compatible with the language model. By compressing the information of the feature vector group, the complexity of the model input is reduced, which helps to improve the operation efficiency of the model. This information can also be effectively converted into a form suitable for language model processing, laying a foundation for subsequent visual language tasks.

[0078] S103, receiving task requirement information, and constructing corresponding task instructions according to the task requirement information and the image features, wherein the task instructions include the image features, task types and instruction contents, and the task instructions are used to prompt the large language model to perform corresponding visual language tasks on the image features based on the task type and instruction contents.

[0079] In this embodiment, the auto insurance agency or surveyor can input corresponding task requirement information on the visual interactive interface according to their own needs, that is, different users pay attention to different content. For example, the auto insurance agency only pays attention to the license plate number, damage site, and damage type, but is not interested in the color of the car, whether the surveyor is wearing work clothes, or whether he is wearing a work ID. In this case, the task requirement information of "need to complete the identification of the license plate number, damage site, and damage type" can be entered, while the surveyor may be more concerned about the extraction of the owner's ID information for verification, so the task requirement information of "need to extract ID information" can be entered.

[0080] Based on different task requirement information, the requirements of the current survey task are defined, and then the corresponding task instructions are generated by combining the task requirement information with the image features. The generated task instructions can guide the large language model to understand the specific requirements of the current task, so as to perform the corresponding visual language tasks on the image features based on the task type and the instruction content. Since the large language model itself has a relatively good semantic understanding ability, it can adapt well to different types of visual language tasks. By building corresponding task instructions through different task requirements, it is possible to achieve visual language multi-task processing in the auto insurance scenario under the same framework. And because the large language has a certain zero shot or few shot capability, for new requirements, there is no need to collect and annotate a large number of samples for training like traditional methods. The training can be completed by constructing task instructions through existing samples and fine-tuning the large language model, which has the advantage of reducing sample collection and annotation costs.

[0081] In one embodiment, Figure 5 As shown, step S103 includes:

[0082] S501, receiving task requirement information, performing task identification on the task requirement information, and obtaining a task type and instruction content of a visual language task to be performed, wherein the task type includes a visual question-answering task and an object detection task;

[0083] S502: Add the image features, task type and instruction content to a specified position in a preset multi-task instruction template to construct a corresponding task instruction.

[0084] In this embodiment, a multi-task instruction template is preset in the auto insurance scenario to realize the construction of task instructions. After receiving the task requirement information, the task requirement information is task identified to obtain the task type and instruction content of the visual language task to be executed, wherein the task type includes visual question-answering tasks and target detection tasks. For example, when the task requirement information is "need to identify the color of the car", the current task type is identified as a visual question-answering task, and the instruction content is "please identify the color of the car"; when the task requirement information is "need to point out the location of the loss", the current task type is identified as a target detection task, and the instruction content is "please detect and point out the location of the damage", thereby meeting the requirements of different visual language tasks. It can be understood that in other embodiments, corresponding task types can also be set according to other task requirements, such as image classification tasks, image segmentation tasks, etc., and this embodiment does not limit this.

[0085] After identifying the task type and instruction content, call the preset multi-task instruction template, which includes image features, task type and instruction content. For example, the multi-task instruction template is: [INST] <imagefeature>[Task Identifier]Instruction[ / INST], where [INST] is the instruction start token and [ / INST] is the instruction end token. and are the start and end tokens of image features respectively, <imagefeature>is the image feature, that is, the output feature of the linear projection layer, [Task Identifier] is the task type, for example, vqa (visual question answer) represents the visual question answering task, detection represents the target detection task, and Instruction is the specific instruction content. By adding the image features, task type and instruction content to the specified position in the preset multi-task instruction template, the corresponding task instructions can be efficiently constructed. The template method ensures the consistency and standardization of task instructions, simplifies the input processing of the model, and the preset template can reduce the time and errors of manually constructing instructions, and supports fast switching of multiple task types, so that the system can flexibly respond to different visual language tasks.

[0086] S104: Input the task instruction into the large language model to obtain language description information of the image feature under the visual language task.

[0087] In this embodiment, the big language model is a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. They can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc. In the visual language task of the auto insurance scenario, the task instructions that meet the input requirements of the big language model are input into the big language model. After receiving the task instructions, the big language model parses and processes the instructions, and performs corresponding visual language tasks on the image features based on the big language model's understanding and analysis capabilities of the image content, thereby generating accurate language description information in combination with the image features. In application scenarios such as auto insurance investigation, the output accurate language description can provide support for decision makers, thereby improving the efficiency of investigation and loss assessment.

[0088] In one embodiment, when the task type is a visual question answering task, inputting the task instruction into the large language model to obtain language description information of the image feature under the visual language task specifically includes:

[0089] The task instruction is input into the large language model, and the large language model is used to perform feature recognition on the image features under the context prompt of the instruction content to obtain a response text corresponding to the instruction content.

[0090] In this embodiment, when the task type is a visual question answering task, the value of [Task Identifier] in the task instruction is [vqa], [vqa] is used for visual question answering tasks, and the output is text, such as asking about the color of the car, asking whether there is damage, asking about the type of damage, asking about the degree of damage, asking about the license plate number, asking about the frame number, asking whether the surveyor is wearing work clothes, asking whether the surveyor is wearing work documents, etc. The task instruction is input into the large language model, and the large language model is used to perform feature recognition on the image features under the context prompt of the instruction content, that is, the large language model performs reasoning based on the input image features and task instructions, and uses its internal representation learning mechanism to understand the relationship between the question and the image content, and generates corresponding answer texts, such as descriptions of the damaged parts, descriptions of the degree of loss, descriptions of the color of the car, descriptions of the license plate number, etc., which correspond to the instruction content, thereby realizing the visual question answering task in the process of motor vehicle insurance investigation, automatically generating answers to questions raised in motor vehicle insurance investigation, reducing the workload of manual image analysis, significantly improving the efficiency of motor vehicle insurance investigation, and speeding up the claim settlement process.

[0091] In one embodiment, when the task type is a target detection task, inputting the task instruction into the large language model to obtain language description information of the image feature under the visual language task specifically includes:

[0092] Inputting the task instruction into the large language model, using the large language model to perform target detection on the image features under the contextual prompt of the instruction content, and determining the position of the target rectangular frame;

[0093] The corresponding rectangular frame coordinate values ​​or position description text are output according to the position of the target rectangular frame.

[0094] In this embodiment, when the task type is a target detection task, the value of [Task Identifier] in the task instruction is [detection]. [detection] is used for target detection tasks to detect the specific location of the target, and the output is the coordinate value of the rectangular box. <x1> <y1> <x2> <y2>Or the position description of the rectangular box, such as "the damaged part is located on the left front side of the vehicle", etc., for example, indicating the location of the damage, indicating the location of the "XXX Insurance" sign on the surveyor's car, indicating the location of the surveyor's work certificate, etc. The task instruction is input into the large language model, and the large language model is used to analyze the image features under the context prompt of the instruction content, identify the target object in the image, and determine the minimum circumscribed rectangle of the target object as the target rectangular box. According to the position of the target rectangular box, the coordinate value of the rectangular box or the position description text is generated, so as to automatically identify and locate the target object in the image, quickly generate the target detection result, and improve the efficiency of the survey process.

[0095] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0096] Further references Figure 6 , as a response to the above Figure 1 The present invention provides an embodiment of a visual language multi-task processing device, and the device embodiment is Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0097] like Figure 6 As shown, the visual language multi-task processing device 60 described in this embodiment includes:

[0098] An acquisition module 601 is used to acquire an image to be processed;

[0099] The image processing module 602 is used to perform image coding on the image to be processed to obtain image features;

[0100] An instruction construction module 603 is used to receive task requirement information, and construct corresponding task instructions according to the task requirement information and the image features, wherein the task instructions include the image features, the task type and the instruction content, and the task instructions are used to prompt the large language model to perform the corresponding visual language task on the image features based on the task type and the instruction content;

[0101] The task processing module 604 is used to input the task instruction into the large language model to obtain the language description information of the image feature under the visual language task.

[0102] The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the visual language multi-tasking execution process than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0103] In one embodiment, the acquisition module 601 includes:

[0104] An image encoding unit, used for encoding the image to be processed by a preset image encoder and outputting a corresponding feature vector;

[0105] The linear projection unit is used to adjust the dimensions of all feature vectors through the trained linear projection layer to generate image features of the image to be processed.

[0106] In one embodiment, the image encoding unit includes:

[0107] A segmentation and embedding unit, used for segmenting the image to be processed into a plurality of image blocks of preset sizes, and performing an embedding conversion on each image block to generate an image block embedding vector;

[0108] A position encoding unit, used for performing position encoding on all image blocks to generate image block position vectors;

[0109] The encoding unit is used to input the sequence of the image block embedding vector and the image block position vector into the Transformer encoder for feature extraction, and output the feature vector of each image block.

[0110] In one embodiment, the linear projection unit comprises:

[0111] A connection unit, used to connect several adjacent feature vectors as a group, and convert all feature vectors into several feature vector groups;

[0112] A projection unit is used to project the feature vector group into a single embedding vector in the feature space of the large language model to generate image features of the image to be processed.

[0113] In one embodiment, the instruction construction module 603 includes:

[0114] A task analysis unit, configured to receive task requirement information, perform task identification on the task requirement information, and obtain a task type and instruction content of a visual language task to be performed, wherein the task type includes a visual question-answering task and an object detection task;

[0115] The instruction construction unit is used to add the image features, task type and instruction content to a specified position in a preset multi-task instruction template to construct a corresponding task instruction.

[0116] In one embodiment, when the task type is a visual question answering task, the task processing module is specifically used to:

[0117] The task instruction is input into the large language model, and the large language model is used to perform feature recognition on the image features under the context prompt of the instruction content to obtain a response text corresponding to the instruction content.

[0118] In one embodiment, when the task type is a target detection task, the task processing module is specifically used to:

[0119] Inputting the task instruction into the large language model, using the large language model to perform target detection on the image features under the contextual prompt of the instruction content, and determining the position of the target rectangular frame;

[0120] The corresponding rectangular frame coordinate values ​​or position description text are output according to the position of the target rectangular frame.

[0121] In the above embodiment, the present invention discloses a visual language multi-task processing device, which obtains an image to be processed; performs image encoding on the image to be processed to obtain image features; receives task requirement information, and constructs corresponding task instructions according to the task requirement information and image features, wherein the task instructions include image features, task types, and instruction contents, and the task instructions are used to prompt the large language model to perform corresponding visual language tasks on the image features based on the task type and instruction contents; and inputs the task instructions into the large language model to obtain language description information of the image features under the visual language task. By constructing corresponding task instructions based on task requirements and the image to be processed to prompt the large language model to perform corresponding visual language tasks, visual language multi-task processing in the auto insurance scenario can be realized under the same framework, simplifying the differences in processing models between different visual language tasks while also reducing maintenance costs, so that different tasks can interact with the model in the same way, improving the visual language processing efficiency of various types of images in the auto insurance scenario, and also being conducive to adapting to changing task requirements and improving the response speed and accuracy to new tasks.

[0122] Another embodiment of the present invention provides a computer device, such as Figure 7 As shown, the computer device 70 includes:

[0123] One or more processors 701 and memory 702, Figure 7 A processor 701 is used as an example for description. The processor 701 and the memory 702 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0124] Processor 701 is used to complete various control logics of computer device 70, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component or any combination of these components. In addition, processor 701 can also be any traditional processor, microprocessor or state machine. Processor 701 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0125] The memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the visual language multi-tasking processing method in the embodiment of the present invention. The processor 701 executes various functional applications and data processing of the computer device 70 by running the non-volatile software programs, instructions and units stored in the memory 702, that is, implementing the visual language multi-tasking processing method in the above method embodiment.

[0126] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device 70, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the computer device 70 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. One or more units are stored in the memory 702, and when executed by one or more processors 701, the steps of the visual language multitasking method in any of the above-mentioned method embodiments are executed.

[0127] In the above embodiment, the present invention discloses a computer device, which obtains an image to be processed; performs image encoding on the image to be processed to obtain image features; receives task requirement information, and constructs corresponding task instructions according to the task requirement information and image features, wherein the task instructions include image features, task types and instruction contents, and the task instructions are used to prompt the large language model to perform corresponding visual language tasks on the image features based on the task type and instruction contents; and inputs the task instructions into the large language model to obtain language description information of the image features under the visual language task. By constructing corresponding task instructions based on task requirements and the image to be processed to prompt the large language model to perform corresponding visual language tasks, it is possible to realize visual language multi-task processing in the auto insurance scenario under the same framework, simplify the differences in processing models between different visual language tasks, and also reduce maintenance costs, so that different tasks can interact with the model in the same way, thereby improving the visual language processing efficiency of various types of images in the auto insurance scenario, and also being conducive to adapting to changing task requirements and improving the response speed and accuracy to new tasks.

[0128] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the visual language multi-tasking method in any of the above method embodiments are executed.

[0129] In the above embodiment, the present invention discloses a non-volatile computer-readable storage medium, which obtains an image to be processed; performs image encoding on the image to be processed to obtain image features; receives task requirement information, and constructs corresponding task instructions according to the task requirement information and image features, wherein the task instructions include image features, task types, and instruction contents, and the task instructions are used to prompt the large language model to perform corresponding visual language tasks on the image features based on the task type and instruction contents; and inputs the task instructions into the large language model to obtain language description information of the image features under the visual language task. By constructing corresponding task instructions based on task requirements and the image to be processed to prompt the large language model to perform corresponding visual language tasks, it is possible to realize visual language multi-task processing in the auto insurance scenario under the same framework, simplify the differences in processing models between different visual language tasks, and also reduce maintenance costs, so that different tasks can interact with the model in the same way, thereby improving the visual language processing efficiency of various types of images in the auto insurance scenario, and also being conducive to adapting to changing task requirements and improving the response speed and accuracy to new tasks.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0131] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0132] In summary, the present invention discloses a visual language multi-task processing method, device, equipment and medium, the method includes: obtaining an image to be processed; performing image encoding on the image to be processed to obtain image features; receiving task requirement information, constructing corresponding task instructions according to the task requirement information and image features, the task instructions including image features, task types and instruction contents, the task instructions are used to prompt the large language model to perform corresponding visual language tasks on the image features based on the task type and instruction contents; inputting the task instructions into the large language model to obtain the language description information of the image features under the visual language task. By constructing corresponding task instructions based on task requirements and the image to be processed to prompt the large language model to perform corresponding visual language tasks, the visual language multi-task processing in the auto insurance scenario can be realized under the same framework, simplifying the differences in processing models between different visual language tasks while also reducing maintenance costs, so that different tasks can interact with the model in the same way, improving the visual language processing efficiency of various types of images in the auto insurance scenario, and also being conducive to adapting to changing task requirements and improving the response speed and accuracy to new tasks.

[0133] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0134] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of the present application, they are merely used for illustration and do not represent actual use. It should be understood that the application of the present invention is not limited to the above examples, and that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes shall fall within the scope of protection of the claims attached to the present invention. < / x2> < / y1> < / x1> < / imagefeature> < / imagefeature>

Claims

1. A visual language multi-task processing method, characterized in that: include: Get the image to be processed; Performing image coding on the image to be processed to obtain image features; receiving task requirement information, and constructing corresponding task instructions according to the task requirement information and the image features, wherein the task instructions include the image features, the task type and the instruction content, and the task instructions are used to prompt the large language model to perform the corresponding visual language task on the image features based on the task type and the instruction content; The task instruction is input into the large language model to obtain language description information of the image feature under the visual language task.

2. The visual language multi-tasking processing method according to claim 1, characterized in that: The step of performing image encoding on the image to be processed to obtain image features includes: Encoding the image to be processed by a preset image encoder and outputting a corresponding feature vector; The dimensions of all feature vectors are adjusted through the trained linear projection layer to generate image features of the image to be processed.

3. The visual language multi-task processing method according to claim 2, characterized in that: The step of encoding the image to be processed by a preset image encoder and outputting a corresponding feature vector comprises: Dividing the image to be processed into a plurality of image blocks of preset sizes, and performing embedding conversion on each image block to generate an image block embedding vector; Perform position encoding on all image blocks to generate image block position vectors; The sequence of the image block embedding vector and the image block position vector is input into a Transformer encoder for feature extraction, and a feature vector of each image block is output.

4. The visual language multi-task processing method according to claim 2, characterized in that: The dimension adjustment of all feature vectors by the trained linear projection layer to generate image features of the image to be processed includes: Connect several adjacent eigenvectors as a group, and convert all eigenvectors into several eigenvector groups; The feature vector group is projected into a single embedding vector in the feature space of the large language model to generate image features of the image to be processed.

5. The visual language multi-task processing method according to claim 1, characterized in that: The receiving task requirement information and constructing corresponding task instructions according to the task requirement information and the image features includes: Receiving task requirement information, performing task identification on the task requirement information, and obtaining a task type and instruction content of a visual language task to be performed, wherein the task type includes a visual question-answering task and an object detection task; The image features, task types and instruction contents are added to the designated positions in the preset multi-task instruction template to construct corresponding task instructions.

6. The visual language multi-task processing method according to claim 5, characterized in that: When the task type is a visual question answering task, inputting the task instruction into the large language model to obtain language description information of the image feature under the visual language task specifically includes: The task instruction is input into the large language model, and the large language model is used to perform feature recognition on the image features under the context prompt of the instruction content to obtain a response text corresponding to the instruction content.

7. The visual language multi-task processing method according to claim 5, characterized in that: When the task type is a target detection task, inputting the task instruction into the large language model to obtain language description information of the image feature under the visual language task specifically includes: Inputting the task instruction into the large language model, using the large language model to perform target detection on the image features under the contextual prompt of the instruction content, and determining the position of the target rectangular frame; The corresponding rectangular frame coordinate values ​​or position description text are output according to the position of the target rectangular frame.

8. A visual language multi-task processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; An image processing module, used for performing image coding on the image to be processed to obtain image features; An instruction construction module, used to receive task requirement information, and construct corresponding task instructions according to the task requirement information and the image features, wherein the task instructions include the image features, the task type and the instruction content, and the task instructions are used to prompt the large language model to perform the corresponding visual language task on the image features based on the task type and the instruction content; The task processing module is used to input the task instruction into the large language model to obtain the language description information of the image feature under the visual language task.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the visual language multi-tasking processing method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the visual language multi-tasking method according to any one of claims 1 to 7.

Images