Task processing method, automatic question answering method and image processing method

Abstract

The embodiment of the invention provides a task processing method, an automatic question answering method and an image processing method. The task processing method comprises the steps of obtaining task data and task description information of a target task; the task data are input into a first task processing model, key text information corresponding to the task data is obtained, the key text information is used for describing key information in the task data, and the first task processing model is used for processing the task data; the key text information, the task data and the task description information are input into a second task processing model, a task processing result of the task data is obtained, and the second task processing model is used for processing the target task. Before the second task processing model is utilized to process the task, the professional ability of the first task processing model is utilized to assist the second task processing model, so that the openness and robustness of the second task processing model are combined with the professional ability of the first task processing model, and the accuracy of the task processing result is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a task processing method, an automatic question-answering method, and an image processing method. Background Art

[0002] With the advancement of computer technology, large models have begun to shine. Their remarkable capabilities in language understanding, generation, interaction, and reasoning have led to their widespread application in natural language processing fields such as dialogue, translation, and code generation. However, due to insufficient pre-training datasets, limitations in model structure, or other factors during the training process, large models can suffer from poor accuracy in handling tasks. Therefore, enhancing the accuracy of content generated by large models has become a particularly important issue in natural language processing.

[0003] Currently, accurate sample data is often manually constructed and used to train large models. However, when training large models with accurate sample data, the models may experience catastrophic forgetting, resulting in poor accuracy in task processing using large models. Therefore, a more accurate task processing solution is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a task processing method. One or more embodiments of this specification also relate to an automatic question-answering method, an image processing method, a task processing apparatus, an automatic question-answering apparatus, an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a task processing method is provided, including:

[0006] Obtain task data and task description information of the target task;

[0007] Inputting the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data;

[0008] The key text information, task data and task description information are input into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0009] According to a second aspect of the embodiments of this specification, an automatic question-answering method is provided, comprising:

[0010] Obtain task data and questions to be answered for the question-answering task;

[0011] Inputting the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data;

[0012] Key text information, task data and questions to be answered are input into a second task processing model to obtain an answer result for the question to be answered, wherein the second task processing model is used to process the question-answering task.

[0013] According to a third aspect of the embodiments of this specification, there is provided an image processing method, including:

[0014] Obtain the target image and task description information of the image processing task;

[0015] Inputting the target image into the first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image;

[0016] The key text information, the target image and the task description information are input into the second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0017] According to a fourth aspect of the embodiments of this specification, there is provided a task processing device, including:

[0018] A first acquisition module is configured to acquire task data and task description information of a target task;

[0019] a first input module configured to input task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data;

[0020] The second input module is configured to input key text information, task data and task description information into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0021] According to a fifth aspect of the embodiments of this specification, an automatic question-answering device is provided, comprising:

[0022] A second acquisition module is configured to acquire task data and questions to be answered for the question-answering task;

[0023] a third input module configured to input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data;

[0024] The fourth input module is configured to input key text information, task data and questions to be answered into the second task processing model to obtain answer results for the questions to be answered, wherein the second task processing model is used to process question-answering tasks.

[0025] According to a sixth aspect of the embodiments of this specification, there is provided an image processing apparatus, including:

[0026] A third acquisition module is configured to acquire a target image and task description information of an image processing task;

[0027] a fifth input module configured to input the target image into the first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image;

[0028] The sixth input module is configured to input key text information, target image and task description information into the second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0029] According to a seventh aspect of the embodiments of this specification, a computing device is provided, including:

[0030] memory and processor;

[0031] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0032] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first aspect, the second aspect, or the third aspect.

[0033] According to the ninth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first aspect, the second aspect, or the third aspect.

[0034] The task processing method provided by one embodiment of this specification includes: obtaining task data and task description information of a target task; inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; inputting the key text information, task data, and task description information into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task. By additionally utilizing the key information extraction capability of the first task processing model to assist the second task processing model before utilizing the second task processing model for task processing, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model, thereby alleviating knowledge forgetting of the second task processing model and improving the accuracy of the task processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is an architecture diagram of a task processing system provided by one embodiment of this specification;

[0036] Figure 2 This is an architecture diagram of another task processing system provided by one embodiment of this specification;

[0037] Figure 3 This is a flowchart of a task processing method provided by one embodiment of this specification;

[0038] Figure 4 This is a flow chart of an automatic question-answering method provided by one embodiment of this specification;

[0039] Figure 5 is a flowchart of an image processing method provided by one embodiment of this specification;

[0040] Figure 6 This is a flow chart of a city governance method provided by one embodiment of this specification;

[0041] Figure 7 This is a flowchart of a processing process of an image processing method provided by one embodiment of this specification;

[0042] Figure 8 This is a structural diagram of a task processing device provided by one embodiment of this specification;

[0043] Figure 9 This is a schematic diagram of the structure of an automatic question-answering device provided by one embodiment of this specification;

[0044] Figure 10 This is a schematic diagram of the structure of an image processing device provided by one embodiment of this specification;

[0045] Figure 11 This is a schematic diagram of the structure of an urban governance device provided by one embodiment of this specification;

[0046] Figure 12 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0047] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0048] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0051] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. By pre-training a large model with large-scale unlabeled corpus, a pre-trained model with more than 100 million parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large-scale language model (LLM) and a multi-modal pre-training model.

[0052] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0053] First, the terms involved in one or more embodiments of this specification are explained.

[0054] Deep Learning: Deep learning is a subfield of machine learning that is based on the concept of artificial neural networks and aims to simulate the workings of the human brain to process data and make decisions. Deep learning uses multi-layered neural network models to learn and understand complex patterns and relationships through layers of abstract representations. The core concept of deep learning is a neural network, which consists of many nodes (called neurons) that are interconnected by connections (called weights). Each node receives input from the nodes in the previous layer and passes the output to the nodes in the next layer. Through multiple levels of node connections, the neural network can learn higher-level features and abstract representations. Deep learning has achieved remarkable success in various fields, including image recognition, speech recognition, natural language processing, and recommendation systems. It can automatically learn from large amounts of data and has the ability to process unstructured and highly complex data.

[0055] Multimodal Large Language Model: A multimodal large language model combines data from multiple modalities (such as text, images, and speech) for modeling and processing. It is an extension of the large-scale language model, capable of ingesting and reasoning about multimodal information. Traditional language models typically use only text data to predict the next word or generate text. However, a multimodal large language model extends this concept by integrating information from multiple modalities to provide a more comprehensive and richer approach to language understanding and generation. The multimodal large language model combines technologies from fields such as natural language processing (NLP) and computer vision (CV). The multimodal large language model can simultaneously process data from multiple modalities, including text, images, and speech, and extract correlations between these modalities. By combining these modalities, the multimodal large language model can better understand and express semantic information, thereby achieving superior performance in multimodal tasks. The multimodal large language model has a wide range of applications in many fields, such as image description generation, visual knowledge question answering, and cross-modal retrieval. It not only provides more accurate and rich language descriptions, but also provides more comprehensive information through interaction with other modalities.

[0056] Multimodal hallucination: Multimodal hallucination occurs when responses generated by a large multimodal language model contain objects, relationships, and attributes that are not present in non-linguistic data. This can be categorized as object hallucination, relationship hallucination, and attribute hallucination.

[0057] F1 score: Also known as F-measure or F-score, the F1 score is a metric used to evaluate the performance of binary or multi-classification models. It combines the precision and recall metrics. The F1 score can be considered the harmonic mean of precision and recall, aiming to provide a single value that reflects the balance between accuracy and completeness of a model.

[0058] Large multimodal language models offer powerful and user-friendly interfaces. By supporting multimodal input, users can interact with the model in more flexible ways, enabling widespread application in many fields. However, as their application grows, hallucination issues have also emerged. Hallucinations occur when the model generates responses that are inconsistent with the input data, even contradicting the input data, or outputting content that does not exist in the input data. These hallucinations severely impact the reliability and accuracy of large multimodal language models in practical applications. For example, in the case of large multimodal language models processing images, hallucinations primarily stem from the model's inadequate understanding of the correlation between image and text, as well as inaccurate understanding of image content. The model may fail to accurately grasp image details and context, or fail to consistently match image content with generated text responses. These issues can be caused by insufficient datasets, limitations in the model architecture, or other factors during training.

[0059] Currently, existing approaches in the field of large multimodal language models primarily use large models or artificially generate hallucination-free data to fine-tune the models. However, generating hallucination-free data is expensive, and research has shown that fine-tuning large multimodal language models can easily cause the model to forget knowledge, resulting in a decrease in task processing performance rather than an improvement.

[0060] In order to reduce the hallucination problem that may occur when the large language model generates text, in the embodiments of this specification, a method for model enhancement can be adopted using a retrieval-enhanced text knowledge base. First, a retrieval-enhanced text knowledge base containing real and accurate information is artificially constructed. The retrieval-enhanced text knowledge base includes background knowledge and common sense information related to the model generation task. The retrieval-enhanced text knowledge base is then integrated into the large language model, which can be used as an additional input source or as part of the model. When generating text, the large language model can use the information of the retrieval-enhanced text knowledge base to assist in decision-making, such as obtaining relevant facts by querying the retrieval-enhanced text knowledge base, and constraining the generated content based on these facts to avoid hallucinations. By introducing the retrieval-enhanced text knowledge base, the large language model can better understand the context and generate accurate and reasonable text, thereby improving the quality and accuracy of the generated text.

[0061] Furthermore, since the retrieval-enhanced text knowledge base can only process text data and the content in the knowledge base is limited, the accuracy of the task processing results may still be poor. Therefore, the embodiment of this specification proposes a solution that combines large and small models to reduce the hallucinations of the multimodal large language model. The multimodal large language model is enhanced by using a small model with higher professionalism and accuracy, combined with the key text information output by the small model. This eliminates the need to retrain the multimodal large language model, and can significantly reduce the hallucinations of the multimodal large language model at a lower cost, thereby improving the accuracy and reliability of model processing. Specifically, task data and task description information of the target task are obtained; the task data is input into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe the key information in the task data; the key text information, task data and task description information are input into the second task processing model to obtain the task processing result of the task data, wherein the model parameters of the first task processing model are smaller than the model parameters of the second task processing model. Therefore, compared with the second task processing model, the first task processing model can be a small model, and the second task processing model can be a large model (such as a pre-trained language model, a multimodal large language model). The first task processing model can process different tasks according to the data type of the task data, such as the first task processing model can process optical character recognition tasks, image recognition tasks, image detection tasks, speech recognition tasks, etc. The first task processing model is such as a visual content detection model and a visual relationship extraction model, and the second task processing model can process text tasks. The second task processing model is such as a BERT (Bidirectional Encoder Representations from Transformers) model, a T5 (Text-to-Text Transfer Transformer) model, a LLaMA (Large Language Model Meta AI) model, and a GPT (Generative Pre-trained Transformers) model.

[0062] By additionally utilizing the key information extraction capability of the first task processing model to assist the second task processing model before utilizing the second task processing model for task processing, as an external knowledge base of the second task processing model, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the task processing results.

[0063] In this specification, a task processing method is provided. This specification also involves an automatic question-answering method, an image processing method, a task processing device, an automatic question-answering device, an image processing device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0064] See also Figure 1 , Figure 1 1 shows an architecture diagram of a task processing system provided by an embodiment of the present specification. The task processing system may include a client 100 and a server 200.

[0065] The client 100 is used to send the task data and task description information of the target task to the server 200;

[0066] The server 200 is configured to input task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; input the key text information, task data, and task description information into a second task processing model to obtain a task processing result for the task data, wherein the second task processing model is used to process the target task; and send the task processing result to the client 100;

[0067] The client 100 is also used to receive the task processing result sent by the server 200.

[0068] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used to perform task processing, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the task processing results.

[0069] See also Figure 2 , Figure 2 The following diagram illustrates the architecture of another task processing system provided by one embodiment of this specification. The task processing system may include multiple clients 100 and a server 200. The clients 100 may include end-side devices, and the server 200 may include cloud-side devices. Multiple clients 100 can establish communication connections through the server 200. In a task processing scenario, the server 200 is used to provide task processing services between multiple clients 100. Multiple clients 100 can act as senders or receivers, respectively, and communicate through the server 200.

[0070] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In the task processing scenario, users can publish data streams to the server 200 through the client 100. The server 200 generates task processing results based on the data stream and pushes the task processing results to other clients with which communication has been established.

[0071] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, or other processing before being released to the server 200.

[0072] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, Hypertext Markup Language 5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application. The client 100 can be based on the software development kit (SDK) of the corresponding service provided by the server 200, such as developed based on the real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0073] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that support background training for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server that integrates a blockchain. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0074] It is worth noting that the task processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server and thus execute the task processing methods provided in the embodiments of this specification. In other embodiments, the task processing methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0075] See also Figure 3 , Figure 3 A flowchart of a task processing method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0076] Step 302: Obtain task data and task description information of the target task.

[0077] In one or more embodiments of this specification, during task processing, task data and task description information of a target task may be obtained, and the task data may be processed based on the task description information to generate a task processing result corresponding to the target task.

[0078] Specifically, task data is the task processing object. Task data can be data of different data types. Task data can be target images, target voices, target videos, etc., where the data type can also be understood as data modality. Task data can be data of different tasks, such as data for intelligent question-answering tasks, data for urban governance tasks, data for object counting tasks, data for optical character recognition tasks, and so on. Task description information is used to describe the processing task corresponding to the task data. Task description information can be different types of information, such as task description text, task description voice, task description video, task description image, and so on. For example, if the task data is the target image corresponding to the object recognition task, the task description information can be "where is a bottle in the target image". For another example, if the target task is the target image corresponding to the optical character recognition (OCR) task, the task description information can be "what is the text in the target image".

[0079] It should be noted that, in the case where the task description information is in a non-textual mode, the task description information can be modally converted to obtain textual mode task description information, thereby facilitating the large model to perform task processing. Among them, there are multiple ways to modally convert the task description information, which can be selected based on the actual situation. The embodiments of this specification do not impose any restrictions on this. Taking the task description information in the voice mode as an example, the task description information in the voice mode can be converted into the task description information in the text mode using a speech-to-text tool. The task description information in the voice mode can also be input into the speech recognition model to obtain the task description information in the text mode.

[0080] In practical applications, there are multiple ways to obtain task data and task description information for a target task, and the specific method to be used depends on the actual situation. This specification does not impose any restrictions on this. In one possible implementation of this specification, task data and task description information of the task data sent by a user through a client can be received. In another possible implementation of this specification, task data and task description information of the task data can be read from other data acquisition devices or databases.

[0081] Step 304: Input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data.

[0082] In one or more embodiments of this specification, after obtaining the task data and task description information of the target task, the task data may be further input into the first task processing model to obtain key text information corresponding to the task data.

[0083] Specifically, the first task processing model is used to extract key information from the task data to determine the key text information corresponding to the task data. The first task processing model can be different types of first task processing models, such as an attribute first task processing model for extracting image color, a visual content detection model for extracting key object information in an image, a visual relationship extraction model for extracting object relationship information, and the like. The key information in the task data can be understood as the core data in the task data, such as visual object data in an image, speaking object data in speech, and the like. The key text information can be text information describing visual objects, text information describing speaking objects, and text information describing the coordinates of objects in an image. The key text information can be structured information or unstructured information, such as natural language text information.

[0084] In an optional embodiment of the present specification, in order to more accurately obtain key text information corresponding to the task data, a first task processing model capable of processing the task data may be determined based on the data type of the task data. That is, before inputting the task data into the first task processing model and obtaining the key text information corresponding to the task data, the following steps may be further included:

[0085] Obtaining data processing types of multiple candidate first task processing models;

[0086] Identify the type of task data and determine the data type of the task data;

[0087] A first task processing model is selected from a plurality of candidate first task processing models according to the data type and the data processing type.

[0088] Specifically, the candidate first task processing model can be a model for processing different types of data, such as a speech processing model for processing speech data, or an image processing model for processing image data. Data processing types include but are not limited to speech data types, image data types, video data types, and the like.

[0089] It should be noted that there are multiple ways to obtain the data processing types of multiple candidate first task processing models, and the specific selection depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the data processing types of multiple candidate first task processing models sent by the user through the client can be received. In another possible implementation of this specification, the data processing types of multiple candidate first task processing models can be read from other data acquisition devices or databases.

[0090] In practical applications, there are many ways to identify the type of task data and determine the data type of the task data, and the specific selection is based on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the task data can be input into a type recognition model to obtain the data type of the task data, wherein the type recognition model is trained based on multiple sample data and type labels corresponding to the multiple sample data. In another possible implementation of this specification, template data corresponding to different data types can be obtained, and the task data and target data corresponding to different data types can be matched, and the data type corresponding to the template data that matches the task data is determined as the data type of the task data.

[0091] Furthermore, after determining the data type of the task data and the data processing types of multiple candidate first task processing models, the candidate first task processing model corresponding to the data processing type identical to the data type may be determined as the first task processing model for processing the task data.

[0092] By applying the solution of the embodiments of this specification, the data processing types of multiple candidate first task processing models are obtained; the task data is identified to determine its data type; and based on the data type and data processing type, the first task processing model is selected from the multiple candidate first task processing models. This ensures that the first task processing model can accurately obtain the key text information corresponding to the task data, thereby improving the accuracy of the key text information.

[0093] Step 306: Input the key text information, task data, and task description information into the second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0094] In one or more embodiments of the present specification, task data and task description information of a target task are obtained; after the task data is input into a first task processing model and key text information in the task data is obtained, the key text information, task data, and task description information can be further input into a second task processing model to obtain a task processing result of the task data.

[0095] Specifically, the second task processing model can be a pre-trained language model or a large multimodal language model. The second task processing model is used to generate task processing results for the task data. The model parameters of the second task processing model can be larger than the model parameters of the first task processing model. Therefore, compared with the first task processing model, the second task processing model is a large model.

[0096] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used to perform task processing, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the task processing results.

[0097] In actual applications, there are multiple ways to input key text information, task data, and task description information into the second task processing model to obtain the task processing result of the task data. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the key text information, task data, and task description information can be directly input into the second task processing model to obtain the task processing result of the task data. In another possible implementation of this specification, the key text information and task description information can be fused using generated prompt information, and the fused data to be processed and task data can be input into the second task processing model to obtain the task processing result of the task data.

[0098] In an optional embodiment of the present specification, the inputting of the key text information, task data, and task description information into the second task processing model to obtain the task processing result of the task data may include the following steps:

[0099] Get the generated prompt information;

[0100] According to the generated prompt information, the key text information and task description information are integrated to obtain the data to be processed;

[0101] The data to be processed and the task data are input into the second task processing model to obtain the task processing result of the task data.

[0102] Specifically, a prompt is generated to guide the second task processing model in performing task processing. The prompt is set based on actual circumstances and is not limited in this specification. For example, the prompt may be: "Answer this question based on your knowledge." Let's see what's in the task data. The task data contains [key text information], and based on the task data, the answer to the question [task description information] is [answer].

[0103] It should be noted that, when the key text information and task description information are integrated according to the generated prompt information, the key text information and task description information can be directly filled into the corresponding positions in the generated prompt information to obtain the data to be processed; the key text information and task description information can also be filled into the corresponding positions in the generated prompt information, and the data obtained after filling can be deduplicated and the information can be rewritten to obtain more fluent and concise data to be processed.

[0104] Using the solution of the embodiments of this specification, generation prompt information is obtained; based on the generation prompt information, key text information and task description information are integrated to obtain data to be processed; the data to be processed and the task data are input into the second task processing model to obtain a task processing result for the task data. By integrating the key text information and task description information based on the generation prompt information, it is ensured that the second task processing model can normally infer and obtain the task processing result.

[0105] In an optional embodiment of the present specification, the above-mentioned steps of fusing the key text information and the task description information based on the generated prompt information and obtaining the data to be processed may also include the following steps:

[0106] When the key text information is structured information, the key text information is formatted to obtain unstructured key text information;

[0107] According to the generated prompt information, the key text information and the task description information are integrated to obtain the data to be processed, which may include the following steps:

[0108] According to the generated prompt information, the unstructured key text information and task description information are fused to obtain the data to be processed.

[0109] Specifically, structured information refers to data with a fixed format, such as the object-relationship triple <object A, relationship, object B>. Unstructured information, such as data in everyday natural languages (such as English and Chinese), lacks a fixed format, does not follow predefined data schemas, and is difficult to organize and store using columns and rows in traditional databases.

[0110] It should be noted that, in order to enable the second task processing model to more accurately implement task processing, before inputting the key text information into the second task processing model for processing, it is possible to determine whether the key text information is structured information. If the key text information is not structured information but unstructured key text information, the key text information can be directly input into the second task processing model; if the key text information is structured information, the key text information can be converted into unstructured key text information, and then the unstructured key text information can be input into the second task processing model for task processing. Of course, if the key text information is structured information, it is also possible to not convert the key text information into a format and directly input the structured key text information into the second task processing model for task processing.

[0111] In practical applications, there are many ways to convert the format of key text information and obtain unstructured key text information. The specific method to be selected depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the key text information in a structured format can be filled into a natural language conversion template to obtain unstructured key text information. In another possible implementation of this specification, the key text information in a structured format can be input into a format conversion model to obtain unstructured key text information, wherein the format conversion model is trained based on multiple sample structured information and the natural language labels corresponding to each sample structured information.

[0112] Furthermore, the implementation method of "fusing unstructured key text information and task description information based on generated prompt information to obtain data to be processed" can refer to the above-mentioned implementation method of "fusing key text information and task description information based on generated prompt information to obtain data to be processed", and the embodiments of this specification will not be repeated.

[0113] For example, assuming that the first task processing model is a visual content detection model, the task data is input into the first task processing model, and the key text information obtained in the task data is "1<0,284,133,424>, 2<0,1,441,417>, 3<308,140,638,421>", where the numbers 1, 2, and 3 represent object category identifiers.<x,y,z,t> Represents object coordinate information, where x represents the coordinate of the x-axis in space, y represents the coordinate of the y-axis in space, z represents the coordinate of the z-axis in space, and t represents the time coordinate of the event. Since the key text information is structured information, the second task processing model is unclear about what the numbers in the key text information represent. In this case, the object category identifiers in the key text information can be converted into the names of the corresponding objects (such as 1 for cup, 2 for dog, and 3 for newspaper). However, the coordinate information is more complicated after conversion to natural language. Therefore, only the object category identifiers can be converted, and the object coordinate information can be left unprocessed. The converted key text information is obtained as "There is a cup <0,284,133,424>, a dog <0,1,441,417>, and a newspaper <308,140,638,421> in the picture."

[0114] Assuming that the first task processing model is a visual relationship extraction model, the task data is input into the first task processing model, and the key text information in the task data is obtained as the object relationship triplet "1M2". Then, the identifiers of the objects and relationship categories in the object relationship triplet can be converted into the corresponding object names and relationship categories (such as 1 represents the cup, 2 represents the dog, and M represents the close relationship), and the key text information "dog is close to the cup" after format conversion can be obtained.

[0115] Applying the solution of the embodiments of this specification, when the key text information is structured, the key text information is formatted to obtain unstructured key text information. Based on the generated prompt information, the unstructured key text information is integrated with the task description information to obtain the data to be processed. By converting the results of the task data processing by the first task processing model into a linguistic form, the second task processing model can more easily understand it, thereby improving the accuracy of task processing.

[0116] In an optional embodiment of the present specification, taking the second task processing model as a pre-trained language model as an example, that is, the second task processing model includes a pre-trained language model; the above-mentioned inputting key text information, task data, and task description information into the second task processing model to obtain the task processing result of the task data may include the following steps:

[0117] Extract features from task data to obtain task coding features;

[0118] According to the data processing format of the pre-trained language model, the task encoding features are mapped to obtain the projected encoding features;

[0119] The key text information, projection coding features and task description information are input into the pre-trained language model to obtain the task processing results of the task data.

[0120] It should be noted that there are many ways to extract features from task data, and the specific selection is based on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, feature (embedding) extraction can be performed by deep learning, that is, the neural network architecture is used to automatically learn feature representation from task data. For example, if the task data is a target image, a convolutional neural network (CNN) can be used to extract high-level abstract features from the target image to obtain task coding features. In another possible implementation of this specification, feature extraction can be performed on task data by an autoencoder (AE) to obtain task coding features.

[0121] In practical applications, according to the data processing format of the pre-trained language model, the task coding features are mapped, and there are many ways to obtain the projection coding features, which are selected according to the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the task coding features can be mapped using a deep hashing method to convert the task coding features into projection coding features that are similar to the text features. In another possible implementation of this specification, the task coding features can be mapped through a dual encoder structure, and two parallel encoders are used to process the task coding features and text input respectively. Ultimately, through pre-training and fine-tuning, the data of two different modalities can coexist and coexist in the same semantic space, thereby realizing effective mapping of task coding features to text features in the space.

[0122] By applying the solution of the embodiments of this specification, feature extraction is performed on the task data to obtain task coding features; the task coding features are mapped according to the data processing format of the pre-trained language model to obtain projection coding features; the key text information, projection coding features and task description information are input into the pre-trained language model to obtain the task processing results of the task data, and the key text information is additionally utilized to alleviate the knowledge forgetting of the second task processing model and improve the accuracy of the task processing results.

[0123] In another optional embodiment of the present specification, taking the second task processing model as a multimodal large language model as an example, that is, the second task processing model includes an encoding unit, a pre-trained language model, and a projection unit, and the projection unit is used to connect the encoding unit and the pre-trained language model; the above-mentioned inputting key text information, task data, and task description information into the second task processing model to obtain the task processing result of the task data may include the following steps:

[0124] The task data is encoded by the encoding unit to obtain the task encoding features;

[0125] Through the projection unit, the task coding features are mapped according to the data processing format of the pre-trained language model to obtain the projected coding features;

[0126] After the pre-trained language model, the projection coding features, key text information and task description information are processed to obtain the task processing results of the task data.

[0127] Specifically, the structure of the multimodal large language model consists of a multimodal encoding unit, a pre-trained language model, and a projector connecting the encoding unit and the pre-trained language model, wherein the encoding unit is a non-language modality encoding unit, such as a vision encoder.

[0128] By applying the solution of the embodiments of this specification, since the second task processing model includes an encoding unit, a pre-trained language model and a projection unit, the task data is encoded by the encoding unit to obtain task encoding features; the task encoding features are mapped by the projection unit according to the data processing format of the pre-trained language model to obtain projection encoding features; the projection encoding features, key text information and task description information are processed by the pre-trained language model to obtain the task processing results of the task data, without the need for separate feature extraction and feature mapping, thereby improving the task processing efficiency.

[0129] It should be noted that after obtaining the task processing results of the task data, the task processing results can be sent to the client so that the client can display the task processing results to the user. Furthermore, the task processing results can also be sent to the client based on the user's display requirement information so that the client can display the task processing results to the user. Among them, the display requirement information represents the user's demand for viewing the task processing results. The display requirement information includes but is not limited to displaying only the task processing results, displaying the task processing results and key text information. The display requirement information is specifically set according to the actual needs of the user, and the embodiments of this specification do not impose any restrictions on this.

[0130] In an optional embodiment of the present specification, after obtaining a task processing result, if the user is dissatisfied with the task processing result, the user may send feedback information based on the task processing result. After inputting the key text information, task data, and task description information into the second task processing model and obtaining the task processing result of the task data, the following steps may also be included:

[0131] generating optimization prompt information in response to feedback information sent by the user regarding the second task processing model, wherein the optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model;

[0132] Receive model optimization data sent by the user based on the optimization prompt information, and adjust model parameters of the second task processing model according to the model optimization data.

[0133] Specifically, the feedback information can include information regarding the accuracy of the second task processing model, such as "Model is inaccurate," or feedback regarding the applicable domain of the second task processing model, such as "Can it handle tasks in the XXX domain?" The model optimization data can include an accurate optimization sample set or the optimization domain of the model, depending on the specific circumstances and is not limited in this specification.

[0134] It should be noted that there are multiple ways to generate optimization prompt information in response to the feedback information sent by the user for the second task processing model. The specific selection depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation method of this specification, the pre-set optimization prompt information can be directly obtained, such as "I am very sorry to have brought you inaccurate information. Please point out the specific inaccuracies or provide the correct answers to relevant questions. I will correct and optimize my answers as soon as possible to better serve you." In another possible implementation method of this specification, the feedback information can be type-identified to determine the information type of the feedback information, and the information type can be further matched with the prompt type of each prompt information in the prompt information library, and the prompt information with the same prompt type and information type can be determined as the optimization prompt information.

[0135] Furthermore, after the optimization prompt information is generated, the optimization prompt information can be sent to the client so that the client displays the optimization prompt information to the user. After receiving the model optimization data sent by the user based on the optimization prompt information, if the model optimization data is an accurate optimization sample set, the optimization sample set can be directly used to fine-tune the second task processing model to obtain an updated second task processing model. If the model optimization data is the optimization field of the model, such as the XXX field, the sample set of the XXX field can be obtained, and the sample set of the XXX field can be used to fine-tune the second task processing model to obtain an updated second task processing model. Among them, the process of fine-tuning the second task processing model is the same as the training process of the second task processing model, and will not be repeated in the embodiment of this specification.

[0136] Using the solution of the embodiments of this specification, in response to feedback information sent by a user regarding the second task processing model, optimization prompt information is generated, wherein the optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model; the model optimization data sent by the user based on the optimization prompt information is received, and the model parameters of the second task processing model are adjusted according to the model optimization data. After receiving the feedback information sent by the user regarding the second task processing model, the model optimization data is obtained through an interactive guidance method, and the parameters of the second task processing model are optimized, making the second task processing model more accurate, while also improving interactivity with the user and increasing user satisfaction.

[0137] In an optional embodiment of the present specification, before inputting the task data into the first task processing model and obtaining the key text information corresponding to the task data, the following steps may be further included:

[0138] Acquire multiple sample task data, wherein the sample task data carries key text labels;

[0139] Inputting a plurality of sample task data into a first task processing model to obtain predicted text information corresponding to the plurality of sample task data respectively;

[0140] Calculate the information extraction loss value based on the key text label and predicted text information;

[0141] According to the information extraction loss value, the model parameters of the first task processing model are adjusted to obtain the trained first task processing model.

[0142] Specifically, the training method of the first task processing model is supervised training, that is, the sample task data carries real key text labels, and the key text labels are the extraction targets of the first task processing model, which are used to guide the training process of the first task processing model. The method of obtaining multiple sample task data can be to read a large amount of sample task data carrying key text labels from other data acquisition devices or databases. It can also be to receive a large amount of sample task data carrying key text labels input by the user. The method of obtaining multiple sample task data is selected according to the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0143] It should be noted that there are many functions for calculating the information extraction loss value, such as cross entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, logarithmic loss function, etc. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0144] In actual applications, according to the information extraction loss value, the stopping conditions for adjusting the model parameters of the first task processing model include but are not limited to the information extraction loss value being less than or equal to a preset threshold and the number of iterations reaching a preset number of iterations, wherein the preset threshold and the preset number of iterations are selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this.

[0145] In one possible implementation of this specification, after calculating the information extraction loss value, the information extraction loss value is compared with a preset threshold. Specifically, if the information extraction loss value is greater than the preset threshold, it indicates that the difference between the key text label and the predicted text information is large, and the first task processing model has poor predictive ability for the sample task data. In this case, the model parameters of the first task processing model can be adjusted, and the process returns to the step of inputting multiple sample task data into the first task processing model to obtain the predicted text information corresponding to the multiple sample task data. The first task processing model is then trained until the information extraction loss value is less than or equal to the preset threshold, indicating that the difference between the key text label and the predicted text information is small, and the preset stopping condition is met, thereby obtaining a trained first task processing model.

[0146] In another possible implementation of this specification, in addition to comparing the information extraction loss value with a preset threshold, the number of iterations may also be used to determine whether the current first task processing model has been trained. Specifically, if the information extraction loss value is greater than the preset threshold, the model parameters of the first task processing model are adjusted, and the process returns to the step of inputting multiple sample task data into the first task processing model to obtain predicted text information corresponding to each of the multiple sample task data. Training of the first task processing model continues until the preset number of iterations is reached, at which point iterations are terminated to obtain a fully trained first task processing model.

[0147] Using the solution of the embodiments of this specification, an information extraction loss value is calculated based on the key text labels and the predicted text information. This information extraction loss value is compared with a preset stopping condition. If the preset stopping condition is not met, the first task processing model is trained continuously until the preset stopping condition is met, completing the training and obtaining the first task processing model. By continuously adjusting the model parameters of the first task processing model, the resulting first task processing model can be made more accurate.

[0148] The following combined Figure 4 , taking the application of the task processing method provided in this specification in the intelligent question-answering scenario as an example, the task processing method is further explained, wherein the question-answering task can be a voice question-answering task, an image question-answering task, or a video question-answering task. Figure 4 A flowchart of an automatic question-answering method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0149] Step 402: Obtain task data and questions to be answered for the question-answering task.

[0150] Step 404: Input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data.

[0151] Step 406: Input the key text information, task data and the question to be answered into the second task processing model to obtain the answer result of the question to be answered, wherein the second task processing model is used to process the question-answering task.

[0152] It should be noted that the implementation of step 402 to step 406 is the same as the implementation of step 302 to step 306 described above, and will not be described in detail in this embodiment of the specification.

[0153] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used to perform task processing, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the response results.

[0154] See also Figure 5 , Figure 5 The flowchart of an image processing method provided by one embodiment of this specification is shown, which specifically includes the following steps:

[0155] Step 502: Obtain the target image and task description information of the image processing task.

[0156] Step 504: Input the target image into the first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image.

[0157] Step 506: Input the key text information, the target image and the task description information into the second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0158] It should be noted that the implementation of steps 502 to 506 is the same as that of steps 302 to 306 above, and will not be described in detail in this embodiment. Image processing tasks include but are not limited to image recognition tasks, image object detection tasks, and image segmentation tasks.

[0159] For example, assuming that the task corresponding to the target image is an object counting task, and the task description information of the target image is "Mark where there is a bottle in the image", by inputting the target image into the first task processing model, the key text information corresponding to the target image is obtained as "You can see bottles, bottles, bottles, bottles, bottles, bottles, bottles, bottles at the YYY position of the picture", and the key text information, target image and task description information are input into the second task processing model, and the task processing result of the target image is obtained as "There are eight bottles at the YYY position of the target image".

[0160] Assume that the task corresponding to the target image is an optical character recognition task, and the task description information of the target image is "What is the text in the target image?". By inputting the target image into the first task processing model, the key text information corresponding to the target image is obtained. The key text information, target image and task description information are input into the second task processing model, and the task processing result of the target image is "The text in the target image is "Happy New Year"."

[0161] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used for image processing, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the image processing results.

[0162] In an optional embodiment of the present specification, the first task processing model includes a visual content detection model and / or a visual relationship extraction model;

[0163] Inputting the target image into the first task processing model to obtain key text information corresponding to the target image may include the following steps:

[0164] Inputting the target image into a visual content detection model to obtain object category text and object location text of key objects in the target image; and / or,

[0165] The target image is input into the visual relationship extraction model to obtain the object relationship text between key objects in the target image.

[0166] Specifically, the visual content detection model can be understood as a small detection model, such as DETR (Detection Transformer), YOLO (You Only Look Once), DINO (DETR with Improved deNoising angle boxes), etc. The visual relationship extraction model can be understood as a scene graph generation model.

[0167] It should be noted that different first-task processing models generally have different structures, but the functions performed by the same type of first-task processing models are the same. For example, the function of the visual content detection model is to detect which objects are in the target image and where they are located; the function of the visual relationship extraction model is to extract the relationship between objects in the target image (such as position relationship, action relationship).

[0168] By applying the solution of the embodiments of this specification, the target image is input into a visual content detection model to obtain object category text and object position text of key objects in the target image; and / or, the target image is input into a visual relationship extraction model to obtain object relationship text between key objects in the target image, thereby achieving additional use of object category text, object position text, and object relationship text to alleviate knowledge forgetting in the second task processing model and improve the accuracy of the task processing results.

[0169] The following combined Figure 6 , taking the application of the task processing method provided in this specification in the urban governance scenario as an example, the task processing method is further explained. Figure 6 A flow chart of a city governance method provided by an embodiment of this specification is shown, which specifically includes the following steps:

[0170] Step 602: Obtain the target city image and task description information of the urban governance task.

[0171] Step 604: Input the target city image into the first task processing model to obtain key text information corresponding to the target city image, wherein the key text information is used to describe key visual information in the target city image. The first task processing model is used to process the target city image.

[0172] Step 606: Input the key text information, target city image and task description information into the second task processing model to obtain the governance event recognition result of the target city image, wherein the second task processing model is used to process the city governance task.

[0173] It should be noted that the implementation of step 602 to step 606 is the same as the implementation of step 302 to step 306 described above, and will not be described in detail in this embodiment of the specification.

[0174] In practical applications, governance tasks in urban governance scenarios include, but are not limited to, geological exploration and environmental assessment. In urban governance scenarios, the first and second task processing models can be leveraged to build services for describing urban governance events, judging rule-based events, and instantly reporting urban images (for governance events), thereby improving the accuracy of identifying urban governance events. Furthermore, the first task processing model can enhance the second task processing model's ability to understand professional geological maps in a fine-grained manner. It automatically performs image recognition, structural analysis, and mineral identification, while also understanding and answering user questions about geological structure, history, and potential resources. In geological exploration, it helps geologists identify possible mineral deposits and predict the distribution of oil and gas fields, thereby optimizing exploration plans and reducing unnecessary physical exploration efforts. In environmental assessment, it analyzes soil and rock images to quickly identify environmental risks and guide environmental protection and disaster prevention efforts. Experiments have shown that combining the second task processing model with the first task processing model, without retraining the second task processing model, improves the F1 score between output and annotation by 135%, and the large model score by 86%.

[0175] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used for urban governance, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the governance event identification results.

[0176] See also Figure 7 , Figure 7 The flowchart of an image processing method provided by one embodiment of this specification is shown, which specifically includes:

[0177] Visual Evidence Extraction: The image input to the second task processing model (multimodal large language model) is simultaneously input to the first task processing model (visual small model) for visual knowledge extraction. For the key text information of the object in the image, the image is input to the visual content detection model, and the output is the object category and object location in the image, that is, {label}.<x1,y1,x2,y2> ; For the key text information of the relationship between objects in the image, the visual relationship extraction model is used, and the output is the object relationship triples existing in the image, that is,<subject,relation,object> ,Integrate the object category, object location and object relationship triples to obtain key text information.

[0178] Visual evidence prompting: The key text information and task description text obtained in the first step are integrated into a generated prompt information to obtain the data to be processed; the integrated data to be processed and the image are then input into the multimodal large language model to obtain the model's response.

[0179] It should be noted that the first task processing model's understanding of the image (such as the object category and object position output by the visual content detection model) is used as additional visual knowledge. This visual knowledge is used as a reference and combined with the task description text before the second task processing model performs reasoning and is added to the generated prompt information. The second task processing model performs normal reasoning and obtains the answer of the second task processing model. This can significantly reduce the hallucinations of the second task processing model and improve the accuracy of the answer.

[0180] For example, Figure 7 As shown, the target image and task description text of the target task are obtained: "Based on the target image, answer the question: Is the cup near the dog? The answer (yes or no) is [answer]", where (yes or no) is used to prompt the second task processing model to output the answer, which can be "yes" or "no". The target image is input into the small visual model to obtain the key text information in the target image; the generated prompt information "Answer this question based on knowledge. Let's see what is in the image. The image contains [key text information]" is obtained. Based on the generated prompt information, the key text information and task description information are fused and then input into the multimodal large language model together with the target image, obtaining the answer "yes".

[0181] In the embodiment of this specification, a multimodal large language model and a small visual model are combined to combine the openness and robustness of the multimodal large language model with the professional knowledge of the small visual model, and the specific domain knowledge of the small visual model is used to assist the large multimodal language model as a visual plug-in knowledge base of the large multimodal language model. Since there is no need to change the model structure and model parameters of the large multimodal language model, only a low cost is required to significantly reduce the degree of hallucination of the large multimodal language model and improve the accuracy and reliability of the model's answers. Therefore, it can be applied to multimodal large language models of various structures and types, and has strong robustness and generalization, while also maximizing the alleviation of model knowledge forgetting. In addition, the small visual model can be flexibly customized according to the application scenario. Using small visual models of different types and sizes can reduce the multimodal hallucination phenomenon of the large multimodal language model to varying degrees, and can enhance the model in other fine-grained tasks that require higher model understanding capabilities (such as object counting, optical character recognition, etc.).

[0182] Corresponding to the above-mentioned task processing method embodiment, this specification also provides a task processing device embodiment, Figure 8 FIG1 shows a schematic diagram of the structure of a task processing device provided by an embodiment of this specification. Figure 8 As shown, the device includes:

[0183] A first acquisition module 802 is configured to acquire task data and task description information of a target task;

[0184] A first input module 804 is configured to input task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data;

[0185] The second input module 806 is configured to input key text information, task data and task description information into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0186] Optionally, the device also includes: a screening module configured to obtain data processing types of multiple candidate first task processing models; perform type identification on the task data to determine the data type of the task data; and screen out the first task processing model from multiple candidate first task processing models based on the data type and the data processing type.

[0187] Optionally, the second input module 806 is further configured to obtain generated prompt information; fuse the key text information and task description information according to the generated prompt information to obtain data to be processed; input the data to be processed and the task data into the second task processing model to obtain the task processing result of the task data.

[0188] Optionally, the device also includes: a conversion module, configured to convert the format of the key text information when the key text information is structured information to obtain unstructured key text information; a second input module 806, further configured to fuse the unstructured key text information and task description information based on the generated prompt information to obtain data to be processed.

[0189] Optionally, the second task processing model includes a pre-trained language model; the second input module 806 is further configured to perform feature extraction on the task data to obtain task coding features; map the task coding features according to the data processing format of the pre-trained language model to obtain projection coding features; input the key text information, projection coding features and task description information into the pre-trained language model to obtain the task processing results of the task data.

[0190] Optionally, the second task processing model includes an encoding unit, a pre-trained language model and a projection unit, and the projection unit is used to connect the encoding unit and the pre-trained language model; the second input module 806 is further configured to encode the task data through the encoding unit to obtain task encoding features; through the projection unit, map the task encoding features according to the data processing format of the pre-trained language model to obtain projection encoding features; through the pre-trained language model, process the projection encoding features, key text information and task description information to obtain the task processing results of the task data.

[0191] Optionally, the device also includes: an adjustment module, configured to generate optimization prompt information in response to feedback information sent by the user regarding the task processing model, wherein the optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model; receive the model optimization data sent by the user based on the optimization prompt information, and adjust the model parameters of the second task processing model according to the model optimization data.

[0192] Optionally, the device also includes: an information adjustment model training module, configured to obtain multiple sample task data, wherein the sample task data carries key text labels; input the multiple sample task data into the first task processing model to obtain predicted text information corresponding to the multiple sample task data; calculate the information extraction loss value based on the key text labels and the predicted text information; adjust the model parameters of the first task processing model based on the information extraction loss value to obtain a trained first task processing model.

[0193] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used to perform task processing, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the task processing results.

[0194] The above is a schematic scheme of a task processing device of this embodiment. It should be noted that the technical scheme of the task processing device and the technical scheme of the task processing method described above are of the same concept. For details not described in detail in the technical scheme of the task processing device, please refer to the description of the technical scheme of the task processing method described above.

[0195] Corresponding to the above-mentioned automatic question-answering method embodiment, this specification also provides an automatic question-answering device embodiment, Figure 9 FIG1 shows a schematic diagram of the structure of an automatic question-answering device provided by an embodiment of this specification. Figure 9 As shown, the device includes:

[0196] The second acquisition module 902 is configured to acquire task data and questions to be answered for the question-answering task;

[0197] The third input module 904 is configured to input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data;

[0198] The fourth input module 906 is configured to input key text information, task data and questions to be answered into the second task processing model to obtain a response result to the question to be answered, wherein the second task processing model is used to process the question-answering task.

[0199] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model before the second task processing model is used for automatic question answering, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the answer results.

[0200] The above is a schematic diagram of an automatic question-answering device according to this embodiment. It should be noted that the technical solution of this automatic question-answering device and the technical solution of the automatic question-answering method described above are based on the same concept. For details not described in detail in the technical solution of the automatic question-answering device, please refer to the description of the technical solution of the automatic question-answering method described above.

[0201] Corresponding to the above-mentioned image processing method embodiment, this specification also provides an image processing device embodiment, Figure 10 FIG. 1 shows a schematic diagram of the structure of an image processing device provided by an embodiment of this specification. Figure 10 As shown, the device includes:

[0202] The third acquisition module 1002 is configured to acquire a target image and task description information of an image processing task;

[0203] a fifth input module 1004 configured to input the target image into the first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image;

[0204] The sixth input module 1006 is configured to input key text information, target image and task description information into the second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0205] Optionally, the first task processing model includes a visual content detection model and / or a visual relationship extraction model; the fifth input module 1004 is further configured to input the target image into the visual content detection model to obtain object category text and object position text of key objects in the target image; and / or, input the target image into the visual relationship extraction model to obtain object relationship text between key objects in the target image.

[0206] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the visual first task processing model before the second task processing model is used for image processing, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the task processing results.

[0207] The above is a schematic diagram of an image processing device according to this embodiment. It should be noted that the technical solution of the image processing device and the technical solution of the above-mentioned image processing method are based on the same concept. For details not described in detail in the technical solution of the image processing device, please refer to the description of the technical solution of the above-mentioned image processing method.

[0208] Corresponding to the above-mentioned city governance method embodiment, this specification also provides a city governance device embodiment, Figure 11 FIG1 shows a schematic diagram of the structure of a city management device provided by an embodiment of this specification. Figure 11 As shown, the device includes:

[0209] The fourth acquisition module 1102 is configured to acquire a target city image and task description information of the urban governance task;

[0210] a seventh input module 1104 configured to input the target city image into the first task processing model to obtain key text information corresponding to the target city image, wherein the key text information is used to describe key visual information in the target city image, and the first task processing model is used to process the target city image;

[0211] The eighth input module 1106 is configured to input key text information, target city image and task description information into the second task processing model to obtain the governance event recognition result of the target city image, wherein the second task processing model is used to process the city governance task.

[0212] By applying the solution of the embodiments of this specification, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the visual first task processing model before the second task processing model is used for urban governance, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the governance event recognition results.

[0213] The above is a schematic diagram of an urban governance device according to this embodiment. It should be noted that the technical solution of the urban governance device and the technical solution of the urban governance method described above are of the same concept. For details not described in detail in the technical solution of the urban governance device, please refer to the description of the technical solution of the urban governance method described above.

[0214] Figure 12 12 shows a block diagram of a computing device according to an embodiment of the present disclosure. Components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0215] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0216] In one embodiment of the present specification, the above components of the computing device 1200 and Figure 12 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 12 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0217] Computing device 1200 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1200 may also be a mobile or stationary server.

[0218] The processor 1220 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned task processing method, automatic question-answering method, or image processing method.

[0219] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and image processing method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or image processing method.

[0220] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned task processing method, automatic question-answering method, or image processing method.

[0221] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and image processing method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or image processing method.

[0222] An embodiment of the present specification further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned task processing method, automatic question-answering method, or image processing method.

[0223] The above is a schematic diagram of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product is based on the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and image processing method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or image processing method.

[0224] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0225] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0226] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0227] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0228] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A task processing method, comprising: Obtain task data and task description information of the target task; Inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; The key text information, the task data and the task description information are input into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

2. The method according to claim 1, before inputting the task data into the first task processing model and obtaining key text information corresponding to the task data, further comprising: Obtaining data processing types of multiple candidate first task processing models; Performing type identification on the task data to determine the data type of the task data; A first task processing model is selected from the plurality of candidate first task processing models according to the data type and the data processing type.

3. The method according to claim 1, wherein inputting the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data comprises: Get the generated prompt information; According to the generated prompt information, the key text information and the task description information are integrated to obtain data to be processed; The data to be processed and the task data are input into a second task processing model to obtain a task processing result of the task data.

4. The method according to claim 3, before fusing the key text information and the task description information based on the generated prompt information to obtain the data to be processed, further comprising: In the case where the key text information is structured information, performing format conversion on the key text information to obtain unstructured key text information; The step of fusing the key text information and the task description information according to the generated prompt information to obtain data to be processed includes: According to the generated prompt information, the unstructured key text information and the task description information are fused to obtain data to be processed.

5. The method according to any one of claims 1 to 4, wherein the second task processing model comprises a pre-trained language model; The step of inputting the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data includes: Performing feature extraction on the task data to obtain task coding features; Mapping the task coding features according to the data processing format of the pre-trained language model to obtain projected coding features; The key text information, the projection coding features and the task description information are input into the pre-trained language model to obtain a task processing result of the task data.

6. The method according to any one of claims 1 to 4, wherein the second task processing model comprises an encoding unit, a pre-trained language model, and a projection unit, wherein the projection unit is configured to connect the encoding unit and the pre-trained language model; The step of inputting the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data includes: The encoding unit encodes the task data to obtain a task encoding feature; The projection unit maps the task coding features according to the data processing format of the pre-trained language model to obtain a projected coding feature; The projected coding features, the key text information and the task description information are processed by the pre-trained language model to obtain a task processing result of the task data.

7. The method according to claim 1, further comprising: inputting the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data; generating optimization prompt information in response to feedback information sent by the user regarding the second task processing model, wherein the optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model; The model optimization data sent by the user based on the optimization prompt information is received, and the model parameters of the second task processing model are adjusted according to the model optimization data.

8. The method according to claim 1, before inputting the task data into the first task processing model and obtaining key text information corresponding to the task data, further comprising: Acquire a plurality of sample task data, wherein the sample task data carries key text labels; Inputting the plurality of sample task data into a first task processing model to obtain predicted text information corresponding to the plurality of sample task data respectively; Calculating an information extraction loss value based on the key text label and the predicted text information; A loss value is extracted according to the information, and model parameters of the first task processing model are adjusted to obtain a trained first task processing model.

9. An automatic question-answering method, comprising: Obtain task data and questions to be answered for the question-answering task; Inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; The key text information, the task data and the question to be answered are input into a second task processing model to obtain an answer result for the question to be answered, wherein the second task processing model is used to process the question-answering task.

10. An image processing method, comprising: Obtain the target image and task description information of the image processing task; Inputting the target image into a first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image; The key text information, the target image and the task description information are input into a second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

11. The method according to claim 10, wherein the first task processing model comprises a visual content detection model and / or a visual relationship extraction model; Inputting the target image into the first task processing model to obtain key text information corresponding to the target image includes: Inputting the target image into a visual content detection model to obtain object category text and object position text of key objects in the target image; and / or, The target image is input into a visual relationship extraction model to obtain object relationship text between key objects in the target image.

12. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 or claim 9 or any one of claims 10 to 11 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8 or claim 9 or any one of claims 10 to 11.

14. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8 or claim 9 or any one of claims 10 to 11.

Images