Task processing method, automatic question-answering method and image processing method

Abstract

The embodiments of the present disclosure provide a task processing method, an automatic question-answering method and an image processing method. The task processing method comprises: acquiring task data and task description information of a target task; inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is configured to describe key information in the task data, and the first task processing model is used for processing the task data; and inputting the key text information, the task data and the task description information into a second task processing model to obtain a task processing result for the task data, wherein the second task processing model is used for processing the target task.

Description

Technical Field of Task Processing Method, Automatic Question Answering Method and Image Processing Method

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a task processing method, an automatic question-answering method, and an image processing method.

[0002] With the advancement of computer technology, large models have begun to shine. Their remarkable capabilities in language understanding, generation, interaction, and reasoning have led to their widespread application in natural language processing fields such as dialogue, translation, and code generation. However, due to insufficient pre-training datasets, limitations in model structure, or other factors during the training process, large models may suffer from poor accuracy in handling tasks. Therefore, enhancing the accuracy of content generated by large models has become a particularly important issue in natural language processing.

[0003] Currently, accurate sample data is usually manually constructed and used to train large models. However, when using accurate sample data to train large models, the large models may experience catastrophic forgetting, resulting in poor accuracy in task processing using large models. Therefore, a task processing solution with higher accuracy is urgently needed.

[0004] In light of this, embodiments of the present disclosure provide a task processing method. One or more embodiments of the present disclosure also include an automatic question-answering method, an image processing method, a task processing apparatus, an automatic question-answering apparatus, an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of the present disclosure, a task processing method is provided, comprising: obtaining task data and task description information of a target task; inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; and inputting the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0006] According to a second aspect of an embodiment of the present disclosure, an automatic question-answering method is provided, comprising: obtaining task data and questions to be answered for a question-answering task; inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; inputting the key text information, the task data, and the questions to be answered into a second task processing model to obtain a reply result to the questions to be answered, wherein the second task processing model is used to process the question-answering task.

[0007] According to a third aspect of an embodiment of the present disclosure, an image processing method is provided, comprising: obtaining a target image and task description information of an image processing task; inputting the target image into a first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image; inputting the key text information, the target image, and the task description information into a second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0008] According to a fourth aspect of an embodiment of the present disclosure, a task processing device is provided, comprising: a first acquisition module configured to acquire task data and task description information of a target task; a first input module configured to input the task data into a first task processing model to obtain a task data corresponding to the task data; Key text information, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; a second input module is configured to input the key text information, task data and task description information into the second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0009] According to a fifth aspect of an embodiment of the present disclosure, an automatic question-answering device is provided, comprising: a second acquisition module, configured to acquire task data and questions to be answered of a question-answering task; a third input module, configured to input the task data into a first task processing model, and obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; a fourth input module, configured to input the key text information, the task data, and the questions to be answered into a second task processing model, and obtain a reply result to the question to be answered, wherein the second task processing model is used to process the question-answering task.

[0010] According to a sixth aspect of an embodiment of the present disclosure, an image processing device is provided, comprising: a third acquisition module, configured to acquire a target image and task description information of an image processing task; a fifth input module, configured to input the target image into a first task processing model, and obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image; and a sixth input module, configured to input the key text information, the target image, and the task description information into a second task processing model, and obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0011] According to a seventh aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, and when the computer program / instructions are executed by the processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0012] According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0013] According to a ninth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method provided in the first, second or third aspect above.

[0014] An embodiment of the present disclosure provides a task processing method, comprising: obtaining task data and task description information of a target task; inputting the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; inputting the key text information, task data, and task description information into a second task processing model to obtain a task processing result for the task data, wherein the second task processing model is used to process the target task. By additionally utilizing the key information extraction capability of the first task processing model to assist the second task processing model before utilizing the second task processing model for task processing, the openness and robustness of the second task processing model are combined with the professional key information extraction capability of the first task processing model, thereby alleviating knowledge forgetting of the second task processing model and improving the accuracy of the task processing results. Description of the Figures

[0015] FIG1 is an architecture diagram of a task processing system provided by one embodiment of the present disclosure;

[0016] Figure 2 is an architectural diagram of another task processing system provided by one embodiment of the present disclosure;

[0017] FIG3 is a flowchart of a task processing method provided by one embodiment of the present disclosure;

[0018] FIG4 is a flow chart of an automatic question-answering method provided by one embodiment of the present disclosure;

[0019] FIG5 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0020] FIG6 is a flow chart of a city governance method provided by one embodiment of the present disclosure;

[0021] FIG7 is a flowchart of a processing process of an image processing method provided by one embodiment of the present disclosure;

[0022] FIG8 is a schematic structural diagram of a task processing device provided by one embodiment of the present disclosure;

[0023] FIG9 is a schematic structural diagram of an automatic question-answering device provided by one embodiment of the present disclosure;

[0024] FIG10 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure;

[0025] FIG11 is a schematic structural diagram of an urban governance device provided by one embodiment of the present disclosure;

[0026] FIG12 is a block diagram of a computing device provided by an embodiment of the present disclosure.

[0027] The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art may make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below.

[0028] The terminology used in one or more embodiments of the present disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used in one or more embodiments of the present disclosure refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] It should be understood that while terms such as "first," "second," and so on may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, "first" could also be referred to as "second," and similarly, "second" could also be referred to as "first," without departing from the scope of one or more embodiments of the present disclosure. Depending on the context, the term "if" as used herein could be interpreted as "when," "when," or "in response to a determination."

[0030] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject.

[0031] In one or more embodiments of the present disclosure, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. Large models, also known as foundation models, are pre-trained using large-scale unlabeled corpora to produce pre-trained models with more than 100 million parameters. Such models are adaptable to a wide range of downstream tasks and have good generalization capabilities. Examples include large language models (LLMs) and multi-modal pre-training models.

[0032] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0033] First, the terms involved in one or more embodiments of the present disclosure are explained.

[0034] Deep Learning: Deep Learning is a subfield of machine learning that is based on artificial neural networks. The concept of a network aims to simulate the workings of the human brain to process data and make decisions. Deep learning uses multi-layered neural network models to learn and understand complex patterns and relationships through layers of abstract representations. The core concept of deep learning is the neural network, which consists of many nodes (called neurons) interconnected by connections (called weights). Each node receives input from the nodes in the previous layer and transmits output to the nodes in the next layer. Through multi-layer node connections, the neural network can learn higher-level features and abstract representations. Deep learning has achieved remarkable success in various fields, including image recognition, speech recognition, natural language processing, and recommendation systems. It can automatically learn from large amounts of data and has the ability to process unstructured and highly complex data.

[0035] Multimodal Large Language Model: A multimodal large language model combines data from multiple modalities (such as text, images, and speech) for modeling and processing. It is an extension of large-scale language models with the ability to absorb and reason about multimodal information. Traditional language models typically use only text data to predict the next word or generate sentences. However, a multimodal large language model extends this concept by integrating information from multiple modalities to provide a more comprehensive and richer approach to language understanding and generation. The multimodal large language model combines technologies from fields such as natural language processing (NLP) and computer vision (CV). The multimodal large language model can simultaneously process data from multiple modalities, including text, images, and speech, and extract correlations between these modalities. By combining these modalities, the multimodal large language model can better understand and express semantic information, thereby achieving superior performance in multimodal tasks. Multimodal large language models have a wide range of applications in many fields, such as image description generation, visual knowledge question answering, and cross-modal retrieval. It can not only provide more accurate and rich language descriptions, but also provide more comprehensive information through interaction with other modalities.

[0036] Multimodal hallucination: Multimodal hallucination occurs when responses generated by a large multimodal language model contain objects, relationships, and attributes that are not present in non-linguistic data. This can be categorized as object hallucination, relationship hallucination, and attribute hallucination.

[0037] F1 score: Also known as F-measure or F-score, the F1 score is a metric used to evaluate the performance of binary or multi-classification models. It combines precision and recall. The F1 score can be considered the harmonic mean of precision and recall, aiming to provide a single value that reflects the balance between accuracy and completeness of a model.

[0038] Large multimodal language models offer powerful and user-friendly interfaces. By supporting multimodal input, users can interact with the model in more flexible ways, resulting in widespread applications across a wide range of fields. However, as their application grows, hallucination issues have also gradually emerged. Hallucination issues occur when the model generates responses that are inconsistent with the input data, even contradicting the input data, or outputting content that does not exist in the input data. These hallucination issues seriously impact the reliability and accuracy of large multimodal language models in practical applications. For example, when large multimodal language models process images, hallucination issues primarily arise from the model's inadequate understanding of the correlation between image and text, as well as inaccurate understanding of image content. The model may fail to accurately grasp image details and context, or fail to consistently match image content with generated text responses. These issues may be due to insufficient datasets, limitations in the model architecture, or other factors during the training process.

[0039] Currently, existing approaches in the field of large multimodal language models primarily use large models or artificially generate hallucination-free data to fine-tune the models. However, generating hallucination-free data is expensive, and research has shown that fine-tuning large multimodal language models can easily lead to knowledge loss, resulting in decreased task processing performance.

[0040] To mitigate the potential for hallucinations when generating text using a large language model, embodiments of the present disclosure utilize a retrieval-enhanced text knowledge base for model enhancement. First, a retrieval-enhanced text knowledge base containing real, accurate information is manually constructed. This retrieval-enhanced text knowledge base includes background knowledge and common sense relevant to the model generation task. The retrieval-enhanced text knowledge base is then integrated into the large language model, either as an additional input source or as part of the model. When generating text, the large language model can utilize information from the retrieval-enhanced text knowledge base to assist in decision-making. For example, it can query the retrieval-enhanced text knowledge base to obtain relevant facts and use these facts to constrain the generated content to avoid hallucinations. By incorporating the retrieval-enhanced text knowledge base, the large language model can better understand context and generate accurate and reasonable text, improving the quality and accuracy of generated text.

[0041] Furthermore, since the search-enhanced text knowledge base can only process text data and the content in the knowledge base is limited, the accuracy of the task processing results may still be poor. Therefore, the embodiment of the present disclosure proposes a solution to reduce the hallucination of the multimodal large language model by combining a small model with high professionalism and accuracy. The key text information output by the small model is used to enhance the multimodal large language model, eliminating the need to retrain the multimodal large language model. At a low cost, the hallucinations of the multimodal large language model can be significantly reduced, thereby improving the accuracy and reliability of model processing. Specifically, task data and task description information of a target task are obtained; the task data is input into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data; the key text information, task data, and task description information are input into a second task processing model to obtain a task processing result of the task data, wherein the model parameters of the first task processing model are smaller than the model parameters of the second task processing model. Therefore, compared with the second task processing model, the first task processing model can be a small model, and the second task processing model can be a large model (such as a pre-trained language model or a multimodal large language model). The first task processing model can process different tasks according to the data type of the task data. For example, the first task processing model can process optical character recognition tasks, image recognition tasks, image detection tasks, speech recognition tasks, etc. The first task processing model may be a visual content detection model or a visual relationship extraction model, and the second task processing model may be a text task. The second task processing model may be a BERT (Bidirectional Encoder Representations from Transformers) model, a T5 (Text-to-Text Transfer Transformer) model, or a LLaMA (Large Language Model) model. Meta Al) model, GPT (Generative Pre-trained Transformers) model.

[0042] By utilizing the key information extraction capabilities of the first task processing model to assist the second task processing model before using the second task processing model for task processing, and serving as an external knowledge base for the second task processing model, the openness and robustness of the second task processing model are combined with the professional key information extraction capabilities of the first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of the task processing results.

[0043] The present disclosure provides a task processing method. The present disclosure also relates to an automatic question-answering method, an image processing method, a task processing apparatus, an automatic question-answering apparatus, an image processing apparatus, a computing device, and a computer-readable storage medium, each of which is described in detail in the following embodiments.

[0044] 1 , which shows an architecture diagram of a task processing system provided by an embodiment of the present disclosure. The task processing system may include a client 100 and a server 200.

[0045] The client 100 is used to send the task data and task description information of the target task to the server 200.

[0046] The server 200 is configured to input task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; input the key text information, task data, and task description information into a second task processing model to obtain a task processing result for the task data, wherein the second task processing model is used to process the target task; and send the task processing result to the client 100.

[0047] The client 100 is also used to receive the task processing result sent by the server 200.

[0048] By applying the solution of the embodiments of the present disclosure, the key information extraction capabilities of the first task processing model are additionally utilized to assist the second task processing model before the second task processing model is used for task processing. This combines the openness and robustness of the second task processing model with the specialized key information extraction capabilities of the first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of task processing results.

[0049] Referring to Figure 2, it shows an architecture diagram of another task processing system provided by one embodiment of the present disclosure. The task processing system may include multiple clients 100 and a server 200. The clients 100 may include a client device, and the server 200 may include a Yunjia 1 device. Multiple clients 100 may establish communication connections through the server 200. In a task processing scenario, the server 200 is used to provide task processing services between the multiple clients 100. The multiple clients 100 may act as senders or receivers, respectively, and communicate through the server 200.

[0050] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100. In a task processing scenario, users can publish data streams to server 200 through client 100. Server 200 generates task processing results based on the data streams and pushes the task processing results to other clients with which communication has been established.

[0051] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by the client 100 may need to undergo encoding, transcoding, compression, and other processing before being released to the server 200.

[0052] The client 100 can be a browser, an application (APP), a web application such as an H5 (HyperText Markup Languages, version 5) application, a lightweight application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be developed based on a software development kit (SDK) for the corresponding service provided by the server 200, such as a real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and rely on the device or certain applications in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0053] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that support backend training for models used by clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers or as a single server. The server can also be a server in a distributed system or a server integrated with blockchain. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data and artificial intelligence platforms, or intelligent cloud computing servers or intelligent cloud hosts equipped with artificial intelligence technology.

[0054] It is worth noting that the task processing methods provided in the embodiments of the present disclosure are generally executed by the server. However, in other embodiments of the present disclosure, the client may also have similar functions to the server and thus execute the task processing methods provided in the embodiments of the present disclosure. In other embodiments, the task processing methods provided in the embodiments of the present disclosure may also be jointly executed by the client and the server.

[0055] 3 , which shows a flowchart of a task processing method provided by an embodiment of the present disclosure, specifically including the following steps 302 to 306.

[0056] Step 302: Obtain task data and task description information of the target task.

[0057] In one or more embodiments of the present disclosure, during task processing, task data and task description information of a target task may be obtained, and the task data may be processed based on the task description information to generate a task processing result corresponding to the target task.

[0058] Specifically, task data is the object of task processing. Task data can be data of different data types, such as target images, target audio, target videos, and so on. Data types can also be understood as data modalities. Task data can be data for different tasks, such as intelligent question-answering tasks, urban governance tasks, object counting tasks, and optical character recognition tasks. Task description information is used to describe the processing task corresponding to the task data. Task description information can be different types of information, such as task description text, task description audio, task description video, and task description image. For example, if the task data is a target image for an object recognition task, the task description information might be "Where is the bottle in the target image?" For another example, if the target task is a target image for an optical character recognition (OCR) task, the task description information might be "What is the text in the target image?"

[59] It should be noted that, when the task description information is in a non-textual mode, the task description information can be modally converted to obtain textual mode task description information, thereby facilitating the large model to perform task processing. There are multiple ways to modally convert the task description information, which can be selected based on actual conditions. The present disclosure does not impose any restrictions on this. Taking the task description information in the voice mode as an example, the voice-to-text tool can be used to convert the voice-modal task description information into textual mode task description information. Alternatively, the voice-modal task description information can be input into a speech recognition model to obtain textual mode task description information.

[0060] In practical applications, there are various ways to obtain task data and task description information for a target task, and the method selected depends on the specific circumstances. The present disclosure does not impose any limitations on this method. In one possible implementation of the present disclosure, task data and task description information associated with the task data sent by a user via a client can be received. In another possible implementation of the present disclosure, task data and task description information associated with the task data can be read from other data acquisition devices or databases.

[0061] Step 304: Input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data.

[0062] In one or more embodiments of the present disclosure, after obtaining the task data and task description information of the target task, the task data may be further input into the first task processing model to obtain key text information corresponding to the task data.

[0063] Specifically, the first task processing model is used to extract key information from the task data to determine key text information corresponding to the task data. The first task processing model can be of different types, such as a first task processing model for extracting image color attributes, a visual content detection model for extracting key object information from images, or a visual relationship extraction model for extracting object relationship information. Key information in the task data can be understood as the core data in the task data, such as visual object data in images, spoken object data in speech, and so on. Key text information can be text information describing visual objects, text information describing spoken objects, or text information describing object coordinates in speech. Key text information can be structured or unstructured, such as natural language text information.

[0064] In an optional embodiment of the present disclosure, in order to more accurately obtain key text information corresponding to task data, a first task processing model capable of processing the task data may be determined based on the data type of the task data. That is, before inputting the task data into the first task processing model and obtaining the key text information corresponding to the task data, the following steps may be included: obtaining data processing types of multiple candidate first task processing models; performing type identification on the task data to determine the data type of the task data; and selecting the first task processing model from the multiple candidate first task processing models based on the data type and the data processing type.

[0065] Specifically, the candidate first task processing model can be a model for processing different types of data, such as a speech processing model for processing speech data or an image processing model for processing image data. Data processing types include, but are not limited to, speech data types, image data types, video data types, and the like.

[66] It should be noted that there are multiple ways to obtain the data processing types of multiple candidate first task processing models, and the specific selection is based on actual conditions. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the data processing types of multiple candidate first task processing models sent by the user through the client can be received. In another possible implementation of the present disclosure, the data processing types of multiple candidate first task processing models can be read from other data acquisition devices or databases.

[0067] In practical applications, there are various ways to identify and determine the data type of task data, and the method of selecting the right method depends on the specific circumstances. The present disclosure does not impose any limitations on this method. In one possible implementation of the present disclosure, the task data can be input into a type identification model to obtain the data type of the task data. The type identification model is trained based on a plurality of sample data and type labels corresponding to the sample data. In another possible implementation of the present disclosure, template data corresponding to different data types can be obtained, and the task data can be matched with target data corresponding to the different data types. The data type corresponding to the template data that matches the task data can be determined as the data type of the task data.

[0068] Furthermore, after determining the data type of the task data and the data processing types of multiple candidate first task processing models, the candidate first task processing model corresponding to the data processing type identical to the data type may be determined as the first task processing model for processing the task data.

[0069] Using the solution of the embodiments of the present disclosure, the data processing types of multiple candidate first task processing models are obtained; the task data is identified to determine its data type; and the first task processing model is selected from the multiple candidate first task processing models based on the data type and data processing type. This ensures that the first task processing model can accurately obtain the key text information corresponding to the task data, thereby improving the accuracy of the key text information.

[0070] Step 306: Input the key text information, task data, and task description information into the second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0071] In one or more embodiments of the present disclosure, task data and task description information of the target task are obtained; ... After the key text information in the task data is obtained by inputting the key text information into the first task processing model, the key text information, the task data and the task description information can be further input into the second task processing model to obtain the task processing result of the task data.

[72] Specifically, the second task processing model can be a pre-trained language model or a multimodal large language model. The second task processing model is used to generate task processing results for task data. The model parameters of the second task processing model can be greater than the model parameters of the first task processing model. Therefore, compared with the first task processing model, the second task processing model is a large model.

[73] By applying the solution of the embodiment of the present disclosure, the key information extraction capability of the first task processing model is additionally utilized to assist the second task processing model before the second task processing model is utilized to perform task processing. This achieves the combination of the openness and robustness of the second task processing model with the professional key information extraction capability of the first task processing model, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the task processing results.

[74] In practical applications, there are multiple ways to input key text information, task data, and task description information into the second task processing model to obtain the task processing result of the task data. The specific method is selected according to the actual situation, and the embodiments of this disclosure do not impose any restrictions on this. In one possible implementation of this disclosure, the key text information, task data, and task description information can be directly input into the second task processing model to obtain the task processing result of the task data. In another possible implementation of this disclosure, the key text information and task description information can be fused by generating prompt information, and the fused data to be processed and task data can be input into the second task processing model to obtain the task processing result of the task data.

[75] In an optional embodiment of the present disclosure, the above-mentioned inputting of key text information, task data and task description information into the second task processing model to obtain the task processing result of the task data may include the following steps: obtaining generation prompt information; fusing the key text information and the task description information according to the generation prompt information to obtain the data to be processed; inputting the data to be processed and the task data into the second task processing model to obtain the task processing result of the task data.

[76] Specifically, the generated prompt information (prompt) is used to guide the second task processing model to perform task processing. The generated prompt information is set according to the actual situation. The embodiment of the present disclosure does not impose any restrictions on this. For example, the generated prompt information can be: Answer this question based on knowledge. Let's see what is in the task data. The task data contains [key text information], and based on the task data, answer the question [task description information], and the answer is [answer

[77] It should be noted that, when the key text information and the task description information are integrated according to the generated prompt information, the key text information and the task description information can be directly filled into the corresponding positions in the generated prompt information to obtain the data to be processed; the key text information and the task description information can also be filled into the corresponding positions in the generated prompt information, and the data obtained after filling can be deduplicated and the information rewritten to obtain more fluent and concise data to be processed.

[78] Applying the solution of the embodiment of the present disclosure, obtaining generation prompt information; fusing key text information and task description information based on the generation prompt information to obtain data to be processed; inputting the data to be processed and the task data into the second task processing model to obtain the task processing result of the task data. By fusing key text information and task description information based on the generation prompt information, it is ensured that the second task processing model can normally infer and obtain the task processing result.

[79] In an optional embodiment of the present disclosure, before the above-mentioned step of fusing the key text information and the task description information according to the generated prompt information to obtain the data to be processed may further include the following steps: when the key text information is structured information, performing format conversion on the key text information to obtain unstructured key text information; and fusing the key text information and the task description information according to the generated prompt information to obtain the data to be processed. The above-mentioned step of fusing the unstructured key text information and the task description information according to the generated prompt information to obtain the data to be processed may further include the following steps: when the key text information is structured information, performing format conversion on the key text information to obtain unstructured key text information.

[80] Specifically, structured information refers to data with a fixed format, for example, an object-relationship triplet 〈object A, relationship, object B〉. Unstructured information refers to data that has no fixed format, does not follow a predefined data model, and is difficult to Information organized and stored using columns and rows in traditional databases, such as data in the form of natural languages used in daily life (such as English, Chinese, etc.).

[81] It should be noted that, in order to enable the second task processing model to more accurately implement task processing, before inputting the key text information into the second task processing model for processing, it is possible to determine whether the key text information is structured information. If the key text information is not structured information but unstructured key text information, the key text information can be directly input into the second task processing model; if the key text information is structured information, the key text information can be converted into unstructured key text information, and then the unstructured key text information can be input into the second task processing model for task processing. Of course, if the key text information is structured information, it is also possible to not convert the key text information into a format and directly input the structured key text information into the second task processing model for task processing.

[0082] In practical applications, there are various ways to convert key text information into unstructured key text information. The method of selecting the right format depends on the specific situation and is not limited in the embodiments of the present disclosure. In one possible implementation of the present disclosure, key text information in a structured format can be entered into a natural language conversion template to obtain unstructured key text information. In another possible implementation of the present disclosure, key text information in a structured format can be input into a format conversion model to obtain unstructured key text information. The format conversion model is trained based on multiple sample structured information and the natural language labels corresponding to each sample structured information.

[0083] Furthermore, the implementation of “fusing the unstructured key text information and the task description information according to the generated prompt information to obtain the data to be processed” can refer to the implementation of “fusing the key text information and the task description information according to the generated prompt information to obtain the data to be processed” described above, and will not be described in detail in the embodiment of the present disclosure.

[0084] For example, assuming that the first task processing model is a visual content detection model, the task data is input into the first task processing model, and the key text information obtained in the task data is "1<0, 284, 133, 424>, 2<0, 1, 441, 417>, 3<308, 140, 638, 421>", where the numbers 1, 2, and 3 represent object category identifiers.<x,y,z,t> represents object coordinate information, where x represents the x-axis coordinate, y represents the y-axis coordinate, z represents the z-axis coordinate, and t represents the time coordinate of the event. Since the key text information is structured, the second task processing model does not know what the numbers in the key text information represent. In this case, the object category identifiers in the key text information can be converted into the names of the corresponding objects (e.g., 1 represents a cup, 2 represents a dog, and 3 represents a newspaper). However, since the coordinate information is more complex after being converted into natural language, only the object category identifiers can be converted, leaving the object coordinate information unprocessed. The converted key text information is obtained as follows: "In the picture, there is a cup <0,284, 133, 424>, a dog <0,1,441,417>, and a newspaper <308,140,638,421>."

[0085] Assuming the first task processing model is a visual relationship extraction model, the task data is input into the first task processing model. The key text information obtained in the task data is the object-relationship triple "1 M 2". The identifiers of the object and relationship categories in the object-relationship triple can be converted into the corresponding object names and relationship categories (for example, 1 represents cup, 2 represents dog, and M represents proximity relationship). This will obtain the key text information "dog is close to cup" after the format conversion.

[0086] Using the solution of the embodiments of the present disclosure, when the key text information is structured, the format of the key text information is converted to obtain unstructured key text information. Based on the generated prompt information, the unstructured key text information is merged with the task description information to obtain the data to be processed. By converting the results of the task data processing by the first task processing model into a language format, the second task processing model can more easily understand it, thereby improving the accuracy of task processing.

[0087] In an optional embodiment of the present disclosure, taking the second task processing model as a pre-trained language model as an example, that is, the second task processing model includes the pre-trained language model; the above-mentioned inputting key text information, task data, and task description information into the second task processing model to obtain a task processing result for the task data may include the following steps: extracting features from the task data to obtain task coding features; mapping the task coding features according to the data processing format of the pre-trained language model to obtain projected coding features; and inputting the key text information, the projected coding features, and the task description information into the pre-trained language model to obtain a task processing result for the task data.

[88] It should be noted that there are many ways to extract features from task data, and the specific method to be selected depends on the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, feature (embedding) extraction can be performed by deep learning, that is, feature representation is automatically learned from task data using a neural network architecture. For example, if the task data is a target image, a convolutional neural network (CNN) can be used to extract high-level abstract features from the target image to obtain task coding features. In another possible implementation of the present disclosure, feature extraction of task data can be performed using an autoencoder (AE) to obtain task coding features.

[0089] In practical applications, there are various methods for mapping task encoding features to obtain projected encoding features, depending on the data processing format of the pre-trained language model. The method of selecting the right method depends on the specific situation and is not limited in the embodiments of this disclosure. In one possible implementation of this disclosure, task encoding features can be mapped using deep hashing to convert them into projected encoding features that are relatively similar to text features. In another possible implementation of this disclosure, task encoding features can be mapped using a dual encoder architecture, with two parallel encoders processing the task encoding features and text input, respectively. Ultimately, through pre-training and fine-tuning, data from the two different modalities can coexist in the same semantic space, thereby achieving a clear and efficient mapping of task encoding features to text features within this space.

[0090] By applying the solution of the embodiments of the present disclosure, feature extraction is performed on task data to obtain task coding features. The task coding features are mapped according to the data processing format of the pre-trained language model to obtain projected coding features. Key text information, projected coding features, and task description information are input into the pre-trained language model to obtain task processing results for the task data. The additional use of key text information alleviates knowledge forgetting in the second task processing model and improves the accuracy of the task processing results.

[0091] In another optional embodiment of the present disclosure, taking a multimodal large language model as an example, the second task processing model includes an encoding unit, a pre-trained language model, and a projection unit, where the projection unit is used to connect the encoding unit and the pre-trained language model. Inputting key text information, task data, and task description information into the second task processing model to obtain a task processing result for the task data may include the following steps: encoding the task data via the encoding unit to obtain task encoding features; mapping the task encoding features via the projection unit according to the data processing format of the pre-trained language model to obtain projected encoding features; and processing the projected encoding features, key text information, and task description information via the pre-trained language model to obtain a task processing result for the task data.

[0092] Specifically, the structure of the multimodal large language model is composed of a multimodal encoding unit, a pre-trained language model, and a projector connecting the encoding unit and the pre-trained language model. The encoding unit is a non-language modality encoding unit, such as a vision encoder.

[0093] In the solution of the embodiments of the present disclosure, since the second task processing model includes an encoding unit, a pre-trained language model, and a projection unit, the encoding unit encodes the task data to obtain task encoding features; the projection unit maps the task encoding features according to the data processing format of the pre-trained language model to obtain projected encoding features; and the pre-trained language model processes the projected encoding features, key text information, and task description information to obtain a task processing result for the task data. This eliminates the need for separate feature extraction and feature mapping, thereby improving task processing efficiency.

[94] It should be noted that after obtaining the task processing result of the task data, the task processing result can be sent to the client so that the client can display the task processing result to the user. Furthermore, the task processing result can also be sent to the client based on the user's display requirement information so that the client can display the task processing result to the user. The display requirement information represents the user's demand for viewing the task processing result. The display requirement information includes but is not limited to displaying only the task processing result, displaying the task processing result and key text information. The display requirement information is specifically set according to the actual needs of the user, and the embodiments of the present disclosure do not impose any restrictions on this.

[0095] In an optional embodiment of the present disclosure, after obtaining a task processing result, if the user is dissatisfied with the task processing result, the user may send feedback information based on the task processing result. After inputting the key text information, task data, and task description information into the second task processing model and obtaining the task processing result of the task data, the following steps may also be included: In response to feedback information sent by the user regarding the second task processing model, generate optimization prompt information, wherein the optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model; receive the model optimization data sent by the user based on the optimization prompt information, and adjust model parameters of the second task processing model according to the model optimization data.

[0096] Specifically, the feedback information may be information regarding the accuracy of the second task processing model, such as "the model is inaccurate," or feedback regarding the applicable domain of the second task processing model, such as "whether it can handle tasks in the XXX domain." The model optimization data may be an accurate optimization sample set or the optimization domain of the model, depending on actual circumstances and is not limited in any way by the presently disclosed embodiments.

[97] It should be noted that there are multiple ways to generate optimization prompt information in response to the feedback information sent by the user for the second task processing model. The specific selection depends on the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the pre-set optimization prompt information can be directly obtained, such as "I am very sorry to provide you with inaccurate information. Please point out the specific inaccuracies or provide the correct answers to relevant questions. I will correct and optimize my answers as soon as possible to better serve you." In another possible implementation of the present disclosure, the feedback information can be identified to determine the information type of the feedback information, and the information type can be further matched with the prompt type of each prompt information in the prompt information library, and the prompt information with the same prompt type as the information type can be determined as the optimization prompt information.

[0098] Furthermore, after generating the optimization prompt information, the optimization prompt information can be sent to the client, so that the client displays the optimization prompt information to the user. After receiving the model optimization data sent by the user based on the optimization prompt information, if the model optimization data is an accurate optimization sample set, the second task processing model can be directly fine-tuned using the optimization sample set to obtain an updated second task processing model. If the model optimization data is for the optimization domain of the model, such as the XXX domain, a sample set for the XXX domain can be obtained and used to fine-tune the second task processing model to obtain an updated second task processing model. The process of fine-tuning the second task processing model is the same as the training process of the second task processing model and will not be further described in this disclosed embodiment.

[0099] Using the solution of the embodiments of the present disclosure, in response to user feedback regarding the second task processing model, optimization prompt information is generated. The optimization prompt information guides the user to submit model optimization data for optimizing the second task processing model. The system then receives the model optimization data sent by the user based on the optimization prompt information and adjusts the model parameters of the second task processing model based on the model optimization data. After receiving the user feedback regarding the second task processing model, the system obtains the model optimization data through interactive guidance and optimizes the parameters of the second task processing model, making the second task processing model more accurate. This improves interactivity with the user and increases user satisfaction.

[0100] In an optional embodiment of the present disclosure, before inputting the task data into the first task processing model and obtaining key text information corresponding to the task data, the following steps may also be included: obtaining multiple sample task data, wherein the sample task data carries key text labels; inputting the multiple sample task data into the first task processing model to obtain predicted text information corresponding to the multiple sample task data; calculating an information extraction loss value based on the key text labels and the predicted text information; and adjusting model parameters of the first task processing model based on the information extraction loss value to obtain a trained first task processing model.

[0101] Specifically, the first task processing model is trained using supervised training, meaning that the sample task data carries real key text labels. The key text labels serve as extraction targets for the first task processing model and are used to guide the training process of the first task processing model. Multiple sample task data can be obtained by reading a large amount of sample task data carrying key text labels from other data acquisition devices or databases. Alternatively, a large amount of sample task data carrying key text labels can be received from users. The method for obtaining multiple sample task data is selected based on actual circumstances and is not limited in any way by the presently disclosed embodiments.

[102] It should be noted that there are many functions for calculating the information extraction loss value, such as the cross entropy loss function, the L1 norm loss function, the maximum loss function, the mean square error loss function, the logarithmic loss function, etc. The specific selection is based on the actual situation, and the embodiments of the present disclosure do not impose any restrictions on this.

[0103] In actual applications, according to the information extraction loss value, the stopping conditions for adjusting the model parameters of the first task processing model include but are not limited to the information extraction loss value being less than or equal to a preset threshold and the number of iterations reaching a preset number of iterations, wherein the preset threshold and the preset number of iterations are selected according to actual conditions, and the embodiments of the present disclosure do not impose any restrictions on this.

[0104] In one possible implementation of the present disclosure, after calculating the information extraction loss value, the information extraction loss value is compared with a preset threshold. Specifically, if the information extraction loss value is greater than the preset threshold, this indicates that the difference between the key text label and the predicted text information is significant, and the first task processing model's predictive ability for the sample task data is poor. In this case, the model parameters of the first task processing model can be adjusted, and the process returns to the step of inputting multiple sample task data into the first task processing model to obtain the predicted text information corresponding to each of the multiple sample task data. The first task processing model is then trained again until the information extraction loss value is less than or equal to the preset threshold, indicating that the difference between the key text label and the predicted text information is small. This meets the preset stopping condition, and the trained first task processing model is obtained.

[0105] In another possible implementation of the present disclosure, in addition to comparing the information extraction loss value with a preset threshold, the number of iterations may also be used to determine whether the current first task processing model has been trained. Specifically, if the information extraction loss value is greater than the preset threshold, the model parameters of the first task processing model are adjusted, and the process returns to the step of inputting multiple sample task data into the first task processing model to obtain predicted text information corresponding to each of the multiple sample task data. Training of the first task processing model continues until the preset number of iterations is reached, at which point iterations are terminated, resulting in a fully trained first task processing model.

[0106] Using the solution of the embodiments of the present disclosure, an information extraction loss value is calculated based on key text labels and predicted text information. This information extraction loss value is compared with a preset stopping condition. If the preset stopping condition is not met, the first task processing model is trained continuously until the preset stopping condition is met, completing the training and obtaining the first task processing model. By continuously adjusting the model parameters of the first task processing model, the resulting first task processing model can be made more accurate.

[0107] The following, combined with Figure 4, further illustrates the task processing method provided by the present disclosure, using its application in an intelligent question-answering scenario as an example. The question-answering task can be a voice question-answering task, an image question-answering task, or a video question-answering task. Figure 4 shows a flowchart of an automatic question-answering method provided by one embodiment of the present disclosure, specifically including steps 402 through 406.

[0108] Step 402: Obtain task data and questions to be answered for the question-answering task.

[0109] Step 404: Input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data.

[0110] Step 406: Input the key text information, task data, and the question to be answered into the second task processing model to obtain an answer result for the question to be answered, wherein the second task processing model is used to process the question-answering task.

[0111] It should be noted that the implementation of step 402 to step 406 is the same as the implementation of step 302 to step 306 described above, and will not be described in detail in this embodiment of the disclosure.

[0112] By applying the solution of the embodiments of the present disclosure, the key information extraction capabilities of the first task processing model are additionally utilized to assist the second task processing model before the second task processing model is used for task processing. This combines the openness and robustness of the second task processing model with the professional key information extraction capabilities of the first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of the response results.

[0113] 5 , which shows a flow chart of an image processing method provided by an embodiment of the present disclosure, specifically including the following steps 502 to 506.

[0114] Step 502: Obtain the target image and task description information of the image processing task.

[0115] Step 504: Input the target image into the first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image.

[0116] Step 506: Input the key text information, the target image and the task description information into the second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0117] It should be noted that the implementation of steps 502 to 506 is the same as the implementation of steps 302 to 306 above. The same, the embodiment of the present disclosure will not be repeated. The image processing task includes but is not limited to an image recognition task, an image target detection task, and an image segmentation task.

[0118] For example, assume that the task corresponding to the target image is an object counting task, and the task description information of the target image is "Mark where there is a bottle in the image." By inputting the target image into the first task processing model, the key text information corresponding to the target image is obtained as "You can see bottle, bottle, bottle, bottle, bottle, bottle, bottle, bottle at position YYY in the image." The key text information, target image, and task description information are input into the second task processing model, and the task processing result of the target image is obtained as "There are eight bottles at position YYY in the target image."

[0119] Assume that the task corresponding to the target image is an optical character recognition task, and the task description information of the target image is "What is the text in the target image?" By inputting the target image into the first task processing model, the key text information corresponding to the target image is obtained. The key text information, target image, and task description information are input into the second task processing model, and the task processing result of the target image is obtained as "The text in the target image is "Happy New Year"."

[0120] By applying the solution of the embodiments of the present disclosure, the key information extraction capabilities of the first task processing model are additionally utilized to assist the second task processing model before the second task processing model is used for image processing. This combines the openness and robustness of the second task processing model with the specialized key information extraction capabilities of the first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of the image processing results.

[0121] In an optional embodiment of the present disclosure, the first task processing model includes a visual content detection model and / or a visual relationship extraction model; inputting a target image into the first task processing model to obtain key text information corresponding to the target image may include the following steps: inputting the target image into the visual content detection model to obtain object category text and object location text of key objects in the target image; and / or, inputting the target image into the visual relationship extraction model to obtain object relationship text between key objects in the target image.

[0122] Specifically, visual content detection models can be understood as small detection models, such as DETR (Detection Transformer) > YOLO (You Only Look Once) > DINO (DETR with Improved deNoising anchOr boxes), etc. Visual relationship extraction models can be understood as scene graph generation models.

[123] It should be noted that different first task processing models generally have different structures, but the functions performed by the same type of first task processing models are the same. For example, the function of the visual content detection model is to detect which objects are in the target image and where they are located; the function of the visual relationship extraction model is to extract the relationship between objects in the target image (such as position relationship, action relationship).

[0124] By applying the solution of the embodiments of the present disclosure, the target image is input into a visual content detection model to obtain object category text and object position text of relevant objects in the target image; and / or, the target image is input into a visual keyword extraction model to obtain object relationship text between key objects in the target image string, thereby achieving the additional use of object category text, object position text, and object relationship text to alleviate knowledge forgetting in the second task processing model and improve the accuracy of the task processing results.

[0125] The following, in conjunction with Figure 6, further illustrates the task processing method provided by the present disclosure, using its application in an urban governance scenario as an example. Figure 6 illustrates a flow chart of an urban governance method provided by one embodiment of the present disclosure, specifically including steps 602 through 606.

[0126] Step 602: Obtain the target city image and task description information of the urban governance task.

[0127] Step 604: Input the target city image into the first task processing model to obtain key text information corresponding to the target city image, wherein the key text information is used to describe key visual information in the target city image. The first task processing model is used to process the target city image.

[0128] Step 606: Input the key text information, the target city image and the task description information into the second task processing model to obtain the governance event recognition result of the target city image, wherein the second task processing model is used to process the city governance task.

[129] It should be noted that the implementation of steps 602 to 606 is the same as the implementation of steps 302 to 306 above. The same is true, and the embodiments of the present disclosure will not be described in detail.

[130] In actual applications, governance tasks in urban governance scenarios include but are not limited to geological exploration tasks and environmental assessment tasks. In urban governance scenarios, the first task processing model and the second task processing model can be used to build services for urban governance event description, rule event judgment, and urban image reporting (reporting governance events), thereby improving the accuracy of urban governance event recognition. In addition, the first task processing model can improve the second task processing model's ability to understand professional geological maps in a fine-grained manner. It automatically performs image recognition, structural analysis, and mineral identification, and can understand and answer users' questions about geological structure, history, and potential resources. In the field of geological exploration, it helps geologists identify possible mineral deposit locations and predict the distribution of oil and gas fields, thereby optimizing exploration plans and reducing unnecessary physical exploration work. In environmental assessment, it analyzes soil and rock images, quickly identifies environmental risks, and guides environmental protection and disaster prevention work. Experiments have shown that the second task processing model combined with the first task processing model does not require retraining the second task processing model, and the Fl score between output and annotation is improved by 135%, and the large model score is improved by 86%.

[131] By applying the solution of the embodiment of the present disclosure, the key information extraction capability of the first task processing model is additionally utilized to assist the second task processing model before the second task processing model is utilized for urban governance. This realizes the combination of the openness and robustness of the second task processing model with the professional key information extraction capability of the first task processing model, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the governance event recognition results.

[132] See FIG7 , which shows a flowchart of a processing process of an image processing method provided by an embodiment of the present disclosure, specifically including the following contents.

[133] Visual Evidence Extraction: The image input to the second task processing model (multimodal large language model) is simultaneously input to the first task processing model (visual small model) for visual knowledge extraction. For the key text information of the object in the image, the image is input to the visual content detection model, and the output is the object category and object location in the image, that is, {label}.<xl,yl,x2,y2> ; For the key text information of the relationship between objects in the image, the visual relationship extraction model is used, and the output is the object relationship triplet existing in the image, namely <subject, relation, object>, integrating the object category, object location and object relationship triplet to obtain the key text information.

[134] Visual evidence prompting: The key text information and task description text obtained in the first step are integrated into a generated prompt information to obtain the data to be processed; then the integrated data to be processed and the image are input into the multimodal large language model to obtain the model's response.

[135] It should be noted that the understanding of the image by the first task processing model (such as the object category and object position output by the visual content detection model) is used as additional visual knowledge. This visual knowledge is used as a reference and combined with the task description text before the second task processing model performs reasoning and is added to the generated prompt information. The normal reasoning of the second task processing model is performed to obtain the answer of the second task processing model. This can significantly reduce the hallucinations of the second task processing model and improve the accuracy of the answer.

[136] For example, as shown in FIG7 , the target image and task description text of the target task are obtained, “Answer the question based on the target image: Is the cup near the dog? The answer (yes or no) is [answer]”, where (yes or no) is used to prompt that the answer output by the second task processing model can be “yes” or “no”. The target image is input into the small visual model to obtain the key text information in the target image; the generated prompt information “Answer this question based on knowledge. Let’s see what is in the image. There is [key text information] in the image” is obtained, and according to the generated prompt information, the key text information and the task description information are fused and input into the multimodal large language model together with the target image to obtain the answer “yes”.

[137] In the disclosed embodiment, a multimodal large language model and a small visual model are combined, thereby combining the openness and robustness of the multimodal large language model with the professional knowledge of the small visual model, and using the specific domain knowledge of the small visual model to assist the large multimodal language model, as a visual plug-in knowledge base for the large multimodal language model. Since there is no need to change the model structure and model parameters of the large multimodal language model, it only takes a low cost to significantly reduce the degree of hallucination of the large multimodal language model and improve the accuracy and reliability of the model's answers. Therefore, it can be applied to multimodal large language models of various structures and types, and has strong robustness and generalization. At the same time, it also minimizes the knowledge forgetting of the model. In addition, the small visual model can be flexibly customized according to the application scenario. Using small visual models of different types and sizes can reduce the multimodal hallucination phenomenon of the large multimodal language model to varying degrees, and can also enhance the model in other fine-grained tasks that require higher model understanding capabilities (such as object counting, optical character recognition, etc.).

[0138] Corresponding to the above-described task processing method embodiment, the present disclosure also provides a task processing device embodiment. FIG8 shows a schematic structural diagram of a task processing device provided by one embodiment of the present disclosure. As shown in FIG8 , the device includes a first acquisition module 802, a first input module 804, and a second input module 806.

[0139] The first acquisition module 802 is configured to acquire task data and task description information of the target task.

[0140] The first input module 804 is configured to input task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data.

[0141] The second input module 806 is configured to input key text information, task data and task description information into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

[0142] Optionally, the apparatus further includes: a screening module configured to obtain data processing types of multiple candidate first task processing models; perform type identification on the task data to determine the data type of the task data; and screen out the first task processing model from the multiple candidate first task processing models based on the data type and the data processing type.

[0143] Optionally, the second input module 806 is further configured to obtain generation prompt information; fuse the key text information and task description information according to the generation prompt information to obtain data to be processed; input the data to be processed and the task data into the second task processing model to obtain a task processing result of the task data.

[0144] Optionally, the device also includes: a conversion module, configured to convert the format of the key text information when the key text information is structured information to obtain unstructured key text information; a second input module 806, further configured to fuse the unstructured key text information and task description information according to the generated prompt information to obtain data to be processed.

[0145] Optionally, the second task processing model includes a pre-trained language model; the second input module 806 is further configured to perform feature extraction on the task data to obtain task coding features; map the task coding features according to the data processing format of the pre-trained language model to obtain projected coding features; and input the key text information, the projected coding features, and the task description information into the pre-trained language model to obtain a task processing result for the task data.

[0146] Optionally, the second task processing model includes an encoding unit, a pre-trained language model, and a projection unit, where the projection unit is used to connect the encoding unit and the pre-trained language model; the second input module 806 is further configured to encode the task data via the encoding unit to obtain task encoding features; map the task encoding features according to the data processing format of the pre-trained language model via the projection unit to obtain projected encoding features; and process the projected encoding features, key text information, and task description information via the pre-trained language model to obtain a task processing result for the task data.

[0147] Optionally, the device further includes: an adjustment module, configured to generate optimization prompt information in response to feedback information sent by the user regarding the task processing model, wherein the optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model; receive the model optimization data sent by the user based on the optimization prompt information, and adjust the model parameters of the second task processing model according to the model optimization data.

[0148] Optionally, the device also includes: an information adjustment model training module, configured to obtain multiple sample task data, wherein the sample task data carries key text labels; input the multiple sample task data into the first task processing model to obtain predicted text information corresponding to the multiple sample task data; calculate the information extraction loss value based on the key text labels and the predicted text information; adjust the model parameters of the first task processing model based on the information extraction loss value to obtain a trained first task processing model.

[0149] By applying the solution of the embodiments of the present disclosure, the key information extraction capabilities of the first task processing model are additionally utilized to assist the second task processing model before the second task processing model is used for task processing. This combines the openness and robustness of the second task processing model with the specialized key information extraction capabilities of the first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of task processing results.

[0150] The above is a schematic diagram of a task processing device according to this embodiment. It should be noted that the technical solution of the task processing device and the technical solution of the task processing method described above are based on the same concept. For details not described in detail in the technical solution of the task processing device, please refer to the description of the technical solution of the task processing method described above.

[0151] Corresponding to the above-mentioned automatic question-answering method embodiment, the present disclosure also provides an automatic question-answering device embodiment, FIG9 shows FIG9 is a schematic diagram of the structure of an automatic question-answering device provided by an embodiment of the present disclosure. As shown in FIG9 , the device includes a second acquisition module 902, a third input module 904, and a fourth input module 906.

[0152] The second acquisition module 902 is configured to acquire task data and questions to be answered of the question-answering task.

[0153] The third input module 904 is configured to input the task data into the first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data.

[0154] The fourth input module 906 is configured to input key text information, task data and questions to be answered into the second task processing model to obtain a response result to the question to be answered, wherein the second task processing model is used to process the question-answering task.

[0155] By applying the solution of the embodiments of the present disclosure, the key information extraction capabilities of the first task processing model are additionally utilized to assist the second task processing model before the second task processing model is used for automatic question answering. This combines the openness and robustness of the second task processing model with the professional key information extraction capabilities of the first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of the answer results.

[0156] The above is a schematic diagram of an automatic question-answering device according to this embodiment. It should be noted that the technical solution of this automatic question-answering device and the technical solution of the automatic question-answering method described above are based on the same concept. For details not described in detail in the technical solution of the automatic question-answering device, please refer to the description of the technical solution of the automatic question-answering method described above.

[0157] Corresponding to the above-mentioned image processing method embodiment, the present disclosure also provides an image processing device embodiment. FIG10 shows a schematic structural diagram of an image processing device provided by one embodiment of the present disclosure. As shown in FIG10 , the device includes a third acquisition module 1002, a fifth input module 1004, and a sixth input module 1006.

[0158] The third acquisition module 1002 is configured to acquire the target image and task description information of the image processing task.

[0159] The fifth input module 1004 is configured to input the target image into the first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image.

[0160] The sixth input module 1006 is configured to input the key text information, the target image and the task description information into the second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

[0161] Optionally, the first task processing model includes a visual content detection model and / or a visual relationship extraction model; the fifth input module 1004 is further configured to input the target image into the visual content detection model to obtain object category text and object position text of key objects in the target image; and / or, input the target image into the visual relationship extraction model to obtain object relationship text between key objects in the target image string.

[0162] By applying the solution of the embodiments of the present disclosure, the key information extraction capabilities of the visual first task processing model are additionally utilized to assist the second task processing model before the second task processing model is used for image processing. This combines the openness and robustness of the second task processing model with the professional key information extraction capabilities of the visual first task processing model, thereby alleviating knowledge forgetting in the second task processing model and improving the accuracy of the task processing results.

[0163] The above is a schematic diagram of an image processing device according to this embodiment. It should be noted that the technical solution of this image processing device and the technical solution of the aforementioned image processing method share the same concept. For details not described in detail in the technical solution of the image processing device, please refer to the description of the technical solution of the aforementioned image processing method.

[0164] Corresponding to the above-mentioned urban governance method embodiment, the present disclosure also provides an urban governance device embodiment. Figure 11 shows a schematic structural diagram of an urban governance device provided in one embodiment of the present disclosure. As shown in Figure 11, the device includes a fourth acquisition module 1102, a seventh input module 1104, and an eighth input module 1106.

[0165] The fourth acquisition module 1102 is configured to obtain the target city image and task description information of the urban governance task.

[0166] The seventh input module 1104 is configured to input the target city image into the first task processing model to obtain key text information corresponding to the target city image, wherein the key text information is used to describe key visual information in the target city image. The first task processing model is used to process the target city image.

[0167] The eighth input module 1106 is configured to input key text information, target city image and task description information into the second task processing model to obtain the governance event recognition result of the target city image, wherein the second task processing model The model is used to handle urban governance tasks.

[0168] By applying the solution of the embodiment of the present disclosure, the key information extraction capability of the visual first task processing model is additionally utilized to assist the second task processing model before the second task processing model is used for urban governance. This achieves the combination of the openness and robustness of the second task processing model with the professional key information extraction capability of the visual first task processing model, thereby alleviating the knowledge forgetting of the second task processing model and improving the accuracy of the governance event recognition results.

[0169] The above is a schematic diagram of an urban governance device according to this embodiment. It should be noted that the technical solution of this urban governance device and the technical solution of the aforementioned urban governance method are based on the same concept. For details not described in detail in the technical solution of the urban governance device, please refer to the description of the technical solution of the aforementioned urban governance method.

[0170] FIG12 shows a block diagram of a computing device according to an embodiment of the present disclosure. The components of computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. Processor 1220 is connected to memory 1210 via a bus 1230. Database 1250 is used to store data.

[0171] The computing device 1200 also includes an access device 1240 that enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0172] In one embodiment of the present disclosure, the aforementioned components of computing device 1200 and other components not shown in FIG. 12 may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 12 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

[0173] Computing device 1200 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1200 can also be a mobile or stationary server.

[0174] The processor 1220 is configured to execute a computer program / instruction, which implements the steps of the above-mentioned task processing method, automatic question-answering method, or image processing method when executed by the processor.

[0175] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device shares the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and image processing method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or image processing method.

[0176] An embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned task processing method, automatic question-answering method, or image processing method when executed by a processor.

[0177] The above is an illustrative embodiment of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium shares the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and image processing method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or image processing method.

[0178] An embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned task processing method, automatic question-answering method, or image processing method.

[0179] The above is an illustrative embodiment of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product shares the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and image processing method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or image processing method.

[0180] The foregoing description describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0181] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[182] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of action combinations. However, those skilled in the art should be aware that the embodiments of the present disclosure are not limited by the order of the actions described, because according to the embodiments of the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present disclosure.

[0183] In the above embodiments, the descriptions of the various embodiments are given with emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0184] The preferred embodiments disclosed above are intended only to illustrate the present disclosure. The alternative embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of the embodiments disclosed. These embodiments are selected and described in detail to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

Claims 1. A task processing method, comprising: Obtain task data and task description information of the target task; Input the task data into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; input the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data, wherein the second task processing model is used to process the target task.

2. The method according to claim 1, further comprising: Obtaining data processing types of multiple candidate first task processing models; Performing type identification on the task data to determine the data type of the task data; A first task processing model is selected from the plurality of candidate first task processing models according to the data type and the data processing type.

3. The method according to claim 1, wherein inputting the key text information, the task data, and the task description information into a second task processing model to obtain a task processing result of the task data comprises: Get the generated prompt information; According to the generated prompt information, the key text information and the task description information are integrated to obtain data to be processed; The data to be processed and the task data are input into a second task processing model to obtain a task processing result of the task data.

4. The method according to claim 3, before fusing the key text information and the task description information according to the generated prompt information to obtain the data to be processed, further comprising: In the case where the key text information is structured information, performing format conversion on the key text information to obtain unstructured key text information; The fusing the key text information and the task description information according to the generation prompt information to obtain the data to be processed includes: fusing the unstructured key text information and the task description information according to the generation prompt information to obtain the data to be processed.

5. The method according to any one of claims 1 to 4, wherein the second task processing model comprises a pre-trained language model; and the step of inputting the key text information, the task data, and the task description information into the second task processing model to obtain a task processing result for the task data comprises: Feature extraction is performed on the task data to obtain task coding features; the task coding features are mapped according to the data processing format of the pre-trained language model to obtain projected coding features; and the key text information, the projected coding features, and the task description information are input into the pre-trained language model to obtain a task processing result for the task data.

6. The method according to any one of claims 1 to 4, wherein the second task processing model comprises an encoding unit, a pre-trained language model, and a projection unit, wherein the projection unit is configured to connect the encoding unit and the pre-trained language model; The step of inputting the key text information, the task data, and the task description information into the second task processing model to obtain the task processing result of the task data includes: encoding the task data by the encoding unit to obtain task encoding features; mapping the task encoding features according to the data processing format of the pre-trained language model by the projection unit to obtain projection encoding features; and processing the projection encoding features, the key text information, and the task description information by the pre-trained language model to obtain the task processing result of the task data.

7. The method according to claim 1, wherein after inputting the key text information, the task data, and the task description information into a second task processing model and obtaining a task processing result of the task data, the method further comprises: generating optimization prompt information in response to feedback information sent by a user regarding the second task processing model, wherein: The optimization prompt information is used to guide the user to send model optimization data for optimizing the second task processing model. Nie receives the model optimization data sent by the user based on the optimization prompt information, and adjusts the model parameters of the second task processing model according to the model optimization data.

8. The method according to claim 1, before inputting the task data into the first task processing model to obtain key text information corresponding to the task data, further comprising: Acquire a plurality of sample task data, wherein the sample task data carry key text labels; input the plurality of sample task data into a first task processing model to obtain predicted text information corresponding to the plurality of sample task data; calculate an information extraction loss value based on the key text labels and the predicted text information; and adjust model parameters of the first task processing model based on the information extraction loss value to obtain a trained first task processing model.

9. An automatic question-answering method, comprising: Obtain task data and questions to be answered for the question-answering task; The task data is input into a first task processing model to obtain key text information corresponding to the task data, wherein the key text information is used to describe key information in the task data, and the first task processing model is used to process the task data; the key text information, the task data, and the question to be answered are input into a second task processing model to obtain an answer result to the question to be answered, wherein the second task processing model is used to process the question-answering task.

10. An image processing method, comprising: Obtain the target image and task description information of the image processing task; The target image is input into a first task processing model to obtain key text information corresponding to the target image, wherein the key text information is used to describe key visual information in the target image, and the first task processing model is used to process the target image; the key text information, the target image and the task description information are input into a second task processing model to obtain an image processing result of the target image, wherein the second task processing model is used to process the image processing task.

11. The method according to claim 10, wherein the first task processing model comprises a visual content detection model and / or a visual relationship extraction model; and inputting the target image into the first task processing model to obtain key text information corresponding to the target image comprises: Inputting the target image into a visual content detection model to obtain object category text and object position text of key objects in the target image; And / or, inputting the target image into a visual relationship extraction model to obtain object relationship text between key objects in the target image.

12. A computing device, comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 or claim 9 or any one of claims 10 to 11 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 or claim 9 or any one of claims 10 to 11 are implemented.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8 or claim 9 or any one of claims 10 to 11.

Images