Visual task processing method, image processing method, task processing model training method, information processing method based on task processing model and task platform

Abstract

The embodiment of the invention provides a visual task processing method, an image processing method, a task processing model training method, an information processing method based on a task processing model and a task platform. The visual task processing method comprises the steps of obtaining a task image and task description information of a target visual task; inputting the task image and the task description information into a task processing model to obtain a multi-modal feature sequence; performing category identification on the multi-modal feature sequence to obtain classification information of the multi-modal feature sequence, the classification information being used for describing an inclusion relation between a task processing text and a task image corresponding to the multi-modal feature sequence; and performing visual feature decoding on the multi-modal feature sequence according to the classification information and the task image to obtain text positioning information of the task image. In the visual task processing process, extra text positioning auxiliary information does not need to be introduced, so that the length of the multi-modal feature sequence can be greatly reduced, and the visual task processing efficiency is further greatly improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to visual task processing methods, image processing methods, task processing model training methods, information processing methods based on task processing models, and task platforms. Background Technology

[0002] With the development of computer technology, the rapid advancement of digitalization and informatization in various industries such as finance, logistics, healthcare, and education has led to a surge in demand for text extraction, processing, and understanding of visual signals (such as document images, card images, and street view videos). The ability to extract text location while recognizing it in visual signals is also crucial for subsequent analysis and processing of those signals.

[0003] Recently, machine learning models have developed rapidly. Driven by massive amounts of image and text data, the reasoning ability of models, the understanding ability of multimodal models of visual signals such as images and videos, and their versatility have been greatly improved. Therefore, there is an urgent need for a solution that can use models to efficiently extract the position of text in visual signals. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a visual task processing method. One or more embodiments of this specification also relate to an image processing method, a task processing model training method, an information processing method based on a task processing model, a task platform, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a visual task processing method is provided, comprising:

[0006] Acquire the task image and task description information for the target visual task;

[0007] Input the task image and task description information into the task processing model to obtain a multimodal feature sequence;

[0008] The classification information of the multimodal feature sequence is obtained by performing category recognition on the multimodal feature sequence. The classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence.

[0009] Based on classification information and task images, visual feature decoding is performed on multimodal feature sequences to obtain text localization information of the task images.

[0010] According to a second aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0011] Acquire the task image and task description information for the target visual task;

[0012] Input the task image and task description information into the task processing model to obtain a multimodal feature sequence;

[0013] The classification information of the multimodal feature sequence is obtained by performing category recognition on the multimodal feature sequence. The classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence.

[0014] Based on classification information and task images, visual feature decoding is performed on multimodal feature sequences to obtain text localization information of task images;

[0015] Based on the text location information, the task image is processed to obtain the target task image.

[0016] According to a third aspect of the embodiments of this specification, a method for training a task processing model is provided, comprising:

[0017] Acquire sample data for the visual task, including sample images, sample description information, and text location labels;

[0018] Input the sample images and sample description information into the task processing model to obtain the sample feature sequence;

[0019] The sample feature sequence is classified to obtain the sample classification information, which describes the inclusion relationship between the sample processing text and the sample image corresponding to the sample feature sequence.

[0020] Based on the sample classification information and sample images, visual feature decoding is performed on the sample feature sequences to obtain the localization prediction information of the sample images;

[0021] Based on the location prediction information and text location labels, the parameters of the task processing model are adjusted to obtain the trained task processing model.

[0022] According to a fourth aspect of the embodiments of this specification, an information processing method based on a task processing model is provided, applied to a task platform, comprising:

[0023] Receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters;

[0024] Based on the model request, the target task processing model is determined from multiple task processing models, wherein the multiple task processing models are trained based on the task processing model training method.

[0025] According to a fifth aspect of the embodiments of this specification, a task platform is provided, including a request interface and a response unit;

[0026] The request interface is used to receive model requests sent by terminal devices. The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters.

[0027] The response unit is used to determine the target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the task processing model training method.

[0028] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0029] Memory and processor;

[0030] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0031] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0032] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0033] This specification provides a visual task processing method according to one embodiment, comprising: acquiring a task image and task description information of a target visual task; inputting the task image and task description information into a task processing model to obtain a multimodal feature sequence; performing category recognition on the multimodal feature sequence to obtain classification information of the multimodal feature sequence, wherein the classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence; and performing visual feature decoding on the multimodal feature sequence based on the classification information and the task image to obtain text localization information of the task image. By determining the classification information of the multimodal feature sequence, the features related to the task image in the multimodal feature sequence can be determined more precisely, thereby achieving more refined visual task processing. By performing visual feature decoding on the multimodal feature sequence, the features output by the model are decoded into an image mask, realizing the function of locating text in the image. At the same time, during the visual task processing, there is no need to introduce additional text localization auxiliary information, which can significantly reduce the length of the multimodal feature sequence and further significantly improve the efficiency of visual task processing. Attached Figure Description

[0034] Figure 1 This is an architecture diagram of a vision task processing system provided in one embodiment of this specification;

[0035] Figure 2 This is a flowchart illustrating a visual task processing method provided in one embodiment of this specification;

[0036] Figure 3 This is a flowchart illustrating an image processing method provided in one embodiment of this specification;

[0037] Figure 4 This is a flowchart illustrating a task processing model training method provided in one embodiment of this specification;

[0038] Figure 5 This is a flowchart illustrating the processing procedure of a visual task processing method provided in one embodiment of this specification.

[0039] Figure 6 This is a flowchart illustrating an information processing method based on a task processing model, provided in one embodiment of this specification.

[0040] Figure 7 This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0041] Figure 8 This is a schematic diagram of the structure of a visual task processing device provided in one embodiment of this specification;

[0042] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification;

[0043] Figure 10 This is a schematic diagram of the structure of a task processing model training device provided in one embodiment of this specification;

[0044] Figure 11 This is a schematic diagram of the structure of an information processing device based on a task processing model provided in one embodiment of this specification;

[0045] Figure 12 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0046] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0047] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0048] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0049] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0050] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundational model (Foundation Model 1). It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multi-modal Pre-training Models (MLMs).

[0051] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0052] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0053] Optical Character Recognition (OCR) is a technology that uses automatic data extraction to quickly convert text images into a machine-readable format. OCR technology is widely used in various scenarios, such as converting text in scanned documents, images, or even handwritten notes into digital text, enabling editing, searching, or further data analysis of this text.

[0054] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computations and has a deep structure. They are one of the representative algorithms of deep learning and are good at handling image processing, especially image recognition and other related machine learning problems.

[0055] Deep self-attention (Transformer) models are deep learning architectures based on attention mechanisms, primarily used for processing sequential data. A Transformer model consists of repeatable, stackable encoder and decoder units. This design allows the Transformer to efficiently learn long-term dependencies, making it suitable for various natural language processing tasks, including machine translation, text summarization, and question answering systems.

[0056] Visual Depth Self-Attention (ViT, Vision Transformer) model: This is a deep learning model that divides the input image into multiple fixed-size image patches, then uses linear embedding to convert these patches into feature vectors, and then feeds these feature vectors together with a special classification token into the Transformer architecture.

[0057] Swin Transformer is a Transformer-based model designed for computer vision tasks. Compared to the ViT model, Swin Transformer introduces a window attention mechanism to address the computational efficiency issue of ViT when processing large images.

[0058] With the wave of informatization and digitalization, traditional OCR technology has significantly improved the efficiency of digitizing cards and documents. By empowering machines to perceive textual information in the human physical world, it has also made more advanced natural interactions between machines and humans possible. Recently, with the rapid development of large-scale models and multimodal large-scale models, driven by massive amounts of image and text data, the reasoning ability of large-scale models and the understanding and versatility of multimodal large-scale models of visual signals such as images and videos have been greatly improved. While using large-scale model technology to recognize text in images, the ability to extract the precise location of the text is crucial for subsequent analysis and processing, and has wide-ranging applications, such as text erasure, verification, and in-image text search.

[0059] Currently, when using large models for text visual localization tasks, additional tokens are typically added to achieve the localization goal. These additional tokens take the form of... <ref> TEXT< / ref> <box> (x1,y1),(x2,y2)< / box> ,in, <ref>< / ref> Two tokens are used to represent the range of text to be located. <box> (x1,y1),(x2,y2)< / box> It is a sequence of coordinates representing the text's coordinate position within the image; the token. <box>It could also be <point>Other forms, such as coordinates, will also take the corresponding form. Specifically, there are three forms of methods for using large models to perform visual text localization tasks: the first is to represent numbers as tokens in text form; the second is to quantize numbers into tokens within a certain range, such as tokens that can represent values ​​from 0 to 1000; and the third is to use special tokens combined with regression layers.

[0060] However, the above methods have several limitations. First, adding extra tokens for positioning severely impacts task processing efficiency, especially for documents and images that are typically text-dense and numerous. Using extra tokens usually increases the output length by 5-10 times, thus significantly increasing inference time. Second, the representational capabilities of coordinate position tokens are limited. Using the center point for positioning cannot determine the range of the text position within the image. Using bounding boxes also fails to represent complex text shapes, such as curved text or artistic fonts. Accurate text positioning greatly facilitates downstream image text processing needs, such as text erasure or text rendering. Finally, adding extra tokens requires representing the text data in a specific format, disrupting the original language form and resulting in insufficient versatility.

[0061] To address the aforementioned issues, this specification proposes a visual task processing scheme based on a task processing model, which can be used for image text localization. During visual task processing, the task image and task description information of the target visual task can be acquired. The task image and task description information are input into the task processing model to obtain a multimodal feature sequence. Category recognition is performed on the multimodal feature sequence to obtain classification information, where the classification information describes the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence. Based on the classification information and the task image, visual feature decoding is performed on the multimodal feature sequence to obtain the text localization information of the task image. The visual task processing scheme decodes the multimodal feature sequence based on classification information and the task image, converting the representation of the task processing model into an image mask, thereby achieving the function of locating text in an image. Simultaneously, no additional localization tokens are required during the inference stage, which significantly reduces the length of the model's prediction results on visual tasks, thereby greatly improving inference efficiency.

[0062] This specification provides a visual task processing method, and also relates to an image processing method, a task processing model training method, an information processing method based on a task processing model, a task platform, a visual task processing device, an image processing device, a task processing model training device, an information processing device based on a task processing model, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0063] Considering the large number of model parameters in the task processing model and the limited computing resources of the client, the visual task processing method proposed in the embodiments of this specification can be applied to, for example... Figure 1 The visual task processing system shown is not limited to this. See also Figure 1 , Figure 1 This specification illustrates an architecture diagram of a vision task processing system according to one embodiment of the present specification. The vision task processing system may include a client 100 and a server 200.

[0064] Client 100 is used to send the target visual task's task image and task description information to server 200;

[0065] Server 200 is used to input the task image and task description information into the task processing model to obtain a multimodal feature sequence; perform category recognition on the multimodal feature sequence to obtain classification information of the multimodal feature sequence, wherein the classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence; perform visual feature decoding on the multimodal feature sequence based on the classification information and the task image to obtain the text positioning information of the task image; and send the text positioning information to client 100.

[0066] Client 100 is also used to receive text location information sent by server 200.

[0067] like Figure 1 As shown, the task processing model is deployed in server 200. Server 200 can connect to one or more clients 100 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client 100 may include, but is not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client 100 can also interact with users through a graphical user interface to invoke the task processing model, thereby implementing the visual task processing method provided in the embodiments of this specification.

[0068] It is worth noting that the visual task processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the task processing model, the client can also have similar functions to the server, thereby executing the visual task processing methods provided in the embodiments of this specification. In other embodiments, the visual task processing methods provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0069] See Figure 2 , Figure 2 This specification illustrates a flowchart of a visual task processing method according to an embodiment, which specifically includes the following steps:

[0070] Step 202: Obtain the task image and task description information for the target vision task.

[0071] In one or more embodiments of this specification, during visual task processing, the task image and task description information of the target visual task can be obtained.

[0072] It should be noted that the target visual task can be a visual task in different scenarios, such as text detection and recognition, text-based visual question answering, document parsing, etc. Since the objective of the target visual task can be visual text localization, the task image can be an image containing text content. The task image can be an image from different scenarios, such as a bill image in a financial scenario, a street view image in a transportation scenario, or a paper / document / webpage image in an academic research scenario. The task image can also be a video frame containing text content from a task video. The task description information refers to the processing requirements of the target visual task. The task description information can describe the processing requirements of the target visual task in different languages, for example, "Please extract the text from the task image and perform text localization." To improve the efficiency of visual task processing, the task description information can only describe the text extraction requirements, such as "text..." \n Read all text in the image.Answer:".

[0073] In practical applications, there are various ways to acquire the task image and task description information of the target visual task, and the specific method should be selected according to the actual situation. This specification does not limit the methods used in this embodiment. In one possible implementation, the target visual task image sent by the user through a client can be received, and pre-set task description information can be used as the target visual task's task description information. In another possible implementation, the target visual task image and task description information can be read from other data acquisition devices or a database.

[0074] Step 204: Input the task image and task description information into the task processing model to obtain the multimodal feature sequence.

[0075] In one or more embodiments of this specification, after obtaining the task image and task description information of the target visual task, the task image and task description information can be processed using a task processing model to obtain a multimodal feature sequence.

[0076] It's important to note that task processing models can utilize self-attention mechanisms to process input data, outputting self-attention-processed features, i.e., multimodal feature sequences. Task processing models can be deep learning models trained on sample data from visual tasks, where sample data includes sample images, sample descriptions, and text location labels. Task processing models can also be pre-trained large models. Since task descriptions are typically text data and task images are image data, task processing models can also be multimodal large language models. Multimodal large language models are large models capable of processing and understanding multiple types of data (such as text, images, and audio). They can simultaneously process text and other forms of data and capture the correlations between different modalities. A multimodal feature sequence refers to a series of feature representations extracted from data of different modalities, which are integrated to provide a richer and more comprehensive data representation.

[0077] In practical applications, there are various ways to input task images and task description information into the task processing model, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation, the task image and task description information can be directly input into the task processing model to obtain a multimodal feature sequence. In another possible implementation, feature extraction can be performed on the task image and task description information outside the task processing model. For example, when using an LLM model, feature extraction can be performed on the task image and task description information separately to obtain task image features and task description features, which are then input into the task processing model.

[0078] In one optional embodiment of this specification, before inputting the task image and task description information into the task processing model to obtain the multimodal feature sequence, the following steps may also be included:

[0079] Feature extraction is performed on the task image to obtain the task image features;

[0080] Feature extraction is performed on the task description information to obtain task description features;

[0081] Inputting the task image and task description information into the task processing model to obtain a multimodal feature sequence may include the following steps:

[0082] Input the task image features and task description features into the task processing model to obtain a multimodal feature sequence.

[0083] It's important to note that feature extraction refers to the process of selecting and transforming a set of features from raw data. These features can effectively represent the key information in the raw data. Task image features refer to the features extracted from task images that are helpful in completing the target visual task. These features can be various visual attributes such as color, texture, shape, and edges. The required features will vary depending on the task. Task description features refer to the features extracted from task description information that are helpful in completing the target visual task. These features can be text attributes such as keywords and grammatical structures.

[0084] In practical applications, there are various ways to extract features from task images and obtain task image features. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods. In one possible implementation, the task image can be input into an image encoding unit to obtain task image features. In another possible implementation, principal component analysis (PCA) can be used to extract features from the task image. Simultaneously, there are various ways to extract features from task description information and obtain task description features. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods. In one possible implementation, the task description information can be input into a text encoding unit to obtain task description features. In another possible implementation, one-hot encoding can be used to extract features from the task description information and obtain task description features.

[0085] By applying the scheme of the embodiments in this specification, feature extraction is performed on the task image and task description information outside the task processing model, so that the input data of the task processing model is the feature-extracted data, and the task processing model does not have to be a multimodal model, further improving the universality of visual task processing.

[0086] In one or more embodiments of this specification, the above-described feature extraction of the task image to obtain task image features may include the following steps:

[0087] The task image is input into the image encoding unit to obtain the task image features;

[0088] Extracting features from task description information to obtain task description features may include the following steps:

[0089] The task description information is input into the text encoding unit to obtain the task description features.

[0090] It should be noted that the image encoder is used to capture key information in the task image and convert it into a vector representation to obtain the task image features. The text encoding unit is used to capture key information in the task description information and convert it into a vector representation to obtain the task description features. Image encoding units include, but are not limited to, CNN models, ViT models, and SwinTransformer models. Text encoding units include, but are not limited to, bag-of-words models and large language models.

[0091] The scheme implemented in this specification simplifies the feature extraction process by using an image encoding unit to extract features from the task image and a text encoding unit to extract features from the task description information. It can also handle complex data structures and improve the accuracy and efficiency of feature extraction.

[0092] Step 206: Perform category recognition on the multimodal feature sequence to obtain the classification information of the multimodal feature sequence, wherein the classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence.

[0093] In one or more embodiments of this specification, a task image and task description information of the target visual task are obtained; the task image and task description information are input into the task processing model to obtain a multimodal feature sequence; furthermore, the multimodal feature sequence can be classified to obtain classification information of the multimodal feature sequence.

[0094] It's important to note that the purpose of category recognition is to determine the inclusion relationship between the task-processing text and the task image corresponding to the multimodal feature sequence. The task-processing text refers to the text output by the task-processing model when performing text recognition on the task image. Since the input to the task-processing model also includes task description information, the task-processing text typically includes some template content. For example, if the task-processing text is "The text in the image is: coffee," then "The text in the image is:" is the template content, while "coffee" is the text in the task image. Therefore, classification information is specifically used to describe the inclusion relationship between each character in the task-processing text and the task image. The text content included in the task image can be called image recognition text. Inclusion relationship refers to whether the text is within the task image, or whether the task image contains text.

[0095] In practical applications, there are various ways to perform category recognition on multimodal feature sequences and obtain their classification information. The specific method chosen depends on the actual situation, and this specification does not limit the methods used in this embodiment. In one possible implementation, the multimodal feature sequence can be input into a sequence classification unit (Tagging Head) to obtain its classification information. The sequence classification unit is trained based on sample data from a visual task, including sample images, sample description information, text location labels, and text recognition labels. In another possible implementation, the BIO annotation scheme can be used to perform category recognition on the multimodal feature sequence and obtain its classification information. Here, B represents the beginning of the image-recognized text, I represents the interior of the image-recognized text, and O represents a word that does not belong to the image-recognized text.

[0096] For example, suppose the task processing text corresponding to the multimodal feature sequence is "The texts in the image are, Baby, T, Cafe". Inputting the multimodal feature sequence into the sequence classification unit, the classification information of the multimodal feature sequence can be [O, O, O, O, O, O, B-, I-, B-, B-, I-]. Here, O in the classification information indicates that the multimodal feature corresponding to "The texts in the image are," in the task processing text is not a feature corresponding to the text in the task image, and "The texts in the image are," is not a word in the image recognition text.

[0097] In one optional embodiment of this specification, after performing category identification on the multimodal feature sequence to obtain the classification information of the multimodal feature sequence, the following steps may be further included:

[0098] The multimodal feature sequence is input into the text decoding unit to obtain the task processing text;

[0099] Based on the classification information and task processing text, determine the image recognition text of the task image.

[0100] It should be noted that the text decoding unit, also known as the Language Mode Head, receives the multimodal feature sequence output from the last layer of the task processing model and converts it into the probability of each word in the vocabulary, thus obtaining the task processing text. When determining the image recognition text of the task image based on the classification information and the task processing text, the text content corresponding to the classification information "B-" and "I-" in the task processing text can be identified as the image recognition text.

[0101] In practical applications, there are various methods for determining the image recognition text of a task image, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on these methods in its embodiments. In one possible implementation, the multimodal feature sequence can be decoded first to obtain the task processing text, and then classification information can be used to extract the image recognition text of the task image from the task processing text. In another possible implementation, classification information can be used to filter features in the multimodal feature sequence to obtain the features corresponding to the image recognition text, and then the features corresponding to the image recognition text can be decoded to obtain the image recognition text.

[0102] The scheme implemented in this specification uses a text decoding unit to decode multimodal feature sequences to obtain task processing text, and uses classification information to extract image recognition text from the task processing text. This achieves both image text localization and image text recognition.

[0103] Step 208: Based on the classification information and the task image, perform visual feature decoding on the multimodal feature sequence to obtain the text localization information of the task image.

[0104] In one or more embodiments of this specification, a task image and task description information of the target visual task are acquired; the task image and task description information are input into a task processing model to obtain a multimodal feature sequence; after classifying the multimodal feature sequence to obtain classification information of the multimodal feature sequence, further, visual feature decoding can be performed on the multimodal feature sequence based on the classification information and the task image to obtain text positioning information of the task image.

[0105] It should be noted that text positioning information is used to describe the position and shape of text in the task image. Text positioning information may include text position information describing the position of text in the task image, and may also include text visual images describing the shape of text in the task image, depending on the specific circumstances. This specification does not impose any limitations on this. Text position information can be a text positioning matrix in matrix form, or it can be text positioning in natural language form. The text positioning matrix can be a 0-1 matrix, where 0 indicates that there is no text at the position corresponding to the matrix element in the task image, and 1 indicates that there is text at the position corresponding to the matrix element in the task image. Through text positioning information, the position coordinates (e.g., 1, 2), position range (e.g., upper left corner), and text shape (e.g., curved, artistic font) of the text in the task image can be determined.

[0106] In practical applications, there are various ways to obtain text location information of a task image by visually decoding a multimodal feature sequence based on classification information and the task image. The specific method chosen depends on the actual situation, and this specification does not limit the specific approach. One possible implementation involves feature extraction of the task image to obtain task image features; visual feature decoding of the multimodal feature sequence based on these features to obtain initial position information; filtering of text location information from the initial position information using classification information; and determining the text location information of the task image based on the text location information. Another possible implementation involves feature processing of the multimodal feature sequence based on classification information to obtain features to be decoded; visual feature decoding of these features based on the task image features to obtain text location information; and determining the text location information of the task image based on the text location information.

[0107] By applying the scheme of the embodiments in this specification, the classification information of the multimodal feature sequence is determined, thereby more precisely identifying the features related to the task image in the multimodal feature sequence and achieving more refined visual task processing. Through visual feature decoding of the multimodal feature sequence, the features output by the model are decoded into an image mask, enabling the location of text in the image. Simultaneously, during visual task processing, no additional text location auxiliary information needs to be introduced, thus significantly reducing the length of the multimodal feature sequence and further greatly improving the efficiency of visual task processing.

[0108] In one optional embodiment of this specification, the above-described method of visually decoding the multimodal feature sequence based on classification information and the task image to obtain text localization information of the task image may include the following steps:

[0109] Feature extraction is performed on the task image to obtain the task image features;

[0110] Based on the classification information, feature processing is performed on the multimodal feature sequence to obtain the features to be decoded;

[0111] Based on the features of the task image, perform visual feature decoding on the features to be decoded to obtain text location information;

[0112] Based on the text location information, determine the text location information of the task image.

[0113] It should be noted that the features to be decoded refer to the encoded features corresponding to the text in the image recognition. Based on the classification information, feature processing of the multimodal feature sequence can include feature filtering and feature dimension alignment.

[0114] In practical applications, there are multiple ways to determine the text location information of a task image based on text location information. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this method. In one possible implementation, the task image can be segmented based on the text location information to obtain the corresponding text visual image. In another possible implementation, the text location coordinates corresponding to the task image can be generated by aligning the text location matrix with each pixel of the task image. When performing feature processing on the multimodal feature sequence based on classification information, a feature mapping unit can be used for rapid processing to obtain the features to be decoded, or the multimodal feature sequence can be traversed one by one based on the classification information to obtain the features to be decoded. When performing visual feature decoding on the features to be decoded based on the task image features, a feature decoding unit can be used for rapid decoding to obtain the text location information, or a vocabulary can be searched to map the features to be decoded back to the text location information.

[0115] By applying the scheme of the embodiments in this specification, visual feature decoding of multimodal feature sequences is performed to decode the features output by the model into an image mask, thereby enabling the location of text in an image. Through text location information, the text location information of the task image can be determined efficiently and accurately.

[0116] In one optional embodiment of this specification, the above-described feature processing of the multimodal feature sequence based on classification information to obtain the features to be decoded may include the following steps:

[0117] The classification information and multimodal feature sequences are input into the feature mapping unit for feature filtering and feature dimension alignment to obtain the features to be decoded;

[0118] Based on the features of the task image, visual feature decoding is performed on the features to be decoded to obtain text location information, which may include the following steps:

[0119] The task image features and the features to be decoded are input into the feature decoding unit for visual feature decoding to obtain text location information.

[0120] It should be noted that the feature mapping unit can be called a pixel mapping layer (mask projection). The feature mapping unit can use classification information to filter features from multimodal feature sequences and align the feature dimensions of the filtered features with the processing dimensions of the feature decoding unit. This feature dimension alignment can be either dimensionality increase or dimensionality decrease, depending on the specific situation; this specification does not impose any limitations on this approach in the embodiments. The feature decoding unit can be called a pixel decoder. The feature decoding unit is used to decode the input features into an image mask, achieving fine-grained image text localization. The feature mapping unit and the visual feature decoding unit can be neural network models trained based on sample data from a sample visual task, where the sample data includes sample images, sample description information, and sample localization labels.

[0121] The scheme implemented in this specification improves the efficiency of text location information generation by using a mapping unit and a feature decoding unit to decode multimodal feature sequences.

[0122] In one optional embodiment of this specification, the text location information includes a text visual image, which reflects the shape of the area occupied by the image recognition text in the task image; determining the text location information of the task image based on the text location information may include the following steps:

[0123] Based on the text location information, the task image is segmented to obtain the corresponding text visual image.

[0124] It should be noted that a text visual image refers to an image obtained by segmenting the image using the boundaries of the text in the image recognition process as the image boundaries. Since a text visual image can reflect the shape of the area occupied by the text in the task image, it can be images of different shapes, such as quadrilaterals, rhombuses, ellipses, etc.

[0125] In practical applications, taking the text location information as the text positioning matrix as an example, when segmenting the task image according to the text positioning matrix, the task image can be divided into multiple image blocks according to the matrix elements of the text positioning matrix. The number of image blocks is the same as the number of matrix elements. Furthermore, the image block corresponding to matrix element 1 is retained, and the image block corresponding to matrix element 0 is cropped to obtain the text visual image.

[0126] By applying the solution of the embodiments in this specification, the task image is segmented according to the text location information to obtain the text visual image corresponding to the task image, thereby realizing the visualization of text location and improving the user experience.

[0127] The following is in conjunction with the appendix Figure 3 Taking the application of the visual task processing method provided in this specification in an image processing scenario as an example, the visual task processing method will be further explained. Figure 3 This specification shows a flowchart of an image processing method according to an embodiment, which specifically includes the following steps:

[0128] Step 302: Obtain the task image and task description information for the target vision task.

[0129] Step 304: Input the task image and task description information into the task processing model to obtain the multimodal feature sequence.

[0130] Step 306: Perform category recognition on the multimodal feature sequence to obtain the classification information of the multimodal feature sequence, wherein the classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence.

[0131] Step 308: Based on the classification information and the task image, perform visual feature decoding on the multimodal feature sequence to obtain the text localization information of the task image.

[0132] Step 310: Based on the text location information, perform text processing on the task image to obtain the target task image.

[0133] It should be noted that the implementation methods of steps 302 to 308 are the same as those of steps 202 to 208 described above, and will not be repeated in this embodiment. Text processing includes, but is not limited to, text erasure, text replacement, text font adjustment, text color adjustment, text translation, text annotation, etc., and the specific selection is made according to the actual situation. This embodiment does not impose any limitations on this. Taking text erasure as an example, the location and shape of the text in the task image can be determined based on the text positioning information, thereby accurately locating the image block where the text content is located. By replacing the image block where the text content is located with a preset image block, the target task image after text erasure can be obtained.

[0134] By applying the solutions of the embodiments in this specification, since the visual task processing method can obtain text location information more efficiently and with greater precision, the text location information obtained by the visual task processing method can efficiently and accurately realize image processing, thereby improving the accuracy of the target task image.

[0135] See Figure 4 , Figure 4 This specification illustrates a flowchart of a task processing model training method according to an embodiment, which specifically includes the following steps:

[0136] Step 402: Obtain sample data for the visual task, including sample images, sample description information, and text location labels.

[0137] Step 404: Input the sample images and sample description information into the task processing model to obtain the sample feature sequence.

[0138] Step 406: Perform category identification on the sample feature sequence to obtain sample classification information of the sample feature sequence, wherein the sample classification information is used to describe the inclusion relationship between the sample processing text and the sample image corresponding to the sample feature sequence.

[0139] Step 408: Based on the sample classification information and sample image, perform visual feature decoding on the sample feature sequence to obtain the localization prediction information of the sample image.

[0140] Step 410: Adjust the parameters of the task processing model based on the location prediction information and text location labels to obtain the trained task processing model.

[0141] It should be noted that the implementation methods of steps 402 to 408 are the same as those of steps 202 to 208 described above. The definition of the sample visual task can refer to the definition of the target visual task described above; the definition of the sample image can refer to the definition of the task image described above; the definition of the sample description information can refer to the definition of the task description information described above; the definition of the sample feature sequence can refer to the definition of the multimodal feature sequence described above; the definition of the sample classification information can refer to the definition of the classification information described above; and the definition of the localization prediction information can refer to the definition of the text localization information described above. These details will not be elaborated further in the embodiments of this specification. The training method of the task processing model is supervised training, that is, the sample image and sample description information carry real text localization information, which is the text localization label. The text localization label is the generation target of the task generation model.

[0142] In practical applications, when adjusting the parameters of the task processing model based on localization prediction information and text localization labels, a mask loss value can be calculated based on the localization prediction information and text localization labels. The model parameters of the task processing model are then adjusted based on the mask loss value until the training process meets a preset stopping condition, resulting in a trained task processing model. The mask loss value is used to optimize the localization mask of the image. Many functions can be used to calculate the mask loss value, such as Binary Cross-Entropy Loss and / or Dice Similarity Loss, etc. The specific function to be selected depends on the actual situation, and this specification does not impose any limitations on this. Binary Cross-Entropy Loss focuses more on pixel-level classification accuracy. Dice Loss focuses more on the similarity of the overall shape and has better robustness to imbalanced datasets. The preset stopping condition includes, but is not limited to, the mask loss value being less than or equal to a preset threshold and the number of iterations reaching a preset number of iterations. The preset threshold and preset number of iterations are selected based on the actual situation, and this specification does not impose any limitations on this.

[0143] In one possible implementation of this specification, after calculating the mask loss value, it can be compared with a preset threshold. For example, if the mask loss value is greater than the preset threshold, it indicates a large difference between the location prediction information and the text location label, indicating poor processing capability of the task processing model. In this case, the model parameters of the task processing model can be adjusted, and the model can continue to be trained using sample data until the mask loss value is less than or equal to the preset threshold, indicating a smaller difference between the location prediction information and the text location label, reaching the preset stopping condition, and obtaining a completed task processing model. In another possible implementation of this specification, in addition to comparing the mask loss value and the preset threshold, the number of iterations can also be considered to determine whether the current task processing model has completed training. For example, if the mask loss value is greater than the preset threshold, the model parameters of the task processing model are adjusted, and the model can continue to be trained using data until the preset number of iterations is reached, at which point iteration is stopped, and a completed task processing model is obtained.

[0144] By applying the scheme of the embodiments of this specification, the parameters of the task processing model are adjusted according to the positioning prediction information and text positioning labels to obtain the trained task processing model. This optimizes the image positioning capability of the task processing model, so that the sample feature sequence output by the trained task processing model can be used for image text positioning. Furthermore, the sample data does not include additional text positioning auxiliary information, which greatly reduces the length of the sample feature sequence and further significantly improves the training efficiency of the task processing model.

[0145] It's worth noting that when training the task processing model, the parameters of the image encoding unit, text encoding unit, text decoding unit, sequence classification unit, feature mapping unit, and feature decoding unit can be adjusted. The training sample data does not need to contain all fine-grained supervised annotations; only a certain proportion (e.g., 20%) of the data is required for training, achieving efficient generalization. Through cross-modal fine-grained supervision, better representations rich in location, visual content, and textual knowledge can be obtained. Leveraging localization capabilities and more direct visual signals, the prediction inaccuracies and illusion problems of the task processing model can be optimized.

[0146] In one optional embodiment of this specification, the above-described method of visually decoding the sample feature sequence based on sample classification information and sample image to obtain the localization prediction information of the sample image may include the following steps:

[0147] Input the sample classification information and sample feature sequence into the feature mapping unit to obtain the sample features to be decoded;

[0148] The sample image features and the sample features to be decoded are input into the feature decoding unit to obtain the sample localization matrix, wherein the sample image features are obtained based on feature extraction of the sample image;

[0149] Based on the sample localization matrix, determine the localization prediction information of the sample image;

[0150] Based on the location prediction information and text location labels, the parameters of the task processing model are adjusted to obtain the trained task processing model, which may include the following steps:

[0151] Based on the location prediction information and text location labels, adjust the parameters of the task processing model, feature mapping unit, and feature decoding unit to obtain the trained task processing model, trained feature mapping unit, and trained feature decoding unit.

[0152] It should be noted that the implementation of "inputting sample classification information and sample feature sequence into the feature mapping unit to obtain sample features to be decoded; inputting sample image features and sample features to be decoded into the feature decoding unit to obtain sample localization matrix, wherein the sample image features are obtained based on feature extraction of the sample image; determining the localization prediction information of the sample image based on the sample localization matrix" can refer to the implementation of "inputting classification information and multimodal feature sequence into the feature mapping unit for feature filtering and feature dimension alignment to obtain features to be decoded; inputting task image features and features to be decoded into the feature decoding unit for visual feature decoding to obtain text location information; determining the text location information of the task image based on the text location information" will not be elaborated further in the embodiments of this specification.

[0153] In practical applications, when adjusting the parameters of the task processing model, feature mapping unit, and feature decoding unit based on the location prediction information and text location tags, the mask loss value can be calculated based on the location prediction information and text location tags. The parameters of the task processing model, feature mapping unit, and feature decoding unit can be adjusted based on the mask loss value until the training process meets the preset stopping condition, thus obtaining the trained task processing model, trained feature mapping unit, and trained feature decoding unit.

[0154] By applying the scheme of the embodiments of this specification, the parameters of the task processing model, feature mapping unit, and feature decoding unit are adjusted according to the positioning prediction information and text positioning labels, thereby obtaining the trained task processing model, trained feature mapping unit, and trained feature decoding unit, and optimizing the image positioning capabilities of the task processing model, feature mapping unit, and feature decoding unit.

[0155] In one optional embodiment of this specification, the sample data further includes text recognition tags; the above-described category identification of the sample feature sequence to obtain sample classification information may include the following steps:

[0156] Input the sample feature sequence into the sequence classification unit to obtain the sample classification information of the sample feature sequence;

[0157] Based on the location prediction information and text location labels, the parameters of the task processing model are adjusted to obtain the trained task processing model, which may include the following steps:

[0158] Based on the location prediction information and text location labels, sample classification information and text recognition labels, adjust the parameters of the task processing model and sequence classification unit to obtain the trained task processing model and trained sequence classification unit.

[0159] It should be noted that the text recognition label refers to the actual text in the sample image. After obtaining the sample classification information of the sample feature sequence, the classification information and the text recognition label can be used to calculate the classification loss value. The classification loss value is used to optimize the model's ability to classify whether the text belongs to the text in the image. There are various functions for calculating the classification loss value, such as the cross-entropy loss function and the focal loss function. The specific function should be selected according to the actual situation, and the embodiments in this specification do not impose any limitations on it.

[0160] In practical applications, after calculating the mask loss value based on the location prediction information and text location labels, and the classification loss value based on the sample classification information and text recognition labels, the classification loss value and the mask loss value can be weighted. The parameters of the task processing model and the sequence classification unit can be adjusted based on the weighted loss value to obtain the trained task processing model and the trained sequence classification unit.

[0161] By applying the scheme of the embodiments in this specification, the parameters of the task processing model and the sequence classification unit are adjusted according to the location prediction information and text location tags, sample classification information and text recognition tags, so as to obtain the trained task processing model and the trained sequence classification unit, thereby optimizing the text classification ability of the task processing model and the sequence classification unit.

[0162] In one optional embodiment of this specification, the sample data further includes text recognition tags; after performing category recognition on the sample feature sequence to obtain sample classification information, the method may further include the following steps:

[0163] The sample feature sequence is input into the text decoding unit to obtain the sample processed text;

[0164] Based on the sample classification information and sample processing text, determine the sample recognition text of the sample image;

[0165] Based on the location prediction information and text location labels, the parameters of the task processing model are adjusted to obtain the trained task processing model, which may include the following steps:

[0166] Based on the location prediction information, text location labels, sample recognition text, and text recognition labels, the parameters of the task processing model and text decoding unit are adjusted to obtain the trained task processing model and trained text decoding unit.

[0167] It should be noted that the implementation method of "inputting the sample feature sequence into the text decoding unit to obtain the sample processing text; determining the sample recognition text of the sample image based on the sample classification information and the sample processing text" is the same as the implementation method of "inputting the multimodal feature sequence into the text decoding unit to obtain the task processing text; determining the image recognition text of the task image based on the classification information and the task processing text," and will not be repeated in this specification. After obtaining the sample recognition text, the text prediction loss value can be calculated using the sample recognition text and the text recognition label. The text prediction loss value is used to optimize the text prediction ability of the model. There are various functions for calculating the text prediction loss value, such as the cross-entropy loss function, the L1 norm loss function, the maximum loss function, the mean squared error loss function, the log loss function, etc. The specific function should be selected according to the actual situation, and this specification does not impose any limitations on it.

[0168] In practical applications, after calculating the mask loss value based on the location prediction information and text location labels, and calculating the text prediction loss value based on the sample recognized text and text recognition labels, the text prediction loss value and the mask loss value can be weighted. The parameters of the task processing model and the text decoding unit can be adjusted based on the weighted loss value to obtain the trained task processing model and the trained text decoding unit.

[0169] By applying the scheme of the embodiments in this specification, the parameters of the task processing model and the text decoding unit are adjusted according to the location prediction information, text location tags, sample recognition text and text recognition tags, so as to obtain the trained task processing model and the trained text decoding unit, thereby optimizing the text prediction capability of the task processing model and the text decoding unit.

[0170] See Figure 5 , Figure 5 This specification illustrates a flowchart of a visual task processing method according to an embodiment. The method is based on a joint approach for visual text localization using an image encoding unit, a text encoding unit, a task processing model, a text decoding unit, a sequence classification unit, a feature mapping unit, and a feature decoding unit. The image encoding unit performs localization encoding on the image; therefore, it can be called a grounding encoder. The visual task processing method specifically includes:

[0171] The process involves acquiring the task image and task description information for the target visual task; inputting the task image into an image encoding unit to obtain task image features; inputting the task description information into a text encoding unit to obtain task description features; inputting the task image features and task description features into a task processing model to obtain a multimodal feature sequence; and inputting the multimodal feature sequence in parallel into a text decoding unit (predicting the task processing text), a sequence classification unit (predicting whether each entity in the task processing text belongs to the task image), and a feature mapping unit (for feature filtering and feature dimension alignment of the multimodal feature sequence). The multimodal feature sequence input into the feature mapping unit is then passed to the feature decoding unit for visual mask decoding. Specifically, the multimodal feature sequence is input into the text decoding unit to obtain the task processing text; the multimodal feature sequence is input into the sequence classification unit to obtain classification information; the classification information and the multimodal feature sequence are input into the feature mapping unit for feature filtering and feature dimension alignment to obtain features to be decoded; the task image features and features to be decoded are input into the feature decoding unit for visual feature decoding to obtain a text localization matrix; and the text localization information of the task image is determined based on the text localization matrix. Furthermore, based on classification information and task processing text, image recognition text of the task image can be determined, that is, words or phrases in the task image can be extracted.

[0172] By combining a feature mapping unit and a feature decoding unit, the representation of the task processing model can be mapped and decoded into an image mask, enabling fine-grained text localization in images. Furthermore, this method requires no additional localization tokens, making it a non-intrusive processing solution that does not alter the format of the text data. This significantly reduces the length of the model's prediction results during the text localization task in the inference stage, particularly in text-dense document images, where it can substantially improve image text localization efficiency.

[0173] See Figure 6 , Figure 6 This specification illustrates a flowchart of an information processing method based on a task processing model, according to an embodiment of the present invention. The information processing method based on the task processing model is applied to a task platform and specifically includes the following steps:

[0174] Step 602: Receive a model request sent by the terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0175] Step 604: Based on the model request, determine the target task processing model from multiple task processing models, wherein the multiple task processing models are trained based on the task processing model training method.

[0176] It should be noted that the target task processing model is a task processing model applicable to the target scenario. There are multiple ways to determine the target task processing model from multiple task processing models based on a model request; the specific method should be selected according to the actual situation, and this specification does not impose any limitations on this method. In one possible implementation of this specification, the corresponding target task processing model can be searched from at least one task processing model included in the model library based on the model request. In another possible implementation, the target task processing model can be trained and obtained based on the model request. In yet another optional implementation, the target task processing model can be constructed based on the model request.

[0177] For example, based on the scene identifier of the target scene, at least one pre-trained task processing model can be searched from the model library. Then, based on the model specification parameters, an initial task processing model can be selected from the at least one task processing model. Finally, based on the scene input data of the target scene, the selected initial task processing model is trained to obtain a target task processing model suitable for user needs. The at least one task processing model can be based on... Figure 4 The task processing model shown was trained using the training method described in this specification, and will not be repeated in the embodiments.

[0178] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0179] In one optional embodiment of this specification, the model request includes a scene identifier of the target scene; the process of determining the target task processing model from multiple task processing models based on the model request may include the following steps:

[0180] Based on the scene identifier of the target scene, the target task processing model suitable for the target scene is searched from the model library. The model library stores multiple task processing models suitable for different visual task scenes.

[0181] It should be noted that scene identifiers are unique or specific labels used to distinguish different visual task scenes. The model library is a database for storing and managing various pre-trained deep learning models. Multiple task processing models adapted to different visual task scenes cover different application scenarios and needs. The model library allows users to select appropriate models according to their needs, or directly use models for visual task processing through application programming interfaces.

[0182] Multiple task processing models adapted to different visual task scenarios are stored in the model library. Each model is optimized for a specific application environment. Any task processing model is based on... Figure 4 The training method shown is used to train the model obtained from the task processing model. For example, based on the scene identifier "document detection and recognition" of the target scene, a target task processing model suitable for the document detection and recognition scene can be found from the model library.

[0183] The solutions implemented in this specification, based on scenario requirements, accurately identify the target task processing model suitable for the scenario through scenario identification, making visual task processing more accurate and more scenario-appropriate, thereby improving user experience and visual task processing quality.

[0184] In one optional embodiment of this specification, the model request includes scene input data of the target scene; the above-mentioned determination of the target task processing model from multiple task processing models based on the model request may include the following steps:

[0185] From multiple task processing models, determine the initial task processing model that is suitable for the target scenario;

[0186] Based on the scene input data of the target scene, the initial task processing model is trained to obtain the target task processing model.

[0187] It's important to note that the initial task processing model refers to the model among multiple task processing models that is suitable for the target scenario. This initial task processing model may not only be applicable to the target scenario but also to other scenarios; it's a general task processing model suitable for different scenarios. While the initial task processing model can be used for visual task processing, the results may not be ideal. In such cases, the initial task processing model can be optimized based on the scene input data of the target scenario. For example, optimizing the initial task processing model based on the scene input data of a document detection and recognition scenario can yield a target task processing model suitable for document recognition. The scene input data of the target scenario can be understood as the sample data of the visual tasks within the target scenario. The method for training the initial task processing model based on the scene input data of the target scenario to obtain the target task processing model can be found in [reference needed]. Figure 4 The training method of the task processing model shown in this specification will not be described again in the embodiments.

[0188] By applying the solutions in the embodiments of this specification, based on scenario requirements, a general initial task processing model is further trained using scenario input data to obtain a target task processing model adapted to the scenario. This makes the target task processing model more closely fit the scenario, thereby improving the user experience and the processing quality of visual tasks.

[0189] In one optional embodiment of this specification, the model request includes model specification parameters; the process of determining the target task processing model from multiple task processing models based on the model request may include the following steps:

[0190] Based on the model specification parameters, the corresponding target task processing model is searched from the model library, which stores multiple task processing models with different model specification parameters.

[0191] It's important to note that model specifications refer to the various parameters that define the model's structure and behavior. These parameters can be broadly categorized into two types: model parameters (learnable parameters) and hyperparameters. Model parameters are those automatically adjusted during model training via backpropagation, including but not limited to weight matrices and biases. For example, in a simple fully connected layer, the weight matrix is ​​a two-dimensional tensor connecting neurons in the input and output layers; the biases are one-dimensional vectors providing additional offset values ​​for each output neuron. Hyperparameters are parameters set before model training begins, controlling the model's learning process and architecture. Hyperparameters include, but are not limited to, the learning rate and the number of neurons per layer, chosen based on specific requirements.

[0192] By applying the solutions in the embodiments of this specification, based on the model specification parameters, the corresponding target task processing model can be accurately found, ensuring the efficient and stable operation of the target task processing model and improving the user experience.

[0193] In one optional embodiment of this specification, after determining the target task processing model from multiple task processing models based on the model request, the following steps may be further included:

[0194] Deploy the target task processing model, and build a visual task processing interface based on the target task processing model so that the terminal device can schedule the target task processing model to execute the target visual task.

[0195] It should be noted that the visual task processing interface is an interactive programming interface for terminal devices to schedule target task processing models for visual task processing, and is usually provided in the form of an application programming interface (API). Through the visual task processing interface, users can input task data for the target visual task, such as task images and task description information, to perform image-text localization.

[0196] In practical applications, there are various ways to deploy the target task processing model, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this approach. One possible implementation of this specification is to deploy the target task processing model on cloud-side devices using infrastructure provided by a cloud service provider. Another possible implementation of this specification is to deploy the target task processing model on edge devices using a lightweight framework. For example, the target task processing model can be deployed on a distributed system, and a visual task processing interface can be built based on the target task processing model and provided to terminal devices, enabling the terminal devices to schedule the target task processing model to execute target visual tasks.

[0197] By applying the solutions provided in the embodiments of this specification, deploying the target task processing model, and building a visual task processing interface based on the target task processing model, the terminal device can efficiently call the target task processing model, thereby improving the processing quality and response speed of the target visual task.

[0198] See Figure 7 , Figure 7 This specification shows a schematic diagram of the structure of a task platform according to an embodiment of the present specification. The task platform includes a request interface 702 and a response unit 704.

[0199] Request interface 702 is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0200] The response unit 704 is used to determine the target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the task processing model training method.

[0201] In one optional embodiment of this specification, the task platform further includes a visual task processing interface, which is constructed based on the target task processing model.

[0202] The visual task processing interface is used to allow terminal devices to schedule and execute target visual tasks.

[0203] By applying the solutions in the embodiments of this specification, the task platform adapts to user needs to obtain target task processing models, realizes personalized model services, provides users with an efficient, flexible and easy-to-use model service platform, and improves user experience.

[0204] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the information processing method based on the task processing model described above belong to the same concept. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution of the information processing method based on the task processing model described above.

[0205] Corresponding to the above embodiments of the visual task processing method, this specification also provides embodiments of the visual task processing device. Figure 8 A schematic diagram of a visual task processing device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0206] The first acquisition module 802 is configured to acquire the task image and task description information of the target visual task;

[0207] The first input module 804 is configured to input the task image and task description information into the task processing model to obtain a multimodal feature sequence;

[0208] The first recognition module 806 is configured to perform category recognition on the multimodal feature sequence to obtain classification information of the multimodal feature sequence, wherein the classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence.

[0209] The first decoding module 808 is configured to perform visual feature decoding on the multimodal feature sequence based on classification information and task image to obtain text positioning information of the task image.

[0210] Optionally, the first decoding module 808 is further configured to extract features from the task image to obtain task image features; perform feature processing on the multimodal feature sequence according to classification information to obtain features to be decoded; perform visual feature decoding on the features to be decoded according to the task image features to obtain text location information; and determine the text location information of the task image according to the text location information.

[0211] Optionally, the first decoding module 808 is further configured to input classification information and multimodal feature sequences into the feature mapping unit for feature filtering and feature dimension alignment to obtain features to be decoded; and input task image features and features to be decoded into the feature decoding unit for visual feature decoding to obtain text location information.

[0212] Optionally, the text location information includes a text visual image, which reflects the shape of the area occupied by the image recognition text in the task image; the first decoding module 808 is further configured to segment the task image according to the text location information to obtain the text visual image corresponding to the task image.

[0213] Optionally, the device further includes: an extraction module configured to extract features from the task image to obtain task image features; and to extract features from the task description information to obtain task description features; and a first input module 804 further configured to input the task image features and task description features into the task processing model to obtain a multimodal feature sequence.

[0214] Optionally, the extraction module is further configured to input the task image into the image encoding unit to obtain task image features; and to input the task description information into the text encoding unit to obtain task description features.

[0215] Optionally, the device further includes: a fourth input module configured to input a multimodal feature sequence into a text decoding unit to obtain task processing text; and to determine image recognition text of the task image based on classification information and task processing text.

[0216] By applying the scheme of the embodiments in this specification, the classification information of the multimodal feature sequence is determined, thereby more precisely identifying the features related to the task image in the multimodal feature sequence and achieving more refined visual task processing. Through visual feature decoding of the multimodal feature sequence, the features output by the model are decoded into an image mask, enabling the location of text in the image. Simultaneously, during visual task processing, no additional text location auxiliary information needs to be introduced, thus significantly reducing the length of the multimodal feature sequence and further greatly improving the efficiency of visual task processing.

[0217] The above is an illustrative scheme of a visual task processing device according to this embodiment. It should be noted that the technical solution of this visual task processing device and the technical solution of the above-described visual task processing method belong to the same concept. For details not described in detail in the technical solution of the visual task processing device, please refer to the description of the technical solution of the above-described visual task processing method.

[0218] Corresponding to the above-described image processing method embodiments, this specification also provides embodiments of image processing apparatus. Figure 9 A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification is shown. Figure 9 As shown, the device includes:

[0219] The second acquisition module 902 is configured to acquire the task image and task description information of the target visual task;

[0220] The second input module 904 is configured to input the task image and task description information into the task processing model to obtain a multimodal feature sequence;

[0221] The second recognition module 906 is configured to perform category recognition on the multimodal feature sequence to obtain classification information of the multimodal feature sequence, wherein the classification information is used to describe the inclusion relationship between the task processing text and the task image corresponding to the multimodal feature sequence.

[0222] The second decoding module 908 is configured to perform visual feature decoding on the multimodal feature sequence based on classification information and task image to obtain text positioning information of the task image.

[0223] The processing module 910 is configured to perform text processing on the task image based on the text positioning information to obtain the target task image.

[0224] By applying the solutions of the embodiments in this specification, since the visual task processing method can obtain text location information more efficiently and with greater precision, the text location information obtained by the visual task processing method can efficiently and accurately realize image processing, thereby improving the accuracy of the target task image.

[0225] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0226] Corresponding to the above-described embodiments of the task processing model training method, this specification also provides embodiments of the task processing model training apparatus. Figure 10 A schematic diagram of a task processing model training device according to one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0227] The third acquisition module 1002 is configured to acquire sample data for the sample visual task, wherein the sample data includes sample images, sample description information and text positioning labels.

[0228] The third input module 1004 is configured to input sample images and sample description information into the task processing model to obtain sample feature sequences.

[0229] The third recognition module 1006 is configured to perform category recognition on the sample feature sequence to obtain sample classification information of the sample feature sequence, wherein the sample classification information is used to describe the inclusion relationship between the sample processing text and the sample image corresponding to the sample feature sequence.

[0230] The third decoding module 1008 is configured to perform visual feature decoding on the sample feature sequence based on the sample classification information and the sample image to obtain the localization prediction information of the sample image.

[0231] The adjustment module 1010 is configured to adjust the parameters of the task processing model based on the location prediction information and text location labels to obtain the trained task processing model.

[0232] Optionally, the third decoding module 1008 is further configured to input sample classification information and sample feature sequence into the feature mapping unit to obtain sample features to be decoded; input sample image features and sample features to be decoded into the feature decoding unit to obtain a sample localization matrix, wherein the sample image features are obtained based on feature extraction of the sample image; determine the localization prediction information of the sample image according to the sample localization matrix; and the adjustment module 1010 is further configured to adjust the parameters of the task processing model, the feature mapping unit, and the feature decoding unit according to the localization prediction information and the text localization label to obtain the trained task processing model, the trained feature mapping unit, and the trained feature decoding unit.

[0233] Optionally, the sample data also includes text recognition labels; the third recognition module 1006 is further configured to input the sample feature sequence into the sequence classification unit to obtain the sample classification information of the sample feature sequence; the adjustment module 1010 is further configured to adjust the parameters of the task processing model and the sequence classification unit according to the positioning prediction information and the text positioning label, the sample classification information and the text recognition label, to obtain the trained task processing model and the trained sequence classification unit.

[0234] Optionally, the sample data also includes text recognition labels; the device further includes: a fifth input module configured to input the sample feature sequence into the text decoding unit to obtain the sample processing text; and to determine the sample recognition text of the sample image based on the sample classification information and the sample processing text; and an adjustment module 1010 further configured to adjust the parameters of the task processing model and the text decoding unit based on the positioning prediction information and the text positioning label, the sample recognition text and the text recognition label, to obtain the trained task processing model and the trained text decoding unit.

[0235] By applying the scheme of the embodiments of this specification, the parameters of the task processing model are adjusted according to the localization prediction information and the text localization label to obtain the trained task processing model. This allows the sample feature sequence output by the trained task processing model to be used for image text localization. Furthermore, the sample data does not include additional text localization auxiliary information, which greatly reduces the length of the sample feature sequence and further significantly improves the training efficiency of the task processing model.

[0236] The above is a schematic scheme of a task processing model training device according to this embodiment. It should be noted that the technical solution of this task processing model training device and the technical solution of the task processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the task processing model training device, please refer to the description of the technical solution of the task processing model training method described above.

[0237] Corresponding to the above-described embodiments of the information processing method based on the task processing model, this specification also provides embodiments of the information processing apparatus based on the task processing model. Figure 11 A schematic diagram of the structure of an information processing device based on a task processing model provided in one embodiment of this specification is shown. Figure 11 As shown, the device is applied to a mission platform and includes:

[0238] The receiving module 1102 is configured to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: a scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0239] The determination module 1104 is configured to determine the target task processing model from multiple task processing models based on a model request, wherein the multiple task processing models are trained based on a task processing model training method.

[0240] Optionally, the model request includes a scene identifier of the target scene; the determination module 1104 is further configured to search for a target task processing model suitable for the target scene from the model library based on the scene identifier of the target scene, wherein the model library stores multiple task processing models suitable for different visual task scenes.

[0241] Optionally, the model request includes scene input data of the target scene; the determination module 1104 is further configured to determine an initial task processing model suitable for the target scene from multiple task processing models; and to train the initial task processing model based on the scene input data of the target scene to obtain the target task processing model.

[0242] Optionally, the model request includes model specification parameters; the determination module 1104 is further configured to search for the corresponding target task processing model from the model library based on the model specification parameters, wherein the model library stores multiple task processing models with different model specification parameters.

[0243] Optionally, the device further includes: a building module configured to deploy a target task processing model and, based on the target task processing model, build a visual task processing interface to enable the terminal device to schedule the target task processing model to execute a target visual task.

[0244] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0245] The above is an illustrative scheme of an information processing device based on a task processing model according to this embodiment. It should be noted that the technical solution of this information processing device based on a task processing model belongs to the same concept as the technical solution of the information processing method based on a task processing model described above. For details not described in detail in the technical solution of the information processing device based on a task processing model, please refer to the description of the technical solution of the information processing method based on a task processing model described above.

[0246] Figure 12 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0247] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0248] In one embodiment of this specification, the aforementioned components of the computing device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0249] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1200 can also be a mobile or stationary server.

[0250] The processor 1220 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned visual task processing method, image processing method, task processing model training method, or information processing method based on task processing model.

[0251] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the above-mentioned visual task processing method, image processing method, task processing model training method, and information processing method based on task processing model. For details not described in detail in the technical solution of the computing device, please refer to the description of the above-mentioned technical solutions of visual task processing method, image processing method, task processing model training method, or information processing method based on task processing model.

[0252] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual task processing method, image processing method, task processing model training method, or information processing method based on a task processing model.

[0253] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the above-mentioned visual task processing method, image processing method, task processing model training method, and information processing method based on task processing model. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the above-mentioned visual task processing method, image processing method, task processing model training method, or information processing method based on task processing model.

[0254] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described visual task processing method, image processing method, task processing model training method, or information processing method based on a task processing model.

[0255] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the above-mentioned visual task processing method, image processing method, task processing model training method, and information processing method based on task processing model. For details not described in detail in the technical solution of the computer program product, please refer to the description of the above-mentioned visual task processing method, image processing method, task processing model training method, or information processing method based on task processing model.

[0256] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0257] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0258] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0259] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0260] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / point> < / box>

Claims

1. A visual task processing method, comprising: obtaining a task image and task description information of a target visual task; inputting the task image and the task description information into a task processing model to obtain a multi-modal feature sequence; performing category recognition on the multi-modal feature sequence to obtain classification information of the multi-modal feature sequence, wherein the classification information is used to describe a containing relationship between a task processing text corresponding to the multi-modal feature sequence and the task image; performing visual feature decoding on the multi-modal feature sequence according to the classification information and the task image to obtain text positioning information of the task image.

2. The method of claim 1, wherein the performing visual feature decoding on the multi-modal feature sequence according to the classification information and the task image to obtain the text positioning information of the task image comprises: performing feature extraction on the task image to obtain a task image feature; performing feature processing on the multi-modal feature sequence according to the classification information to obtain a to-be-decoded feature; performing visual feature decoding on the to-be-decoded feature according to the task image feature to obtain text position information; and determining the text positioning information of the task image according to the text position information.

3. The method of claim 2, wherein the performing feature processing on the multi-modal feature sequence according to the classification information to obtain the to-be-decoded feature comprises: inputting the classification information and the multi-modal feature sequence into a feature mapping unit to perform feature screening and feature dimension alignment to obtain the to-be-decoded feature; and the performing visual feature decoding on the to-be-decoded feature according to the task image feature to obtain the text position information comprises: inputting the task image feature and the to-be-decoded feature into a feature decoding unit to perform visual feature decoding to obtain the text position information.

4. The method of claim 2, wherein the text positioning information comprises a text visual image, and the text visual image is used to reflect a shape of an area occupied by the image recognition text in the task image; and the determining the text positioning information of the task image according to the text position information comprises: segmenting the task image according to the text position information to obtain a text visual image corresponding to the task image.

5. The method of claim 1, wherein before the inputting the task image and the task description information into the task processing model to obtain the multi-modal feature sequence, the method further comprises: performing feature extraction on the task image to obtain a task image feature; and performing feature extraction on the task description information to obtain a task description feature; and the inputting the task image and the task description information into the task processing model to obtain the multi-modal feature sequence comprises: inputting the task image feature and the task description feature into the task processing model to obtain the multi-modal feature sequence.

6. The method of claim 5, wherein the performing feature extraction on the task image to obtain the task image feature comprises: inputting the task image into an image encoding unit to obtain the task image feature. The feature extraction is performed on the task description information to obtain task description features. The task description information is input into a text encoding unit to obtain task description features.

7. The method of claim 1, after the category recognition on the multi-modal feature sequence to obtain classification information of the multi-modal feature sequence, further comprising: inputting the multi-modal feature sequence into a text decoding unit to obtain a task processing text; determining an image recognition text of the task image according to the classification information and the task processing text.

8. An image processing method, comprising: obtaining a task image and task description information of a target visual task; inputting the task image and the task description information into a task processing model to obtain a multi-modal feature sequence; performing category recognition on the multi-modal feature sequence to obtain classification information of the multi-modal feature sequence, wherein the classification information is used to describe a containing relationship between a task processing text corresponding to the multi-modal feature sequence and the task image; performing visual feature decoding on the multi-modal feature sequence according to the classification information and the task image to obtain text positioning information of the task image; performing text processing on the task image according to the text positioning information to obtain a target task image.

9. A task processing model training method, comprising: obtaining sample data of a sample visual task, wherein the sample data comprises a sample image, sample description information and a text positioning label; inputting the sample image and the sample description information into a task processing model to obtain a sample feature sequence; performing category recognition on the sample feature sequence to obtain sample classification information of the sample feature sequence, wherein the sample classification information is used to describe a containing relationship between a sample processing text corresponding to the sample feature sequence and the sample image; performing visual feature decoding on the sample feature sequence according to the sample classification information and the sample image to obtain positioning prediction information of the sample image; adjusting parameters of the task processing model according to the positioning prediction information and the text positioning label to obtain a trained task processing model.

10. The method of claim 9, wherein the visual feature decoding on the sample feature sequence according to the sample classification information and the sample image to obtain the positioning prediction information of the sample image comprises: inputting the sample classification information and the sample feature sequence into a feature mapping unit to obtain sample to-be-decoded features; inputting sample image features and the sample to-be-decoded features into a feature decoding unit to obtain a sample positioning matrix, wherein the sample image features are obtained by performing feature extraction on the sample image; determining the positioning prediction information of the sample image according to the sample positioning matrix; the adjusting of the parameters of the task processing model according to the positioning prediction information and the text positioning label to obtain the trained task processing model comprises: According to the positioning prediction information and the text positioning label, parameters of the task processing model, the feature mapping unit and the feature decoding unit are adjusted to obtain a trained task processing model, a trained feature mapping unit and a trained feature decoding unit.

11. The method of claim 9, wherein the sample data further comprises a text recognition label. The category recognition on the sample feature sequence comprises: The sample feature sequence is input into a sequence classification unit to obtain sample classification information of the sample feature sequence. According to the positioning prediction information and the text positioning label, parameters of the task processing model and the sequence classification unit are adjusted to obtain a trained task processing model and a trained sequence classification unit.

12. The method of claim 9, wherein the sample data further comprises a text recognition label. After the category recognition on the sample feature sequence to obtain sample classification information of the sample feature sequence, the method further comprises: The sample feature sequence is input into a text decoding unit to obtain a sample processing text. According to the sample classification information and the sample processing text, sample recognition text of the sample image is determined. According to the positioning prediction information and the text positioning label, parameters of the task processing model are adjusted to obtain a trained task processing model. According to the positioning prediction information and the text positioning label, parameters of the task processing model and the text decoding unit are adjusted to obtain a trained task processing model and a trained text decoding unit.

13. An information processing method based on a task processing model, applied to a task platform, comprising: receiving a model request sent by a terminal device, wherein the model request comprises at least one of a scene identifier of a target scene, scene input data of the target scene and a model specification parameter; based on the model request, determining a target task processing model from a plurality of task processing models, wherein the plurality of task processing models are trained based on any one of the methods of claims 9 to 12.

14. The method of claim 13, wherein the model request comprises the scene identifier of the target scene. The determination of the target task processing model from the plurality of task processing models based on the model request comprises: based on the scene identifier of the target scene, searching for a target task processing model suitable for the target scene from a model library, wherein the model library stores a plurality of task processing models suitable for different visual task scenes.

15. The method of claim 13, wherein the model request comprises the scene input data of the target scene. The determination of the target task processing model from the plurality of task processing models based on the model request comprises: ​ From the plurality of task processing models, determine an initial task processing model adaptive to the target scene; Train the initial task processing model based on scene input data of the target scene, to obtain a target task processing model.

16. The method of claim 13, wherein the model request comprises a model specification parameter; The determining of the target task processing model from the plurality of task processing models based on the model request comprises: Searching for a corresponding target task processing model from a model library based on the model specification parameter, wherein the model library stores task processing models of a plurality of different model specification parameters.

17. The method of any one of claims 13 to 16, wherein after the determining of the target task processing model from the plurality of task processing models based on the model request, the method further comprises: Deploying the target task processing model, and constructing a visual task processing interface based on the target task processing model, to enable the terminal device to schedule the target task processing model to execute a target visual task.

18. A task platform, comprising a request interface and a response unit; The request interface is configured to receive a model request sent by the terminal device, wherein The model request comprises at least one of a scene identifier of a target scene, scene input data of the target scene, and a model specification parameter; The response unit is configured to determine a target task processing model from a plurality of task processing models based on the model request, wherein the plurality of task processing models are trained based on the method of any one of claims 9 to 12.

19. The task platform of claim 18, further comprising a visual task processing interface constructed based on the target task processing model; The visual task processing interface is configured to be used by the terminal device to schedule execution of a target visual task.

20. A computing device, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method of any one of claims 1 to 17.

21. A computer-readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 17.

22. A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 17.

Images