Image processing method, computing device, storage medium and computer program product

Abstract

The embodiment of the invention relates to the technical field of computers, in particular to an image processing method, computing equipment, a storage medium and a computer program product, and the image processing method comprises the following steps: obtaining a to-be-processed image, a prompt text associated with the to-be-processed image, and visual position information of the to-be-processed image; obtaining a target text feature corresponding to the prompt text, a target image feature corresponding to the to-be-processed image, and a target position feature corresponding to the visual position information; and according to the target text feature, the target image feature and the target position feature, obtaining an image processing result corresponding to the to-be-processed image, the image processing result being a result of processing the to-be-processed image according to the text content of the prompt text.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to image processing methods, computing devices, storage media, and computer program products. Background Technology

[0002] Currently, a large number of image documents exist in industrial production and commercial activities. These image documents serve as primary carriers and mediums of information dissemination, containing rich textual and visual content. Deep learning models are typically used to recognize the information contained in these image documents. However, for complex textual and graphical content within these documents, such as e-commerce product details, natural scene images, complex documents from the financial and medical industries, and complex forms from the transportation and distribution industries, current deep learning models struggle to accurately recognize them, resulting in poor image recognition accuracy. Therefore, an effective technical solution is urgently needed to address these issues. Summary of the Invention

[0003] In view of this, embodiments of this specification provide two image processing methods. One or more embodiments of this specification simultaneously relate to two image processing apparatuses, a model training platform, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0005] Acquire the image to be processed, the prompt text associated with the image to be processed, and determine the visual position information for the image to be processed;

[0006] Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0007] Based on the target text features, the target image features, and the target location features, an image processing result corresponding to the image to be processed is obtained, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text.

[0008] According to a second aspect of the embodiments of this specification, an image processing apparatus is provided, comprising:

[0009] The first acquisition module is configured to acquire an image to be processed, a prompt text associated with the image to be processed, and visual position information for the image to be processed.

[0010] The second acquisition module is configured to acquire the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0011] The third acquisition module is configured to acquire the image processing result corresponding to the image to be processed based on the target text features, the target image features, and the target location features, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text.

[0012] According to a third aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0013] Receive the image to be processed and the prompt text associated with the image to be processed sent by the client, and receive the selection instruction for the image to be processed sent by the client, and determine the visual position information of the image to be processed;

[0014] Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0015] Based on the target text features, the target image features, and the target location features, an image processing result corresponding to the image to be processed is obtained, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text;

[0016] The image processing results are sent to the client and displayed through the client's interface.

[0017] According to a fourth aspect of the embodiments of this specification, an image processing apparatus is provided, comprising:

[0018] The receiving module is configured to receive the image to be processed and the prompt text associated with the image to be processed sent by the client, and to receive the selection instruction for the image to be processed sent by the client, and to determine the visual position information of the image to be processed.

[0019] The first acquisition module is configured to acquire the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0020] The second acquisition module is configured to acquire the image processing result corresponding to the image to be processed based on the target text features, the target image features, and the target location features, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text;

[0021] The sending module is configured to send the image processing results to the client and display them through the client's display interface.

[0022] According to a fifth aspect of the embodiments of this specification, a model training platform is provided, including a request interface unit, a model training unit, and a response unit;

[0023] The request interface unit is used to receive a model training request, wherein the model training request includes model information of the machine learning model to be trained.

[0024] The model training unit is used to determine the machine learning model to be trained based on the model information, and to train the machine learning model to be trained to obtain a trained multimodal processing model. The multimodal processing model is used to process the target text features of the prompt text, the target image features of the image to be processed, and the target position features corresponding to the visual position information to obtain the image processing result of the image to be processed. The prompt text is the prompt text associated with the image to be processed, and the visual position information is the visual position information of the image to be processed.

[0025] The response unit is used to output the multimodal processing model.

[0026] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0027] Memory and processor;

[0028] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0029] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0030] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0031] One embodiment of this specification determines the visual position in the image to be processed by identifying the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed. Subsequently, it determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on these multimodal features, such as target text features, target image features, and target position features, it processes the image to be processed according to the text content of the prompt text, obtains the image processing result corresponding to the visual position information, and achieves accurate identification of local regions of the image to be processed. Furthermore, because it processes based on features from multiple modalities, it is suitable for multimodal processing tasks and can achieve accurate identification of complex graphic content, thereby ensuring the accuracy of the image processing results. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of an image processing method provided in one embodiment of this specification;

[0033] Figure 2 This is a flowchart illustrating an image processing method provided in one embodiment of this specification;

[0034] Figure 3 This is a flowchart illustrating the processing procedure of an image processing method provided in one embodiment of this specification.

[0035] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification;

[0036] Figure 5 This is a flowchart of another image processing method provided in one embodiment of this specification;

[0037] Figure 6 This is a schematic diagram of the structure of another image processing apparatus provided in one embodiment of this specification;

[0038] Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0039] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0040] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0042] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0043] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0044] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0045] In practical applications, a large number of image documents exist in industrial production and commercial activities, serving as primary carriers and mediums of information dissemination. Image document recognition and analysis aims to enable machines to understand the textual and visual content within image document scenes, much like humans do. This is not only an important research topic in the field of machine learning but also has immense value for industry applications.

[0046] Deep learning small models have reached a bottleneck in image and document analysis tasks; they cannot independently handle complex image and text parsing and understanding tasks. Typical complex images and texts include e-commerce product detail images and text, natural scene images and text, various complex documents in the financial and medical industries, and complex forms in the transportation and distribution industries. These complex images and texts are the core objects for the informatization of their respective industries. Therefore, an effective technical solution is urgently needed to address these problems.

[0047] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0048] OCR (Optical Character Recognition) is a technology used to convert printed or handwritten text into editable text.

[0049] Document parsing: A technique for detecting, recognizing, and extracting text, graphics, and image elements from images.

[0050] Document comprehension: Understanding key information in a document, such as entity extraction, event extraction, sentiment analysis, and topic classification.

[0051] Multimodal joint learning: a technique that combines data from multiple modalities (such as text, images, audio, etc.) for learning.

[0052] Visual token: The feature vector obtained by passing an image through a visual encoder and then a sampler.

[0053] Language token: The feature vector obtained from the text through a text encoder.

[0054] Large Language Models (LLMs) typically contain billions or even hundreds of billions of parameters. These parameters learn complex patterns and rules of language, thereby enabling them to generate high-quality contextual understanding and text generation capabilities.

[0055] VIT: Vision Transformer, a type of visual encoding neural network that can convert an input image into a series of tokens.

[0056] ResNet: A visual encoding neural network.

[0057] prompt: refers to the initial input or instruction provided to the model, with the aim of generating the required output or response.

[0058] Instruction: This usually refers to the direct instructions or task descriptions provided by the model, informing the model of the specific operations to be performed or the content to be generated. Similar to "prompt", but "instruction" places more emphasis on explicitly instructing the model to perform a specific task.

[0059] Swin Transformers: A model based on the Transformer architecture designed for computer vision tasks. It reduces computational complexity by introducing a window mechanism while maintaining both local and global attention.

[0060] Backbone: The backbone network can be understood as the basic structure in a model, or the network layer used to extract basic features.

[0061] Resampling: A resampling mechanism, commonly used to describe the process of adjusting the sampling rate or resolution of data in digital signal processing, image processing, and data science. Resampling can be applied to one-dimensional signals (such as audio signals), two-dimensional images, or higher-dimensional datasets. It is primarily used to change the resolution of data to better suit different application scenarios or processing needs. In image processing, resampling is used to adjust the resolution of an image, either by downsampling (reducing the image size) or upsampling (enlarging the image size).

[0062] MLP (Multilayer Perceptron) is a basic form of artificial neural network. MLPs are commonly used to solve supervised learning problems, such as classification and regression tasks. It consists of multiple layers, including an input layer, one or more hidden layers, and an output layer.

[0063] This specification provides two image processing methods, and also relates to two image processing apparatuses, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0064] See Figure 1 , Figure 1 The illustration shows an application scenario of an image processing method according to an embodiment of this specification, which includes the following steps.

[0065] Obtain the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed;

[0066] Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0067] Based on the target text features, the target image features, and the target location features, an image processing result corresponding to the image to be processed is obtained, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text.

[0068] Specifically, Figure 1 It includes end-side device 102 and cloud-side device 104.

[0069] In practice, the user can send the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed to the cloud device 104 through the terminal device 102. The cloud device 104 determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on the target text features, target image features, and target position features, the cloud device 104 determines the image processing result corresponding to the image to be processed and sends the image processing result to the terminal device 102, which then displays it to the user through the display interface of the terminal device 102.

[0070] The edge device 102 may include a browser, an app (APP), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a Real-Time Communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0071] Cloud-side device 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that cloud-side device 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Cloud-side device 104 can also be a server for a distributed system, or a server integrated with blockchain. Cloud-side device 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0072] It is worth noting that the image processing method provided in the embodiments of this specification can be executed by the cloud-side device 104 or by the edge-side device 102; in other embodiments, the image processing method provided in the embodiments of this specification can also be executed jointly by the edge-side device 102 and the cloud-side device 104.

[0073] See Figure 2 , Figure 2 A flowchart of an image processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0074] Step 202: Obtain the image to be processed, the prompt text associated with the image to be processed, and the visual location information of the image to be processed.

[0075] Specifically, the image processing methods provided in the embodiments of this specification can be applied to document image recognition tasks. Document image recognition tasks are divided into three main categories: visual tasks, language tasks, and multimodal tasks. Visual tasks refer to tasks related to basic document image processing capabilities, typically serving as pre-processing steps for subsequent language and multimodal tasks. Specific tasks include: document image shading, document image noise reduction, and document image surface correction. Language tasks refer to tasks that rely solely on text content, including: document type identification, text question answering, text translation, and text generation. Multimodal tasks refer to those that rely on both images and semantics. These tasks require alignment within a large multimodal model to produce multimodal output results. Multimodal tasks include: multilingual text detection and recognition, table structure recognition and reconstruction, chart structure parsing, image and text information extraction, and document image content generation.

[0076] In this context, the image to be processed can be understood as the document image that requires image processing. The prompt text associated with the image to be processed can be understood as instructions indicating which type of processing to perform. These instructions can relate to queries about the image content, specific task requirements, etc., and could be phrases like "Help me identify the text within the target area of ​​the image," "Help me remove the shadows within the target area of ​​the image," or "Help me identify the table in the image." The visual position information of the image to be processed can be understood as the location information within the image that needs processing, such as the target area within the image, which can be represented by points, boxes, and masks.

[0077] Based on this, we can determine the image to be processed, the text indicating which processing to perform on the image, and the location information of the target region in the image to be processed.

[0078] In practical applications, it can receive images to be processed uploaded by users through the client, prompt text associated with images to be processed input by users through the client, and visual position information obtained by users clicking or smearing on the images to be processed.

[0079] In specific implementation, obtaining the visual position information of the image to be processed includes:

[0080] Receive a selection command for the image to be processed;

[0081] Based on the selection instruction, the visual position information for the image to be processed is determined.

[0082] Among them, the point selection instructions can include click instructions, box selection instructions, and mask instructions for the area to be processed in the image to be processed.

[0083] Specifically, it can receive a point selection instruction from the user via the client for the area to be processed in the image to be processed, and determine the visual position information of the target area of ​​the image to be processed based on the point selection instruction.

[0084] In practical applications, visual location information can include points, boxes, and masks. A point prompt can be understood as the user selecting one or more key points in an image; the model identifies the location and contextual information of these points to more accurately process and understand image regions. A box prompt can be understood as the user selecting a specific region in the image, providing the model with the area of ​​interest. A mask prompt can be understood as the user providing a fine-grained image mask, defining the shape and boundaries of specific regions in the image. These prompts mainly correspond to location, boundaries, and shape, and the visual location information can be directly encoded into location tokens and input into the multimodal processing model.

[0085] In summary, by determining the visual location information of the image to be processed, it is possible to more precisely identify or detect and segment specific target regions in the image, thereby achieving accurate processing of the image.

[0086] Specifically, a dynamic resolution adjustment mechanism can intelligently select the image resolution suitable for the current task and hardware environment. By introducing an adaptive algorithm, the resolution of the image to be processed can be dynamically adjusted according to the complexity of the input image, the requirements of the image processing task, and computing resources. The specific implementation method is as follows.

[0087] After obtaining the image to be processed, the following steps are also included:

[0088] If it is determined that the image to be processed contains text characters, the resolution of the image to be processed is adjusted according to the image to be processed and the text characters to obtain an adjusted image to be processed, wherein the resolution of the adjusted image to be processed meets a preset resolution condition.

[0089] Specifically, if it is determined that the image to be processed contains text characters, the resolution of the image to be processed can be adjusted according to the image size and the size of the text characters to obtain an adjusted image that meets a preset resolution condition. This preset resolution condition can be understood as the resolution meeting a preset resolution threshold, or it can also be that the resolution meets the computing resource requirements of the current processing device and the image processing task requirements.

[0090] In practical applications, scaling can be combined with text size and image aspect ratio for image cropping. In one embodiment of this specification, edge detection and connected component algorithms can be applied to the image to calculate the size of the text characters. At the original image size, text characters between 16 and 32 pixels are not scaled; those smaller than 16 pixels are enlarged by *(16 / character height); and those larger than 32 pixels are reduced by *(32 / character height). Furthermore, after scaling, the image is cropped according to a 256*256 window, divided into sub-images using the rounded-up function cei l(h / 256, w / 256). For example, a 1000*500 image is divided into 4*2 sub-images. Here, cei l is the round-up function, which rounds a given value up to the nearest integer.

[0091] Furthermore, the resolution of the image to be processed can be dynamically adjusted based on the complexity of the input image, the requirements of the image processing task, and the computing resources, whether the image contains text characters or not.

[0092] In summary, by dynamically adjusting the resolution of the image to be processed, we can make full use of the detailed information of the image, which facilitates the improvement of the flexibility and adaptability of the multimodal processing model and improves computational efficiency while maximizing the use of image details.

[0093] Step 204: Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information.

[0094] Among them, the target text features can be understood as the text feature vector of the prompt text, the target image features can be understood as the image feature vector of the image to be processed, and the target position features corresponding to the visual position information can be understood as the position feature encoding of the visual position information.

[0095] In specific implementation, obtaining the target text features corresponding to the prompt text includes:

[0096] Feature extraction is performed on the prompt text to obtain the target text features corresponding to the prompt text.

[0097] In practical applications, feature extraction of the prompt text can be performed using a text encoder to obtain the target text features (i.e., text token) corresponding to the prompt text. Alternatively, feature extraction of the prompt text can be achieved using a neural network model; however, this specification does not limit the specific implementation of this method.

[0098] In summary, by providing feature-based prompts to the prompt text, the instructions input by the user through natural language can be encoded, facilitating subsequent integration with visual features to achieve multimodal processing tasks.

[0099] In specific implementation, obtaining the target image features corresponding to the image to be processed includes:

[0100] Feature extraction is performed on the image to be processed to obtain the initial image features corresponding to the image to be processed;

[0101] The initial image features are sampled to obtain the target image features corresponding to the image to be processed.

[0102] Specifically, a visual encoder can be used to extract features from the image to be processed to obtain the initial image features corresponding to the image to be processed. Then, the initial image features can be sampled and processed by a neural network to obtain the target image features (i.e., visual tokens) corresponding to the image to be processed.

[0103] Image spatial information can be preserved in both the initial image features and the target image features.

[0104] In practical applications, deep learning architectures such as ResNet, Vision Transformers (ViT), or Swin Transformers are used. These backbones extract image features at multiple levels and scales, possessing strong representational capabilities. After feature extraction by the encoder, the vision part further encodes and processes the visual tokens through several layers of transformers in the resampling MLP. This not only preserves and compresses the spatial information of the image but also aligns it with pre-trained tasks and language features, thereby enabling more accurate cross-modal matching in the visual-language model.

[0105] In summary, since the number of features in the initial image is usually greater than that in the target text, image and text alignment can be achieved through sampling.

[0106] In specific implementation, the visual position information includes the position information of the part to be processed in the image to be processed;

[0107] The step of obtaining the target location features corresponding to the visual location information includes:

[0108] Based on the location information of the part to be processed, the visual location information is subjected to location encoding processing to obtain the target location features corresponding to the visual location information.

[0109] The location information of the part to be processed can be understood as the location information of the target area to be processed, which may include location coordinates, boundary information and shape information.

[0110] Based on this, the visual location information can be encoded into target location features (i.e., location tokens) by using the regional location information of the target area in the image to be processed.

[0111] In summary, encoding visual position information facilitates the accurate determination of the location to be processed during subsequent image processing.

[0112] Step 206: Based on the target text features, the target image features, and the target location features, obtain the image processing result corresponding to the image to be processed, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text.

[0113] Specifically, the image processing result of the image to be processed can be determined based on the target text features, target image features, and target location features. The image processing result can be related to the prompt text. For example, if the prompt text is "Help me identify the table in the image," then the image processing result is the table in the image to be processed. If the prompt text is "Help me remove the shadow in the image," then the image processing result is the image to be processed with the shadow removed.

[0114] In one embodiment of this specification, determining the image processing result corresponding to the image to be processed based on the target text features, the target image features, and the target location features includes:

[0115] The target text features, the target image features, and the target position features are aligned to obtain aligned target text features.

[0116] Based on the aligned target text features and the target image features, the image processing result corresponding to the image to be processed is determined.

[0117] Specifically, the target text features, target image features, and target location features can be aligned to obtain aligned target text features. The aligned target text features can then be decoded to obtain text decoding information. The target image features can be visually decoded to obtain image decoding information. Finally, the image decoding information and text decoding information can be mapped to obtain the image processing result of the image to be processed.

[0118] In practical applications, aligning target text features, target image features, and target location features can be understood as the process of correctly corresponding target text features, target image features, and target location features spatially or semantically. Specifically, spatial alignment can be achieved using methods such as upsampling, downsampling, skip connections, and pyramid structures; temporal alignment can be achieved using methods such as temporal interpolation, recurrent neural networks, and attention mechanisms; and semantic alignment can be achieved using methods such as joint embedding space, cross-modal attention mechanisms, and multi-task learning. The embodiments in this specification do not limit these methods.

[0119] In summary, by aligning image features, text features, and location features, precise alignment of language and vision is achieved, ensuring the processing performance of multimodal tasks.

[0120] In another embodiment of this specification, determining the image processing result corresponding to the image to be processed based on the target text features, the target image features, and the target location features includes:

[0121] The target text features, the target image features, and the target location features are input into a multimodal processing model to obtain the aligned target text features output by the multimodal processing model.

[0122] The target image features are input into a visual decoder to obtain image decoding information of the target image features;

[0123] The aligned target text features are input into the text decoder to obtain text decoding information;

[0124] The image decoding information and the text decoding information are mapped to obtain the image processing result corresponding to the image to be processed.

[0125] Specifically, when mapping image decoding information and text decoding information, the visual decoder and text decoder can be mapped.

[0126] In practical applications, cross-modal mapping is required to achieve the conversion between image information and text information. Specifically, after obtaining the target image features and aligned target text features, an encoder-decoder architecture can be used to map the image decoding information and the text decoding information. The encoder is used to encode the image features, and the decoder is used to generate the text sequence. An attention mechanism can also be introduced into the encoder-decoder architecture. The attention mechanism allows the decoder to focus on different parts of the image when generating text, and to use the target position features when obtaining the aligned target text features. The description is generated step by step based on the target image features, thereby improving the relevance between the generated image processing results and the image content.

[0127] In one embodiment of this specification, inputting the aligned target text features into a text decoder to obtain text decoding information includes:

[0128] The aligned target text features are input into the text decoder, where autoregressive decoding is performed on the aligned target text features to obtain text decoding information.

[0129] The process of mapping the image decoding information and the text decoding information to obtain the image processing result corresponding to the image to be processed includes:

[0130] The image decoding information and the text decoding information are mapped to obtain the text result corresponding to the image to be processed.

[0131] In this context, the text result corresponding to the image to be processed can be understood as the answer based on plain text output. In this case, the multimodal processing model can process language tasks, and the prompt text can be related to the language task.

[0132] In practical applications, during the processing of language tasks, autoregressive decoding can be used to provide answers based on plain text input by leveraging the capabilities of LLM itself.

[0133] In another embodiment of this specification, mapping the image decoding information and the text decoding information to obtain the image processing result corresponding to the image to be processed includes:

[0134] The image decoding information and the text decoding information are mapped to obtain a multimodal result at the visual location information corresponding to the image to be processed, wherein the multimodal result includes at least text information and structural information.

[0135] In the case where the multimodal task in the multimodal processing model is a table recognition task, this structural information can be understood as a table structure.

[0136] In practical applications, convolutional image decoders can be used to efficiently process different image tasks in image processing tasks. For document denoising and shadow removal, generative networks are used with mean squared error loss (MSE Loss) to minimize the difference between the denoised and undenoised images; the input is a noisy image, and the output is the denoised image. In document correction tasks, deformable networks, such as Spatial Transformer Networks (STN), are used; the input is a tilted or deformed document image, and the output is the corrected image. In multimodal tasks, autoregressive decoding is also used. Unlike language tasks, multimodal tasks require the input to include corresponding positional and structural information at the output. For example, if the image to be processed contains a table containing ID, name, and region, then the image processing result output by the multimodal processing model after processing the image to be processed will be a JSON structure of table recognition: [{"ID":1,"Name":"XX","Age":30,"Region":"Region A"},{"ID":2,"Name":"XXX","Age":25,"Region":"Region B"},{"ID":3,"Name":"XXY","Age":35,"Region":"Region C"}].

[0137] Furthermore, the multimodal processing model provided in the embodiments of this specification can process multilingual tasks and can be connected to different large language models, thus enabling stronger language processing capabilities.

[0138] In summary, by combining a visual encoder, a multimodal processing model, and a visual decoder, precise alignment between vision and language is achieved. In vision-related tasks, a convolutional-based visual decoder is used to process people, and generative and deformable networks are used to improve the realism and quality of images. In language-related tasks, an autoregressive decoding mechanism is used, and a large language model can provide answers based on plain text output. In multimodal tasks, visual and positional structural information are combined to perform multimodal output, ensuring efficient processing of complex tasks.

[0139] In practical applications, the training steps of the multimodal processing model include:

[0140] The training steps of the multimodal processing model include:

[0141] Determine the training data corresponding to the image processing task, wherein the image processing task includes language task and multimodal task;

[0142] Based on the training data, the initial multimodal processing model is trained until a multimodal processing model that meets the preset training stopping condition is obtained.

[0143] The initial multimodal processing model can be understood as a machine learning model that has undergone basic pre-training. In practical applications, the initial multimodal processing model can be a large language model. Meeting the preset training stopping conditions can be understood as the model loss value reaching a preset loss value threshold and / or the number of training iterations reaching a preset number of iterations threshold.

[0144] In practical applications, a two-stage, multi-task pre-training process is performed on top of a large language model. Language tasks include document type identification, question answering, text translation, and text generation. Multimodal tasks can be understood as tasks that rely on both images and semantics. These tasks need to be aligned within the large multimodal model before producing multimodal output results.

[0145] In specific implementation, the training data corresponding to the image processing task includes:

[0146] Identify image samples and determine the text labels corresponding to the image samples;

[0147] The step of training the initial multimodal processing model based on the training data until a multimodal processing model that meets the preset training stopping condition is obtained includes:

[0148] The initial multimodal processing model is trained based on the image samples and the text labels until a multimodal processing model that meets the preset training stopping condition is obtained.

[0149] Specifically, in the basic pre-training stage, a machine learning model can be pre-trained using a large amount of data to obtain an initial multimodal processing model. Based on this basic pre-training, the initial multimodal processing model is then trained for an image processing task. For example, in the case of text recognition in an image, the corresponding image samples and their corresponding text labels can be determined. Based on these image samples and text labels, the initial multimodal processing model is trained to adapt to the text recognition task in the image until a multimodal processing model that meets a preset training stopping condition is obtained, enabling the multimodal processing model to handle the text recognition task in the image.

[0150] In practical applications, an LLM can be pre-trained as the language model component of a large-scale document graph model. This model is already pre-trained for basic language understanding. A second stage of pre-training can be used to enhance its ability to handle document-related tasks. This second stage primarily focuses on document-specific tasks, including text detection and recognition, table parsing and recognition, graph parsing, and image-text question answering. Through this second stage of pre-training, the large-scale model not only aligns with both visual and linguistic modalities but also expands the LLM's ability to process image tokens and represent the text-graph space.

[0151] In summary, based on the pre-trained large language model, multimodal two-stage multi-task pre-training can be performed, expanding the model's language and visual processing capabilities.

[0152] Furthermore, the training data includes text data in multiple language types;

[0153] The step of training the initial multimodal processing model based on the training data until a multimodal processing model that meets the preset training stopping condition is obtained includes:

[0154] Based on the text data of the various language types, the initial multimodal processing model is trained until a multimodal processing model that meets the preset training stopping condition is obtained.

[0155] Language type can be understood as language, and multiple language types can include Chinese, English, French, Spanish, Portuguese, Russian, Japanese, Indonesian, Vietnamese, Thai, Hindi, Arabic, etc.

[0156] Specifically, when training a multimodal processing model, a multilingual text task can be added. That is, the initial multimodal processing model is trained based on text data of multiple language types and image data corresponding to specific image processing tasks until a multimodal processing model that meets the preset training stopping conditions is obtained.

[0157] In summary, by incorporating multilingual text tasks, the multimodal processing model is able to handle multiple language tasks.

[0158] One embodiment of this specification determines the visual position in the image to be processed by identifying the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed. Subsequently, it determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on these multimodal features, such as target text features, target image features, and target position features, it processes the image to be processed according to the text content of the prompt text, obtains the image processing result corresponding to the visual position information, and achieves accurate identification of local regions of the image to be processed. Furthermore, because it processes based on features from multiple modalities, it can ensure the accuracy of the image processing results while being applicable to multimodal tasks.

[0159] The following is in conjunction with the appendix Figure 3 Taking the image processing method provided in this specification in the application of image recognition as an example, the image processing method will be further explained. Among other things, Figure 3A flowchart illustrating the processing steps of an image processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0160] Step 302: Determine the image to be processed, determine the prompt text associated with the image to be processed, and determine the visual location information for the image to be processed.

[0161] For example, it can receive the image to be processed uploaded by the user, the prompt text associated with the image to be processed input by the user, "Help me identify the text content in the target area", and the visual position information of the target area obtained by the user selecting the image to be processed. The visual position information is the coordinate information of the target area selected by the user.

[0162] Step 304: Extract features from the prompt text to obtain the target text features corresponding to the prompt text.

[0163] Specifically, the prompt text can be encoded using a text encoder to extract its features and obtain the target text features (i.e., text token) corresponding to the prompt text.

[0164] Step 306: If it is determined that the image to be processed contains text characters, adjust the resolution of the image to be processed according to the image to be processed and the text characters to obtain the adjusted image to be processed.

[0165] Specifically, if it is determined that the image to be processed contains text characters, the text characters in the image to be processed can be scaled according to the size of the image to be processed and the size of the text characters, thereby adjusting the resolution of the image to be processed and obtaining the adjusted image to be processed.

[0166] Step 308: Extract features from the adjusted image to obtain the initial image features corresponding to the adjusted image to obtain the target image features corresponding to the adjusted image to obtain the target image features corresponding to the adjusted image to obtain the target image features.

[0167] Specifically, the adjusted image to be processed can be subjected to multi-level and multi-scale feature extraction through a visual encoder to obtain initial image features. Then, the initial image features can be further sampled through a resampling mechanism (i.e., an MLP network including multiple transformer network layers) to obtain target image features (i.e., visual tokens).

[0168] Step 310: Perform position encoding processing on the visual position information to obtain the target position features corresponding to the visual position information.

[0169] Specifically, the visual location information can be encoded to obtain the target location features (i.e., location tokens) corresponding to the visual location information.

[0170] Step 312: Input the target text features, target image features, and target location features into the multimodal processing model to obtain the aligned target text features output by the multimodal processing model.

[0171] Specifically, target text features, target image features, and target location features can be input into a multimodal processing model. The target text features, target image features, and target location features are then aligned in the multimodal processing model to obtain aligned target text features.

[0172] Step 314: Input the target image features into the visual decoder to obtain the image decoding information output by the visual decoder.

[0173] Step 316: Input the aligned target text features into the text decoder to obtain the text decoding information output by the text decoder.

[0174] Step 318: Map the image decoding information and text decoding information to obtain the image processing result of the image to be processed.

[0175] The image processing result of the image to be processed can be determined according to the specific image processing task. For example, the image processing result can be the image processing result of a visual task, the image processing result of a language task, or the image processing result of a multimodal task.

[0176] Using the previous example, the image processing result of the image to be processed is the text content within the target area that is identified.

[0177] One embodiment of this specification determines the visual position in the image to be processed by identifying the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed. Subsequently, it determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on these multimodal features, such as target text features, target image features, and target position features, it processes the image to be processed according to the text content of the prompt text, obtains the image processing result corresponding to the visual position information, and achieves accurate identification of local regions of the image to be processed. Furthermore, because it processes based on features from multiple modalities, it can ensure the accuracy of the image processing results while being applicable to multimodal tasks.

[0178] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 4A schematic diagram of the structure of an image processing apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes:

[0179] The first acquisition module 402 is configured to acquire an image to be processed, a prompt text associated with the image to be processed, and visual position information for the image to be processed.

[0180] The second acquisition module 404 is configured to acquire the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0181] The third acquisition module 406 is configured to acquire the image processing result corresponding to the image to be processed based on the target text features, the target image features and the target location features, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text.

[0182] In an optional embodiment, the first acquisition module 402 is further configured to:

[0183] If it is determined that the image to be processed contains text characters, the resolution of the image to be processed is adjusted according to the image to be processed and the text characters to obtain an adjusted image to be processed, wherein the resolution of the adjusted image to be processed meets a preset resolution condition.

[0184] In an optional embodiment, the second acquisition module 404 is further configured to:

[0185] Feature extraction is performed on the image to be processed to obtain the initial image features corresponding to the image to be processed;

[0186] The initial image features are sampled to obtain the target image features corresponding to the image to be processed.

[0187] In an optional embodiment, the third acquisition module 406 is further configured to:

[0188] The target text features, the target image features, and the target location features are aligned to obtain aligned target text features.

[0189] Based on the aligned target text features and the target image features, the image processing result corresponding to the image to be processed is determined.

[0190] In an optional embodiment, the third acquisition module 406 is further configured to:

[0191] The target text features, the target image features, and the target location features are input into a multimodal processing model to obtain the aligned target text features output by the multimodal processing model.

[0192] The target image features are input into a visual decoder to obtain image decoding information of the target image features;

[0193] The aligned target text features are input into the text decoder to obtain text decoding information;

[0194] The image decoding information and the text decoding information are mapped to obtain the image processing result corresponding to the image to be processed.

[0195] In an optional embodiment, the third acquisition module 406 is further configured to:

[0196] The aligned target text features are input into the text decoder, where autoregressive decoding is performed on the aligned target text features to obtain text decoding information.

[0197] The process of mapping the image decoding information and the text decoding information to obtain the image processing result corresponding to the image to be processed includes:

[0198] The image decoding information and the text decoding information are mapped to obtain the text result corresponding to the image to be processed.

[0199] In an optional embodiment, the third acquisition module 406 is further configured to:

[0200] The image decoding information and the text decoding information are mapped to obtain a multimodal result at the visual location information corresponding to the image to be processed, wherein the multimodal result includes at least text information and structural information.

[0201] In an optional embodiment, the device further includes a training module configured to:

[0202] Determine the training data corresponding to the image processing task, wherein the image processing task includes language task and multimodal task;

[0203] Based on the training data, the initial multimodal processing model is trained until a multimodal processing model that meets the preset training stopping condition is obtained.

[0204] In an optional embodiment, the training module is further configured to:

[0205] Identify image samples and determine the text labels corresponding to the image samples;

[0206] The initial multimodal processing model is trained based on the image samples and the text labels until a multimodal processing model that meets the preset training stopping condition is obtained.

[0207] In one optional embodiment, the training data includes text data in multiple language types;

[0208] The training module is further configured as follows:

[0209] Based on the text data of the various language types, the initial multimodal processing model is trained until a multimodal processing model that meets the preset training stopping condition is obtained.

[0210] In an optional embodiment, the first acquisition module 402 is further configured to:

[0211] Receive a selection command for the image to be processed;

[0212] Based on the selection instruction, the visual position information of the image to be processed is determined.

[0213] In an optional embodiment, the visual position information includes the position information of the portion to be processed in the image to be processed;

[0214] The second acquisition module 404 is further configured as follows:

[0215] Based on the location information of the part to be processed, the visual location information is subjected to location encoding processing to obtain the target location features corresponding to the visual location information.

[0216] In an optional embodiment, the second acquisition module 404 is further configured to:

[0217] Feature extraction is performed on the prompt text to obtain the target text features corresponding to the prompt text.

[0218] One embodiment of this specification determines the visual position in the image to be processed by identifying the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed. Subsequently, it determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on these multimodal features, such as target text features, target image features, and target position features, it processes the image to be processed according to the text content of the prompt text, obtains the image processing result corresponding to the visual position information, and achieves accurate identification of local regions of the image to be processed. Furthermore, because it processes based on features from multiple modalities, it can ensure the accuracy of the image processing results while being applicable to multimodal tasks.

[0219] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0220] See Figure 5 , Figure 5 A flowchart of another image processing method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0221] Step 502: Receive the image to be processed and the prompt text associated with the image to be processed sent by the client, and receive the selection instruction for the image to be processed sent by the client, and determine the visual position information of the image to be processed;

[0222] Step 504: Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0223] Step 506: Based on the target text features, the target image features, and the target location features, obtain the image processing result corresponding to the image to be processed, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text;

[0224] Step 508: Send the image processing result to the client and display it through the client's display interface.

[0225] One embodiment of this specification determines the visual position in the image to be processed by identifying the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed. Subsequently, it determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on these multimodal features, such as target text features, target image features, and target position features, it processes the image to be processed according to the text content of the prompt text, obtains the image processing result corresponding to the visual position information, and achieves accurate identification of local regions of the image to be processed. Furthermore, because it processes based on features from multiple modalities, it can ensure the accuracy of the image processing results while being applicable to multimodal tasks.

[0226] The above is an illustrative scheme of an image processing method according to this embodiment. It should be noted that the technical solution of this image processing method belongs to the same concept as the technical solution of the image processing method described above. For details not described in detail in the technical solution of the image processing method, please refer to the description of the technical solution of the image processing method described above.

[0227] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 6 A schematic diagram of another image processing apparatus provided in one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0228] The receiving module 602 is configured to receive the image to be processed and the prompt text associated with the image to be processed sent by the client, and to receive the selection instruction for the image to be processed sent by the client, and to determine the visual position information of the image to be processed.

[0229] The first acquisition module 604 is configured to acquire the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information;

[0230] The second acquisition module 606 is configured to acquire the image processing result corresponding to the image to be processed based on the target text features, the target image features and the target location features, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text;

[0231] The sending module 608 is configured to send the image processing result to the client and display it through the client's display interface.

[0232] One embodiment of this specification determines the visual position in the image to be processed by identifying the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed. Subsequently, it determines the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information. Based on these multimodal features, such as target text features, target image features, and target position features, it processes the image to be processed according to the text content of the prompt text, obtains the image processing result corresponding to the visual position information, and achieves accurate identification of local regions of the image to be processed. Furthermore, because it processes based on features from multiple modalities, it can ensure the accuracy of the image processing results while being applicable to multimodal tasks.

[0233] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0234] Corresponding to the above method embodiments, this specification also provides a model training platform embodiment, which includes a request interface unit, a model training unit, and a response unit;

[0235] The request interface unit is used to receive a model training request, wherein the model training request includes model information of the machine learning model to be trained.

[0236] The model training unit is used to determine the machine learning model to be trained based on the model information, and to train the machine learning model to be trained to obtain a trained multimodal processing model. The multimodal processing model is used to process the target text features of the prompt text, the target image features of the image to be processed, and the target position features corresponding to the visual position information to obtain the image processing result of the image to be processed. The prompt text is the prompt text associated with the image to be processed, and the visual position information is the visual position information of the image to be processed.

[0237] The response unit is used to output the multimodal processing model.

[0238] In an optional embodiment, the model training platform further includes a data receiving unit.

[0239] The data receiving unit is used to receive training data input by the user and send the training data to the model training unit;

[0240] The model training unit is further configured to train the machine learning model to be trained based on the training data, thereby obtaining a trained multimodal processing model.

[0241] In an optional embodiment, the model training platform further includes a model library, wherein the model library stores multiple machine learning models;

[0242] The model training unit is further configured to determine the machine learning model to be trained from the model library based on the model information.

[0243] In summary, by using a model training platform to train the multimodal processing model, the processing performance of the multimodal processing model is guaranteed, enabling it to handle multimodal tasks in subsequent applications.

[0244] Figure 7 A structural block diagram of a computing device 700 according to one embodiment of this specification is shown. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0245] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 740 may include one or more of any type of wired or wireless network interface (e.g., network interface controller), such as IEEE 802.11 Wireless Local Area Network (WLAN) interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0246] In one embodiment of this application, the aforementioned components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0247] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0248] The processor 720 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described image processing method.

[0249] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the image processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the image processing method embodiments.

[0250] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the image processing method described above.

[0251] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the image processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the image processing method embodiments.

[0252] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described image processing method.

[0253] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the image processing method described above.

[0254] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0255] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0256] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0257] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0258] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Obtain the image to be processed, the prompt text associated with the image to be processed, and the visual position information of the image to be processed; Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information; Based on the target text features, the target image features, and the target location features, an image processing result corresponding to the image to be processed is obtained, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text.

2. The image processing method according to claim 1, further comprising, after acquiring the image to be processed: If it is determined that the image to be processed contains text characters, the resolution of the image to be processed is adjusted according to the image to be processed and the text characters to obtain an adjusted image to be processed, wherein the resolution of the adjusted image to be processed meets a preset resolution condition.

3. The image processing method according to claim 1 or 2, wherein obtaining the target image features corresponding to the image to be processed includes: Feature extraction is performed on the image to be processed to obtain the initial image features corresponding to the image to be processed; The initial image features are sampled to obtain the target image features corresponding to the image to be processed.

4. The image processing method according to claim 1, wherein obtaining the image processing result corresponding to the image to be processed based on the target text features, the target image features, and the target location features includes: The target text features, the target image features, and the target location features are aligned to obtain aligned target text features. Based on the aligned target text features and the target image features, the image processing result corresponding to the image to be processed is determined.

5. The image processing method according to claim 1, wherein obtaining the image processing result corresponding to the image to be processed based on the target text features, the target image features, and the target location features includes: The target text features, the target image features, and the target location features are input into a multimodal processing model to obtain the aligned target text features output by the multimodal processing model. The target image features are input into a visual decoder to obtain image decoding information of the target image features; The aligned target text features are input into the text decoder to obtain text decoding information; The image decoding information and the text decoding information are mapped to obtain the image processing result corresponding to the image to be processed.

6. The image processing method according to claim 5, wherein inputting the aligned target text features into a text decoder to obtain text decoding information includes: The aligned target text features are input into the text decoder, where autoregressive decoding is performed on the aligned target text features to obtain text decoding information. The process of mapping the image decoding information and the text decoding information to obtain the image processing result corresponding to the image to be processed includes: The image decoding information and the text decoding information are mapped to obtain the text result corresponding to the image to be processed.

7. The image processing method according to claim 5, mapping the image decoding information and the text decoding information to obtain the image processing result corresponding to the image to be processed, includes: The image decoding information and the text decoding information are mapped to obtain a multimodal result at the visual location information corresponding to the image to be processed, wherein the multimodal result includes at least text information and structural information.

8. The image processing method according to any one of claims 5-7, wherein the training step of the multimodal processing model includes: Determine the training data corresponding to the image processing task, wherein the image processing task includes language task and multimodal task; Based on the training data, the initial multimodal processing model is trained until a multimodal processing model that meets the preset training stopping condition is obtained.

9. The image processing method according to claim 8, wherein determining the training data corresponding to the image processing task includes: Identify image samples and determine the text labels corresponding to the image samples; The step of training the initial multimodal processing model based on the training data until a multimodal processing model that meets the preset training stopping condition is obtained includes: The initial multimodal processing model is trained based on the image samples and the text labels until a multimodal processing model that meets the preset training stopping condition is obtained.

10. The image processing method according to claim 8, wherein the training data includes text data in multiple language types; The step of training the initial multimodal processing model based on the training data until a multimodal processing model that meets the preset training stopping condition is obtained includes: Based on the text data of the various language types, the initial multimodal processing model is trained until a multimodal processing model that meets the preset training stopping condition is obtained.

11. The image processing method according to claim 1, wherein obtaining the visual position information of the image to be processed includes: Receive a selection command for the image to be processed; Based on the selection instruction, the visual position information of the image to be processed is determined.

12. The image processing method according to claim 12, wherein the visual position information includes the position information of the portion to be processed in the image to be processed; The step of obtaining the target location features corresponding to the visual location information includes: Based on the location information of the part to be processed, the visual location information is subjected to location encoding processing to obtain the target location features corresponding to the visual location information.

13. An image processing method, comprising: Receive the image to be processed and the prompt text associated with the image to be processed sent by the client, and receive the selection instruction for the image to be processed sent by the client, and determine the visual position information of the image to be processed; Obtain the target text features corresponding to the prompt text, the target image features corresponding to the image to be processed, and the target position features corresponding to the visual position information; Based on the target text features, the target image features, and the target location features, an image processing result corresponding to the image to be processed is obtained, wherein the image processing result is the result of processing the image to be processed based on the text content of the prompt text; The image processing results are sent to the client and displayed through the client's interface.

14. A model training platform, comprising a request interface unit, a model training unit, and a response unit; The request interface unit is used to receive a model training request, wherein the model training request includes model information of the machine learning model to be trained. The model training unit is used to determine the machine learning model to be trained based on the model information, and to train the machine learning model to be trained to obtain a trained multimodal processing model. The multimodal processing model is used to process the target text features of the prompt text, the target image features of the image to be processed, and the target position features corresponding to the visual position information to obtain the image processing result of the image to be processed. The prompt text is the prompt text associated with the image to be processed, and the visual position information is the visual position information of the image to be processed. The response unit is used to output the multimodal processing model.

15. The model training platform according to claim 14 further includes a data receiving unit. The data receiving unit is used to receive training data input by the user and send the training data to the model training unit; The model training unit is further configured to train the machine learning model to be trained based on the training data, thereby obtaining a trained multimodal processing model.

16. The model training platform according to claim 14 further includes a model library, wherein, The model library stores multiple machine learning models; The model training unit is further configured to determine the machine learning model to be trained from the model library based on the model information.

17. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

18. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images