Image processing method, image processing model training method and image segmentation method

Abstract

The embodiment of the invention provides an image processing method, an image processing model training method and an image segmentation method, and the image processing method comprises the steps: obtaining a to-be-processed image, a reference image and prompt information, the reference image being used for providing an object reference feature of a first class object, and the prompt information being used for providing an object prompt feature of a second class object; according to the to-be-processed image, the reference image and the prompt information, a task processing instruction is generated, the task processing instruction is used for describing an object processing mode of a first class object in the to-be-processed image, and the object processing mode is obtained based on the object reference feature and the object prompt feature; and determining an object processing result of the first class object in the to-be-processed image based on the task processing instruction by using the image processing model. Through adding other types of prompt information, richer and more comprehensive guidance information is provided, the limitation of the reference image is avoided, and the processing accuracy of the image processing model is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and particularly to an image processing method, an image processing model training method, and an image segmentation method. Background Art

[0002] With the development of computer technology, image processing increasingly relies on computers, which has the advantages of high efficiency and high automation. Image processing is a process of identifying and extracting images, and has been widely applied in various fields of people's daily lives.

[0003] Currently, an image processing model is usually used to perform image processing tasks. For example, an image segmentation model is used to segment an image to extract the region of a target object. For complex image processing tasks, reference information can be used to guide the image processing model to process the image. However, due to the limitation of the reference information, the processing accuracy of the image processing model is relatively low. Therefore, there is an urgent need for a more accurate image processing solution. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide an image processing method. One or more embodiments of this specification simultaneously relate to an image processing model training method, an image segmentation method, an image processing apparatus, an image processing model training apparatus, an image segmentation apparatus, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to the first aspect of the embodiments of this specification, an image processing method is provided, including:

[0006] Obtain an image to be processed, a reference image, and prompt information, where the reference image is used to provide object reference features of a first type of object, and the prompt information is used to provide object prompt features of a second type of object;

[0007] Generate a task processing instruction according to the image to be processed, the reference image, and the prompt information, where the task processing instruction is used to describe an object processing manner of the first type of object in the image to be processed, and the object processing manner is obtained based on the object reference features and the object prompt features;

[0008] Use an image processing model to determine an object processing result of the first type of object in the image to be processed based on the task processing instruction.

[0009] According to the second aspect of the embodiments of this specification, an image processing model training method is provided, which is applied to a cloud-side device and includes:

[0010] Obtain a sample set, where the sample set includes processed sample images, reference sample images, and prompt sample information. The reference sample images are used to provide sample reference features of a first category of objects, the prompt sample information is used to provide sample prompt features of a second category of objects, and the processed sample images carry processed result labels;

[0011] Generate a sample task instruction according to the processed sample image, the reference sample image, and the prompt sample information, where the sample task instruction is used to describe the object processing method of the first category of objects in the processed sample image, and the object processing method is obtained based on the sample reference features and the sample prompt features;

[0012] Use an image processing model to determine the object prediction result of the first category of objects in the processed sample image based on the sample task instruction;

[0013] Adjust the model parameters of the image processing model according to the processed result label and the object prediction result to obtain a trained image processing model.

[0014] According to the third aspect of the embodiments of this specification, an image segmentation method is provided, including:

[0015] Obtain an image to be segmented, a segmentation reference image, and segmentation prompt information, where the segmentation reference image is used to provide reference edge features of a first category of objects, and the segmentation prompt information is used to provide prompt edge features of a second category of objects;

[0016] Generate an image segmentation instruction according to the image to be segmented, the segmentation reference image, and the segmentation prompt information, where the image segmentation instruction is used to describe the object segmentation method of the image to be segmented, and the object segmentation method is obtained based on the reference edge features and the prompt edge features;

[0017] Use an image segmentation model to determine the target edge information of the first category of objects in the image to be segmented based on the image segmentation instruction.

[0018] According to the fourth aspect of the embodiments of this specification, an image processing apparatus is provided, including:

[0019] A first acquisition module configured to acquire an image to be processed, a reference image, and prompt information, where the reference image is used to provide object reference features of a first category of objects, and the prompt information is used to provide object prompt features of a second category of objects;

[0020] A first generation module, configured to generate a task processing instruction according to the image to be processed, the reference image, and the prompt information, where the task processing instruction is used to describe an object processing manner of a first category of objects in the image to be processed, and the object processing manner is obtained based on the object reference feature and the object prompt feature;

[0021] A first determination module, configured to use an image processing model to determine an object processing result of the first category of objects in the image to be processed based on the task processing instruction.

[0022] According to a fifth aspect of the embodiments of the present specification, there is provided an image processing model training device, which is applied to a cloud-side device and includes:

[0023] A second acquisition module, configured to acquire a sample set, where the sample set includes a processed sample image, a reference sample image, and prompt sample information, the reference sample image is used to provide a sample reference feature of a first category of objects, the prompt sample information is used to provide a sample prompt feature of a second category of objects, and the processed sample image carries a processing result label;

[0024] A second generation module, configured to generate a sample task instruction according to the processed sample image, the reference sample image, and the prompt sample information, where the sample task instruction is used to describe an object processing manner of a first category of objects in the processed sample image, and the object processing manner is obtained based on the sample reference feature and the sample prompt feature;

[0025] A second determination module, configured to use an image processing model to determine an object prediction result of the first category of objects in the processed sample image based on the sample task instruction;

[0026] A training module, configured to adjust model parameters of the image processing model according to the processing result label and the object prediction result to obtain a trained image processing model.

[0027] According to a sixth aspect of the embodiments of the present specification, there is provided an image segmentation device, including:

[0028] A third acquisition module, configured to acquire an image to be segmented, a segmentation reference image, and segmentation prompt information, where the segmentation reference image is used to provide a reference edge feature of a first category of objects, and the segmentation prompt information is used to provide a prompt edge feature of a second category of objects;

[0029] A third generation module, configured to generate an image segmentation instruction according to the image to be segmented, the segmentation reference image, and the segmentation hint information, where the image segmentation instruction is used to describe an object segmentation method of the image to be segmented, and the object segmentation method is obtained based on the reference edge feature and the hint edge feature;

[0030] A third determination module, configured to use an image segmentation model to determine target edge information of the first category of objects in the image to be segmented based on the image segmentation instruction.

[0031] According to a seventh aspect of the embodiments of the present specification, there is provided a computing device, including:

[0032] A memory and a processor;

[0033] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method provided in the above first aspect or second aspect or third aspect are implemented.

[0034] According to an eighth aspect of the embodiments of the present specification, there is provided a computer-readable storage medium, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the method provided in the above first aspect or second aspect or third aspect are implemented.

[0035] According to a ninth aspect of the embodiments of the present specification, there is provided a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the method provided in the above first aspect or second aspect or third aspect are implemented.

[0036] The image processing method provided by an embodiment of this specification obtains an image to be processed, a reference image, and prompt information. Among them, the reference image is used to provide object reference features of a first type of object, and the prompt information is used to provide object prompt features of a second type of object; according to the image to be processed, the reference image, and the prompt information, a task processing instruction is generated. Among them, the task processing instruction is used to describe the object processing method of the first type of object in the image to be processed, and the object processing method is obtained based on the object reference features and the object prompt features; using an image processing model, based on the task processing instruction, determine the object processing result of the first type of object in the image to be processed. By adding the prompt information of the second type of object, combining the reference image of the first type of object and the prompt information of the second type of object, generating a task processing instruction, and indicating the object processing method of the first type of object through this task processing instruction, thereby guiding the image processing model to process the object of the first type of object in the image to be processed. By adding the prompt information of other categories, more abundant and comprehensive guiding information is provided, avoiding being restricted by the reference image, and improving the processing accuracy of the image processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is an architecture diagram of an image processing system provided by an embodiment of this specification;

[0038] Figure 2 is an architecture diagram of another image processing system provided by an embodiment of this specification;

[0039] Figure 3 is a flowchart of an image processing method provided by an embodiment of this specification;

[0040] Figure 4 is a schematic diagram of an image to be processed, a reference image, and prompt information provided by an embodiment of this specification;

[0041] Figure 5 is a flowchart of an image processing model training method provided by an embodiment of this specification;

[0042] Figure 6 is a flowchart of the processing process of an image segmentation method provided by an embodiment of this specification;

[0043] Figure 7a is an overall architecture diagram of an image segmentation network provided by an embodiment of this specification;

[0044] Figure 7b is a schematic diagram of the processing process of an image segmentation provided by an embodiment of this specification;

[0045] Figure 8It is a schematic structural diagram of an image processing device provided by an embodiment of this specification;

[0046] Figure 9 It is a schematic structural diagram of an image processing model training device provided by an embodiment of this specification;

[0047] Figure 10 It is a schematic structural diagram of an image segmentation device provided by an embodiment of this specification;

[0048] Figure 11 It is a block diagram of the structure of a computing device provided by an embodiment of this specification. Detailed implementation manners

[0049] In the following description, many specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of this specification. Therefore, this specification is not limited by the specific implementations disclosed below.

[0050] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more of the associated listed items.

[0051] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0052] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for the user to select to authorize or reject.

[0053] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, usually including hundreds of millions, tens of billions, hundreds of billions, trillions or even more than one quadrillion model parameters. A large model can also be referred to as a Foundation Model. Through pre-training of the large model with a large amount of unlabeled corpus, a pre-trained model with more than one billion parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLMs), multi-modal pre-training models, etc.

[0054] When a large model is actually applied, only a small number of samples are needed to fine-tune the pre-trained model for application to different tasks. Large models can be widely applied in the fields of natural language processing (NLP), computer vision, etc. Specifically, they can be applied to tasks in the field of computer vision such as visual question answering (VQA), image captioning (IC), image generation, etc., and tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, machine translation, etc. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0055] First, the noun terms involved in one or more embodiments of this specification are explained.

[0056] Semantic segmentation: It is an important task in the field of computer vision, aiming to label each pixel in an image with the corresponding semantic category. However, in practical applications, a challenging problem is often faced, namely few-shot semantic segmentation.

[0057] Few-shot semantic segmentation: It refers to performing the semantic segmentation task when the training data set has a very limited number of samples.

[0058] Multi-modal: It refers to learning by combining data of at least two modalities. In the embodiments of this specification, it is learning by combining data of language and vision modalities.

[0059] Image segmentation: For the input image, pixel-level classification is performed.

[0060] Mask: Also known as a mask, an image mask is generally used to block (all or part of) the image to be processed to control the area or process of image processing.

[0061] In the field of computer vision, basic tasks include image segmentation, which has a wide range of applications. Deep learning algorithms trained on large-scale datasets have brought significant progress to this field. However, training in this way requires extremely large resources. Image segmentation models trained based on a small amount of data often fail to meet the performance requirements due to the lack of data. The progress of large language models (LLMs) can enable image segmentation. Specifically, large language models pre-trained on large-scale corpora have accumulated a large amount of prior knowledge, which can effectively supplement the insufficient information in the reference images and provide more effective guidance for training.

[0062] In few-shot semantic segmentation, the model should develop category-specific segmentation capabilities based on only a small amount of annotated data, namely reference images. To achieve this goal, a support feature guidance framework is usually adopted. In this framework, relevant features of the target category are extracted from the annotated reference images and used as guidance information to segment the image to be segmented. However, the very limited reference images only contain a small part of incomplete and biased feature information. Therefore, the framework relying on these feature information for image segmentation is essentially limited by the reference images and cannot achieve a high enough accuracy.

[0063] To solve the above problems, the embodiments of this specification propose a framework for introducing large language models to solve the few-shot semantic segmentation problem. By adding other categories except the category to be processed as prompt information, combining the reference images and the prompt information, and developing category-specific segmentation capabilities, this framework can utilize richer and more comprehensive information to solve the visual segmentation problem.

[0064] Specifically, obtain the image to be processed, the reference image, and the prompt information. Among them, the reference image is used to provide the object reference features of the first category of objects, and the prompt information is used to provide the object prompt features of the second category of objects; generate a task processing instruction according to the image to be processed, the reference image, and the prompt information. The task processing instruction is used to describe the object processing method of the first category of objects in the image to be processed, and the object processing method is obtained based on the object reference features and the object prompt features; use an image processing model to determine the object processing result of the first category of objects in the image to be processed based on the task processing instruction. By adding the prompt information of the second category of objects, combining the reference image of the first category of objects and the prompt information of the second category of objects, generating a task processing instruction, and indicating the object processing method of the first category of objects through this task processing instruction, so as to guide the image processing model to process the objects of the first category in the image to be processed. By adding the prompt information of other categories, richer and more comprehensive guidance information is provided, avoiding being restricted by the reference images and improving the processing accuracy of the image processing model.

[0065] In this specification, an image processing method is provided. One or more embodiments of this specification are also related to an image processing model training method, an image segmentation method, an image processing apparatus, an image processing model training apparatus, an image segmentation apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail one by one in the following embodiments.

[0066] See Figure 1 , Figure 1 shows an architecture diagram of an image processing system provided according to an embodiment of this specification. The image processing system may include a client 100 and a server 200;

[0067] The client 100 is configured to send an image to be processed to the server 200;

[0068] The server 200 is configured to obtain an image to be processed, a reference image, and prompt information. The reference image is used to provide object reference features of a first category of objects, and the prompt information is used to provide object prompt features of a second category of objects; generate a task processing instruction based on the image to be processed, the reference image, and the prompt information, where the task processing instruction is used to describe the object processing method of the first category of objects in the image to be processed, and the object processing method is obtained based on the object reference features and the object prompt features; use an image processing model to determine an object processing result of the first category of objects in the image to be processed based on the task processing instruction; and send the object processing result to the client 100;

[0069] The client 100 is further configured to receive the object processing result sent by the server 200.

[0070] Applying the solution of the embodiment of this specification, by adding the prompt information of the second category of objects, combining the reference image of the first category of objects and the prompt information of the second category of objects, generating a task processing instruction, and indicating the object processing method of the first category of objects through this task processing instruction, the image processing model is guided to process the objects of the first category in the image to be processed. By adding the prompt information of other categories, richer and more comprehensive guiding information is provided, avoiding being restricted by the reference image, and improving the processing accuracy of the image processing model.

[0071] See Figure 2 , Figure 2The figure shows an architecture diagram of another image processing system provided by an embodiment of this specification. The image processing system may include multiple clients 100 and a server 200. Among them, the client 100 may include an end-side device, and the server 200 may include a cloud-side device. Communication connections can be established between multiple clients 100 through the server 200. In an image processing scenario, the server 200 is used to provide image processing services between multiple clients 100. Multiple clients 100 can be used as senders or receivers respectively, and communicate through the server 200.

[0072] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In an image processing scenario, it can be that the user publishes a data stream to the server 200 through the client 100, and the server 200 generates an object processing result based on the data stream and pushes the object processing result to other clients that have established communication.

[0073] Among them, a connection is established between the client 100 and the server 200 through a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. The data transmitted by the client 100 may need to be processed such as encoded, transcoded, compressed, etc. before being published to the server 200.

[0074] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5) application, or a light application (also known as a mini-program, a lightweight application program), or a cloud application, etc. The client 100 can be developed based on a software development kit (SDK) provided by the server 200 for the corresponding service, such as developed based on a real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device or certain APPs in the device to run, etc. The electronic device can, for example, have a display screen and support information browsing, etc., such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can usually be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0075] The server 200 may include servers that provide various services. For example, a server that provides communication services for multiple clients, or a server for background training that supports models used on the client, or a server that processes data sent by the client, etc. It should be noted that the server 200 may be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server may also be a server of a distributed system, or a server combined with a blockchain. The server may also be a cloud server of basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN, Content Delivery Network), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0076] It should be noted that the image processing method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have a similar function as the server, so as to execute the image processing method provided in the embodiments of this specification. In other embodiments, the image processing method provided in the embodiments of this specification may also be jointly executed by the client and the server.

[0077] See Figure 3 , Figure 3 which shows a flowchart of an image processing method provided in an embodiment of this specification, specifically including the following steps 302-306:

[0078] Step 302: Obtain an image to be processed, a reference image, and prompt information, where the reference image is used to provide object reference features of a first category of objects, and the prompt information is used to provide object prompt features of a second category of objects.

[0079] In one or more embodiments of this specification, when performing image processing, an image to be processed, a reference image, and prompt information may be obtained, so as to provide guiding information of a first category of objects based on the reference image, provide guiding information of a second category of objects based on the prompt information, and guide the processing of the first category of objects in the image to be processed to obtain an object processing result.

[0080] Among them, the first category of objects is the category that the task to be processed needs to process, and the second category of objects is other categories except the first category of objects. For example, if the first category of objects is "horse", the second category of objects is "sheep", "cow",... etc.

[0081] In addition, the image to be processed, the reference image, and the prompt information can each be at least one. That is to say, it is possible to process one image to be processed at a time, or to batch process two or more images to be processed. Additionally, when processing the image to be processed, there can be at least one reference image providing the object reference features of the first category of objects, and there can also be at least one prompt information providing the object prompt features of the second category of objects. If there are two or more pieces of prompt information providing the object prompt features of the second category of objects, the second category of objects corresponding to these two or more pieces of prompt information can be the same or different.

[0082] Specifically, image processing can be applied to different processing scenarios, such as image segmentation, object category recognition, image classification, and other scenarios. The image to be processed is the image that needs to be processed, such as an image that needs to be segmented; the reference image is an image with object annotation data of the first category of objects, which can provide the object reference features of the first category of objects for image processing; the prompt information includes the image of the second category of objects and the corresponding processing results, providing the object prompt features of the second category of objects for image processing.

[0083] In actual implementation, according to the task to be processed, an image with object annotation data of the first category of objects can be obtained as the reference image, and the annotation data refers to the processing result data of the first category of objects in the image. Additionally, an image including the second category of objects and the corresponding processing result of this image can also be obtained as the prompt information. Among them, the task to be processed can indicate the processing task and the object category required, such as the image segmentation task of the first category of objects, the category recognition task of the first category of objects, the classification task of the image of the first category of objects, and so on.

[0084] For example, in the case of the image segmentation scenario, if the task to be processed is to segment the "horse" in the image to be processed, an image with the edge coordinates of the "horse" can be obtained as the reference image at this time, and an image with a "sheep" and the contour image of the "sheep" in this image can be obtained, and the image of the "sheep" and the contour image of the "sheep" are used as the prompt information. Subsequently, based on the reference image and the prompt information, the image processing model is guided to determine the edge coordinates of the "horse" in the image to be processed. Taking the object category recognition scenario as an example, if the task to be processed is to identify whether the object in the image to be processed is a "horse", an image that has identified a "horse" can be obtained as the reference image at this time, and an image that has identified a "sheep" and the local feature image of the "sheep" can be obtained, and the image of the "sheep" and the local feature image of the "sheep" are used as the prompt information. Subsequently, based on the reference image and the prompt information, the image processing model is guided to identify whether the object in the image to be processed is a horse.

[0085] In the embodiments of this specification, the reference image and the hint information are used to provide different types of guidance information to guide the subsequent image processing model to process the image to be processed. Since a training set with processing results needs to be obtained when training the image processing model, both the reference image and the hint information can be selected from the training set; alternatively, the reference image and the hint information can also be obtained from other platforms, as long as they carry the processing results.

[0086] Step 304: Generate a task processing instruction according to the image to be processed, the reference image, and the hint information, where the task processing instruction is used to describe the object processing method of the first type of object in the image to be processed, and the object processing method is obtained based on the object reference feature and the object hint feature.

[0087] In the embodiments of this specification, after obtaining the image to be processed, the reference image, and the hint information, further, a task processing instruction can be generated by combining the image to be processed, the reference image, and the hint information. The object processing method of the first type of object in the image to be processed is indicated by this task processing instruction, which facilitates the subsequent image processing model to process according to the object processing method indicated by the task processing instruction and obtain the object processing result. Since the object processing method integrates the object reference feature and the object hint feature, by adding hint information of other types, more rich and comprehensive guidance information is provided, avoiding being restricted by the reference image, and improving the processing accuracy of the image processing model.

[0088] In an optional implementation manner of this embodiment, taking image segmentation as an example, the image to be processed is an image to be object-segmented, and the hint information includes a segmentation hint image and an object segmentation image corresponding to the segmentation hint image;

[0089] Generating a task processing instruction according to the image to be processed, the reference image, and the hint information includes:

[0090] Generate an image segmentation instruction according to the image to be processed, the reference image, the segmentation hint image, and the object segmentation image.

[0091] It should be noted that the object segmentation image corresponding to the segmentation hint image refers to the contour image of the second type of object in the segmentation hint image. Combining the segmentation hint image and the contour image can provide more rich guidance information for image segmentation outside the reference image and improve the accuracy of image segmentation.

[0092] Exemplarily, Figure 4 shows a schematic diagram of an image to be processed, a reference image, and hint information provided according to an embodiment of this specification, as Figure 4As shown, taking the image segmentation scenario as an example, the reference image is an image including a "horse", the image to be processed is also an image including a "horse", and the task to be processed is to segment the "horse" from the image to be processed. The prompt information includes a segmentation prompt image and a contour image. The segmentation prompt image is an image of a "sheep", and the contour image is a contour image of a "sheep".

[0093] In an optional implementation manner of this embodiment, the prompt information includes a prompt image and a processed result image of the prompt image; according to the image to be processed, the reference image, and the prompt information, a task processing instruction is generated, including:

[0094] Using an image encoder, the first encoded information of the image to be processed, the second encoded information of the reference image, and the third encoded information corresponding to the prompt image and the processed result image are obtained;

[0095] According to the first encoded information, the second encoded information, and the third encoded information, a task processing instruction is generated.

[0096] Among them, the image encoder is an image feature extractor that can encode the visual features in the image to obtain the corresponding encoded information. In addition, during the training process of the entire image processing network, the image encoder can be kept frozen.

[0097] In actual implementation, the image to be processed, the reference image, the prompt image, and the processed result image can be used as input images and input to the image encoder to obtain the first encoded information of the image to be processed, the second encoded information of the reference image, and the third encoded information corresponding to the prompt image and the processed result image. Then, combining the first encoded information, the second encoded information, and the third encoded information, a task processing instruction is generated, so that the task processing instruction can fuse the object reference features of the reference image and the object prompt features of the prompt information to indicate the object processing method of the first category of objects.

[0098] In an optional implementation manner, taking the image segmentation scenario as an example, the prompt image is the above-mentioned segmentation prompt image, and the processed result image is the above-mentioned object segmentation image, that is, the contour image of the second type of image in the segmentation prompt image. In another optional implementation manner, taking the object category recognition scenario as an example, the prompt image is an image including the second category of objects, and the processed result image is a local feature image of the second object.

[0099] In the embodiments of this specification, an image encoder can be used to extract the visual features in the image to be processed, the reference image, and the prompt information, so as to use the extracted visual features for semantic segmentation and process objects of specific categories subsequently.

[0100] In an optional implementation manner of this embodiment, according to the image to be processed, the reference image, and the prompt information, a task processing instruction is generated, including:

[0101] Using a visual alignment layer, visually align the image to be processed and the reference image to generate visual alignment marks;

[0102] Generate a task processing instruction according to the prompt information and the visual alignment marks.

[0103] It should be noted that the objects in the image to be processed and the reference image are of the same category. In order to process the image to be processed based on the reference image, the image to be processed and the reference image can be visually aligned to generate visual alignment marks.

[0104] In actual implementation, the visual features in the image to be processed and the reference image can be respectively extracted through an image encoder to obtain the first encoded information and the second encoded information. Therefore, the first encoded information and the second encoded information can be input into the visual alignment layer, and the visual features in the first encoded information and the second encoded information are visually aligned by the visual alignment layer to obtain a set of aligned visual tokens. Then, combining the prompt information and the visual alignment marks, a task processing instruction is generated. Specifically, the visual alignment marks and the third encoded information corresponding to the prompt information can be fused to obtain the task processing instruction.

[0105] Applying the solution of the embodiment of this specification, a visual alignment layer is added. By using the visual alignment layer to visually align the encoded information of the reference image and the image to be processed, the visual alignment marks obtained through visual alignment can more accurately provide which feature a certain visual feature should refer to for processing, providing corresponding visual references, thereby improving the accuracy of image processing.

[0106] In an optional implementation manner of this embodiment, the generating a task processing instruction according to the prompt information and the visual alignment marks may include the following steps:

[0107] Generate a segmented task instruction according to the visual alignment marks and the set processing rules corresponding to the first category of objects, where the segmented task instruction is used to describe the output representation of the image processing model;

[0108] Using a context fusion layer, obtain a context fusion instruction according to the prompt information and the visual alignment marks;

[0109] Merge the segmented task instruction and the context fusion instruction to obtain the task processing instruction.

[0110] In actual implementation, corresponding processing rules can be configured for each category. In different task scenarios, the processing rules can be the same or different. For example, in the image segmentation scenario, the set processing rule corresponding to the first category object can be to use a set polygon to represent the edge contour of the first category object; or, the set processing rule corresponding to the first category object can also be to use a rectangle to enclose the first category object. The embodiments of this specification do not limit this and can be configured based on actual scenario requirements.

[0111] In addition, the trained context fusion layer can be used to perform multimodal fusion on the prompt information and the visual alignment mark to obtain a context fusion instruction, and use the context fusion instruction to guide the image processing model on how to analyze and recognize the visual features of the image to be processed, so as to obtain the corresponding processing result.

[0112] In the embodiments of this specification, the task processing instruction includes two parts. One part is the segmented task instruction, which is used to indicate how semantic segmentation should be performed under the first category object and the restrictions on the output data of the image processing model; the other part is the context fusion instruction, which is used to describe how to specifically process the object in the reference image and the prompt information, and provide guidance on the processing method for the image processing model. The segmented task instruction is used to indicate what data should be output, and the context fusion instruction is used as a demonstration example to guide how to segment the first category object, so as to facilitate subsequent use of the image processing model to implement image segmentation of specific categories.

[0113] In an optional implementation manner of this embodiment, taking the image segmentation scenario as an example, the task processing instruction is an image segmentation instruction; according to the visual alignment mark and the set processing rule corresponding to the first category object, a segmented task instruction is generated, including:

[0114] According to the visual alignment mark and the set processing rule corresponding to the first category object, determine the edge format corresponding to the first category object, where the edge format includes at least one of the number of vertices of the polygon forming the edge, the vertex arrangement method, the vertex coordinate format, and the coordinate range;

[0115] Generate a segmented task instruction according to the edge format corresponding to the first category object.

[0116] In actual implementation, in the image segmentation scenario, taking the set processing rule corresponding to the first category of objects as an example where the edge contour of the first category of objects is represented by a set polygon, the number of vertices, vertex arrangement pattern, and vertex coordinate format of the polygon forming the edge can be determined based on the set processing rule corresponding to the first category of objects. Additionally, the visual alignment marker includes the visual features of the image to be processed and the reference image. Therefore, based on the visual alignment marker, the limitations on the output data of the image processing model can be determined, such as the limitation of the coordinate range. Thus, the coordinate range can be determined using the aligned visual alignment marker, and then a segmentation task instruction can be generated.

[0117] For example, taking the image segmentation scenario as an example, the set processing rule corresponding to the first category of objects is to represent the edge contour of the first category of objects using a 16-sided polygon. Based on the visual alignment marker, it is determined that the coordinate values of the edge contour should be within (0, 384). At this time, the edge format corresponding to the first category of objects includes that the number of vertices of the polygon forming the edge is 16, the vertices are arranged in a clockwise direction, the vertex coordinate format is (x, y), and the coordinate range is (0, 384). The generated segmentation task instruction is: "For each object in the image similar to the 'first category of objects', output the coordinates of a 16-point polygon containing the object. These points should be arranged in a clockwise direction, and the output should be (c1, c2,..., cn), where cn is the coordinate of the nth object, and its format should be ((x1, y1), (x2, y2), …, (x16, y16)), and the coordinate values should be within (0, 384)".

[0118] It should be noted that in the image segmentation scenario, based on the edge format such as the number of vertices, vertex arrangement pattern, vertex coordinate format, and coordinate range of the polygon forming the edge, the output representation of the image processing model can be described, so as to guide the image processing model to output the corresponding processing result according to this output representation, enabling the image processing model to have the ability to analyze and recognize visual features and achieve image segmentation.

[0119] In an optional implementation manner of this embodiment, taking the image segmentation scenario as an example, the above-mentioned obtaining of the context fusion instruction by using the context fusion layer according to the prompt information and the visual alignment marker may include the following steps:

[0120] Input the visual alignment marker and the prompt information into the trained context fusion layer, and use the cross-attention mechanism in the context fusion layer to perform multi-modal fusion on the visual alignment marker and the prompt information to obtain a fusion feature;

[0121] Generate a context fusion instruction based on the fusion feature.

[0122] Among them, the context fusion layer can be obtained through training in the following manner:

[0123] Obtain a processed sample image, a reference sample image, and prompt sample information, where the prompt sample information includes a prompt sample image and a corresponding sample contour;

[0124] Use an image encoder to obtain first sample encoding information of the processed sample image, second sample encoding information of the reference sample image, and third sample encoding information of the prompt sample image;

[0125] Input the visually aligned first sample encoding information and second sample encoding information, as well as the third sample encoding information, into a context fusion layer, and use the cross-attention mechanism in the context fusion layer to obtain a predicted fusion feature;

[0126] Multiply the predicted fusion feature by the third sample encoding information to obtain a predicted contour of the prompt sample image;

[0127] Calculate a loss function based on the predicted contour and the sample contour, and adjust the parameters of the context fusion layer based on this loss function to obtain a trained context fusion layer.

[0128] In actual implementation, inputting the visually aligned first sample encoding information and second sample encoding information, as well as the third sample encoding information, into a context fusion layer, and using the cross-attention mechanism in the context fusion layer to obtain a predicted fusion feature may include the following steps: randomly mask the first sample encoding information, second sample encoding information, and third sample encoding information in the context fusion layer to obtain first masked encoding information, second masked encoding information, and third masked encoding information; then, use the cross-attention mechanism to perform multimodal fusion on the first masked encoding information, second masked encoding information, and third masked encoding information to obtain a fusion feature.

[0129] Exemplarily, the input of the context fusion layer is the first sample encoding information visual token Fq of the image to be processed, the second sample encoding information visual token Fs of the reference sample image, the third sample encoding information visual token Fi of the prompt sample image, and the prompt sample image and the corresponding sample contour G. Then the training process of the context fusion layer is: first randomly mask Fs, Fq, Fi, that is, randomly replace some of their features (tokens) with fixed values (such as 0); then, use the cross-attention mechanism to perform multimodal fusion on the three masked ones, obtain a fusion feature using the following formula (1), and then multiply the fusion feature by the original Fi to obtain a predicted contour; calculate a loss function between the predicted contour and the sample contour G to achieve the training purpose.

[0130]

[0131] Among them, Attn(Q, K, V) is the fused feature; Q is the third sample encoding information visualtoken Fi of the prompt sample image; K is the second sample encoding information visual token Fs of the reference sample image; V is the first sample encoding information visual token Fq of the image to be processed. Represents a scaling factor.

[0132] It should be noted that the training process of the context fusion layer and the training of the entire image processing network can be carried out simultaneously, using the same or different training samples. Among them, the entire image processing network can include a visual alignment layer, a context fusion layer, an image processing model, and an optimization network, etc.

[0133] In actual implementation, the context fusion layer can be used to fuse the attribute-region correspondence information under the first category of objects indicated by the reference prompt information and the contour segmentation method indicated by the prompt information to obtain a fused feature, and then generate a context fusion instruction. The attribute-region correspondence information under the first category of objects can indicate the regions corresponding to different attributes under the first category of objects in the reference image. Specifically, in the context fusion layer, artificial intelligence can be used to extract the detailed attributes of the first category of objects, and then, guided by the contour division method in the prompt information, different regions in the reference image can be divided, and the regions corresponding to the detailed attributes of the first category of objects can be searched in each region. The generated attribute-region correspondence information of the first category of objects fuses the object contour of the second category of objects in the prompt information. This is used as a demonstration example, and subsequently, it can guide the image processing model to identify the categories in the image in a more human-like and finer-grained manner for semantic segmentation.

[0134] For example, a set of visual alignment markers (the first encoding information of the image to be processed, the second encoding information of the reference image), and the third encoding information of the prompt information can be input into the trained context fusion layer. Using the context fusion layer, with the object contour of the second category of objects as the guiding information, the corresponding relationship between the attributes and regions under the first category of objects can be fused, such as "smooth coat - ((a1, b2), ……, (an, bn))", "eyes - ((c1, d2), ……, (cn, dn))", "mane - ((e1, f2), ……, (en, fn))", "pointed ears - ((g1, h2), ……, (gn, hn))".

[0135] In the embodiments of this specification, the object reference features of the first category of objects provided by the reference image and the object hint features of the second category of objects provided by the hint information are fused to perform multimodal fusion to generate a context fusion instruction. Then, the segmented task instruction and the context fusion instruction are spliced to obtain a task processing instruction. The object processing method of the first category of objects is indicated by this task processing instruction, so as to guide the image processing model to process the first category of objects in the image to be processed. By adding hint information of other categories, richer and more comprehensive guidance information is provided, avoiding being restricted by the reference image and improving the processing accuracy of the image processing model.

[0136] Continuing with the above example, the generated task processing instruction can be "For each object in the image similar to a 'horse', output the coordinates of a 16-point polygon containing the object. These points should be arranged in a clockwise direction, and the output should be (c1, c2,..., cn), where cn is the coordinate of the nth object, and its format should be ((x1, y1), (x2, y2), …, (x16, y16)). The coordinate values should be within (0, 384). To complete this task, you can refer to the following attributes of a 'horse': smooth coat, eyes, mane, pointed ears. For example, for the reference image, the output should be ((m1, u2), ……, (m16, u16)) because, in these areas ((a1, b2), ……, (an, bn)) is the smooth coat, ((c1, d2), ……, (cn, dn)) are the eyes, ((e1, f2), ……, (en, fn)) is the mane, ((g1, h2), ……, (gn, hn)) are the pointed ears. Then, for the image to be processed, what is the output?"

[0137] In an optional implementation manner of this embodiment, after generating the task processing instruction according to the image to be processed, the reference image, and the hint information, it further includes:

[0138] Using a text editor, perform text encoding on the task processing instruction to obtain the task text encoding feature corresponding to the task processing instruction.

[0139] It should be noted that the task processing instruction generated by fusing the image to be processed, the reference image, and the hint information describes the visual features of the image. Therefore, a text editor can also be used to perform text encoding on the task processing instruction to obtain the task text encoding feature corresponding to the task processing instruction. Among them, the task text encoding feature obtained by encoding can include the class feature "Class Ec" of the first category of objects and the task feature "Task Et" of the task to be processed. Subsequently, the image processing model can be used to perform semantic segmentation on the task text encoding feature to obtain the corresponding processing result.

[0140] In the embodiments of this specification, a text editor is newly added to the image processing network. The text editor is used to convert the visual features of the image into text features, facilitating subsequent analysis and processing by the image processing model, thereby improving the accuracy of image processing.

[0141] Step 306: Using the image processing model, based on the task processing instruction, determine the object processing result of the first category of objects in the image to be processed.

[0142] In an optional implementation manner of this embodiment, the image processing model is a large language model. The large language model is used to output the corresponding object processing result according to the image output representation under the guidance of the task processing instruction, and the task processing instruction is the text information describing the image output representation.

[0143] It should be noted that the image processing model can be a large language model (LLM) pre-trained on a large-scale corpus, which has strong analysis capabilities for text features. The large language model performs semantic segmentation on the task processing instruction, labels each pixel in the image as the corresponding semantic category, and outputs the corresponding object processing result, enabling the large language model to have the analysis and recognition capabilities of visual features, realizing image processing, and effectively supplementing the insufficient information in the reference image by using the large amount of prior knowledge accumulated by the large language model.

[0144] In actual implementation, taking image segmentation as an example, the task processing instruction is an image segmentation instruction, and the object processing result of the first category of objects in the image to be processed is the polygon vertices of the first category of objects in the image to be processed; taking the object category recognition scenario as an example, the task processing instruction is an object category recognition instruction, and the object processing result of the first category of objects in the image to be processed is whether the image to be processed includes the first category of objects.

[0145] Applying the solution of the embodiments of this specification, the prompt information of the second category of objects is added. Combining the reference image of the first category of objects and the prompt information of the second category of objects, a task processing instruction is generated. Through this task processing instruction, the object processing method of the first category of objects is indicated, thereby guiding the image processing model to process the first category of objects in the image to be processed. By adding the prompt information of other categories, richer and more comprehensive guiding information is provided, avoiding being restricted by the reference image, and improving the processing accuracy of the image processing model.

[0146] In an optional implementation manner of this embodiment, if the task processing instruction is text-encoded to obtain the corresponding task text encoding feature, then the above-mentioned step of using the image processing model, based on the task processing instruction, to determine the object processing result of the first category of objects in the image to be processed may include the following steps:

[0147] Encode the task text encoding features into the image processing model to obtain the object processing result of the first category of objects in the image to be processed.

[0148] It should be noted that the task text encoding features may include the class feature "Class Ec" of the first category of objects and the task feature "Task Et" of the task to be processed. In the large language model serving as the image processing model, the class feature "Class Ec" of the first category of objects and the task feature "Task Et" of the task to be processed are processed together to enable the mutual integration of Task and Class information. The process of the large language model processing the task text encoding features is specifically to use a multi-layer Self-Attention network. The large language model performs semantic segmentation on the input task text encoding features, labels each pixel in the image as the corresponding semantic category, and outputs the corresponding object processing result, enabling the large language model to have specific image processing capabilities.

[0149] In actual implementation, for the image segmentation scenario, after the image processing model outputs the polygon vertex coordinates of the first category of objects, the polygon enclosing area can be filled with "1", and the area outside the polygon can be filled with "0" to obtain the binary segmentation mask of the first category of objects. Since the accuracy of the polygon vertex coordinates output by the image processing model may be poor, resulting in uneven edges of the first category of objects, an optimization model can be further introduced to correct the segmentation result of the image processing model.

[0150] In an optional implementation manner of this embodiment, after using the image processing model to determine the object processing result of the first category of objects in the image to be processed based on the task processing instruction, it further includes:

[0151] Input the task text encoding features and the object processing result into the optimization network to obtain the optimized processing result of the first category of objects.

[0152] Specifically, the optimization network can be any segmentation network, including a pixel decoder and a mask converter. The pixel decoder gradually increases the feature map size of the image to be processed, and the mask converter is used to refine the mask.

[0153] It should be noted that the task text encoding features and the object processing result output by the image processing model can be jointly input into the optimization network, and the optimization network is used to fuse the task text encoding features and the image processing model to optimize the object processing result of the first category of objects to obtain the optimized processing result of the first category of objects.

[0154] In actual implementation, taking the image segmentation scenario as an example, a binary mask can be constructed based on the vertex coordinates output by the image processing model, and then the binary mask and the task text encoding features are used as the input of the optimization network. The optimization network is used to generate an optimized segmentation mask, that is, the optimized processing result of the first category of objects.

[0155] In the embodiments of this specification, the task text encoding features are used as the guiding information of the optimization network to guide the optimization network to optimize the object processing result of the image processing model, improving the optimization effect of the optimization network, and thus improving the accuracy of the processing result.

[0156] In an optional implementation manner of this embodiment, inputting the task text encoding features and the object processing result into the optimization network to obtain the optimized processing result of the first category of objects may include the following steps:

[0157] Select the visual features of the target layer of the optimization network;

[0158] Perform text-visual fusion on the task text encoding features and the visual features of the target layer to obtain the optimized processing result of the first category of objects.

[0159] In actual implementation, the object processing result obtained by the image processing model is image information, that is, the features in the visual dimension, while the task text encoding features are the text features describing the object processing result.

[0160] It should be noted that the object processing result obtained by the image processing model can be input into the optimization network. The optimization network is a segmentation network, including multiple layers of visual feature analysis layers, which can gradually increase the mapping size of the input image features, that is, gradually refine the segmentation of the input mask image, optimize the subdivided regions, and realize the refined mask. Specifically, in the optimization network, the visual features of the target layer of the optimization network can be selected. The visual features of the target layer refer to the mask information extracted by the target visual feature analysis layer for the refined segmentation of the input mask image. Then, using the cross-attention mechanism, text-visual fusion is performed on the task text encoding features and the visual features of the target layer, fusing the text features and the video features, and using the result of the multi-modal fusion to realize the optimization of the processing result.

[0161] In actual implementation, select the visual feature of the z-th layer of the optimization network, denoted as The task text encoding feature Text Embedding input into the optimization network is denoted as F t , and then a "text-visual interaction" fusion process is performed. Specifically, it can be calculated using the cross-attention mechanism, as shown in the following formula (2):

[0162]

[0163]

[0164] Among them, φ q ,φ k ,φ v represent three MLP layers (Multilayer Perceptron), represents a scaling factor. Finally, the result of Attn(Q z , K, V) is assigned to to complete the fusion of text and visual features, and obtain the optimized processing result of the first category of objects. By performing fusion calculations in multiple layers of the optimization network, a better fusion effect can be achieved.

[0165] Applying the solution of the embodiments of this specification, the task text encoding features can be used as the guiding information of the optimization network to fuse the text features and visual features, optimize the output result of the image processing model, and obtain the optimized processing result of the first category of objects, which greatly improves the accuracy of image processing.

[0166] In an optional implementation manner of this embodiment, before determining the object processing result of the first category of objects in the to-be-processed image based on the task processing instruction by using the image processing model, it further includes:

[0167] Obtain a sample set, where the sample set includes processed sample images, reference sample images, and prompt sample information. The reference sample images are used to provide sample reference features of the first category of objects, the prompt sample information is used to provide sample prompt features of the second category of objects, and the processed sample images carry processing result labels;

[0168] Generate a sample task instruction according to the processed sample images, reference sample images, and prompt sample information, where the sample task instruction is used to describe the object processing method of the first category of objects in the processed sample images, and the object processing method is obtained based on the sample reference features and sample prompt features;

[0169] Use the image processing model to determine the object prediction result of the first category of objects in the processed sample images based on the sample task instruction;

[0170] Adjust the model parameters of the image processing model according to the processing result labels and object prediction results to obtain a trained image processing model.

[0171] Specifically, the training method of the image processing model is supervised training based on prompt learning. That is, the reference sample images and prompt sample information in the sample set can provide information guidance for the processing process of the processed sample images. The processed sample images carry real sample labels, that is, processing result labels, which are used to guide the training process of the image processing model. The method of obtaining the sample set can be to read a large number of reference sample images and prompt sample information, as well as processed sample images carrying processing result labels, from other data acquisition devices or databases. It can also be to receive a large number of reference sample images and prompt sample information, as well as processed sample images carrying processing result labels, input by the user. The method of obtaining the sample set is specifically selected according to the actual situation, and the embodiments of this specification do not make any limitations in this regard.

[0172] It should be noted that the processing result label is the real processing result of the processed sample image. Specifically, a sample task instruction can be generated according to the processed sample image, reference sample image, and prompt sample information to guide the image processing model to output the object prediction result of the first category of objects; then, the loss value is calculated using the processing result label and the object prediction result, and the model parameters of the image processing model are adjusted to obtain a trained image processing model.

[0173] It is worth noting that the implementation method of "generating a sample task instruction according to the processed sample image, reference sample image, and prompt sample information" is the same as the implementation method of "generating a task processing instruction according to the image to be processed, reference image, and prompt information" above, and the embodiments of this specification will not elaborate further. The implementation method of "using the image processing model to determine the object prediction result of the first category of objects in the processed sample image based on the sample task instruction" is the same as the implementation method of "using the image processing model to determine the object processing result of the first category of objects in the image to be processed based on the task processing instruction" above, and the embodiments of this specification will not elaborate further.

[0174] In practical applications, when adjusting the model parameters of the image processing model according to the processing result label and the object prediction result, the loss value can be calculated based on the processing result label and the object prediction result, and the model parameters of the image processing model are adjusted based on the loss value until a preset stop condition is reached to obtain a trained image processing model. The preset stop condition includes but is not limited to the loss value being less than or equal to a preset threshold, and the number of iterations reaching a preset number of iterations.

[0175] Applying the solution of the embodiment of this specification, a loss value is calculated based on the processing result label and the object prediction result. The loss value is compared with a preset stop condition. If the preset stop condition is not met, the image processing model is continuously trained until the preset stop condition is reached, and the training of the image processing model is completed. By continuously adjusting the model parameters of the image processing model, the finally obtained image processing model can be made more accurate.

[0176] In addition, if the image processing network further includes an optimization network to optimize the object processing result of the image processing model, since constructing a binary mask using the vertex coordinates output by the image processing model and then using it as the input of the optimization network is a non-differentiable process, two independent loss functions can be used to train the optimization network and the image processing model respectively.

[0177] See Figure 5 , Figure 5 shows a flowchart of an image processing model training method provided by an embodiment of this specification. The image processing model training method is applied to a cloud-side device and specifically includes the following steps:

[0178] Step 502: Obtain a sample set, where the sample set includes a processed sample image, a reference sample image, and prompt sample information. The reference sample image is used to provide sample reference features of the first category of objects, and the prompt sample information is used to provide sample prompt features of the second category of objects. The processed sample image carries a processing result label.

[0179] Step 504: Generate a sample task instruction according to the processed sample image, the reference sample image, and the prompt sample information. The sample task instruction is used to describe the object processing method of the first category of objects in the processed sample image, and the object processing method is obtained based on the sample reference features and the sample prompt features.

[0180] Step 506: Use the image processing model to determine the object prediction result of the first category of objects in the processed sample image based on the sample task instruction.

[0181] Step 508: Adjust the model parameters of the image processing model according to the processing result label and the object prediction result to obtain a trained image processing model.

[0182] It should be noted that the implementation manners of steps 502 to 508 can be found in detail in the training manner of the image processing model in the above image processing method, and the embodiments of this specification do not make any limitations on this.

[0183] In practical applications, after obtaining the trained image processing model, the model parameters of the trained image processing model can be sent to the end-side device so that the user can construct an image processing model locally based on the model parameters to complete the image processing task.

[0184] Applying the solution of the embodiment of this specification, according to the processing result label and the object prediction result, adjust the model parameters of the image processing model to obtain a trained image processing model. By continuously adjusting the model parameters of the image processing model, the finally obtained image processing model can be made more accurate.

[0185] The following combines the attached Figure 6 , taking the application of the image processing method provided in this specification in the image segmentation scenario as an example, to further illustrate the image processing method. Among them, Figure 6 shows the processing procedure flowchart of an image segmentation method provided by an embodiment of this specification, which specifically includes the following steps.

[0186] Step 602: Obtain the image to be segmented, the segmentation reference image, and the segmentation prompt information. Among them, the segmentation reference image is used to provide the reference edge features of the first category of objects, and the segmentation prompt information is used to provide the prompt edge features of the second category of objects.

[0187] Step 604: Generate an image segmentation instruction according to the image to be segmented, the segmentation reference image, and the segmentation prompt information. Among them, the image segmentation instruction is used to describe the object segmentation method of the image to be segmented, and the object segmentation method is obtained based on the reference edge features and the prompt edge features.

[0188] Step 606: Use the image segmentation model to determine the target edge information of the first category of objects in the image to be segmented based on the image segmentation instruction.

[0189] It should be noted that the implementation manners of steps 602 to 606 are the same as those of steps 302 to 306 above, and this specification does not make any limitations on this.

[0190] Applying the solution of the embodiment of this specification, add the segmentation prompt information of the second category of objects, combine the segmentation reference image of the first category of objects and the segmentation prompt information of the second category of objects to generate an image segmentation instruction, and use this image segmentation instruction to indicate the object segmentation method of the first category of objects, so as to guide the large language model to segment the objects of the first category in the image to be segmented. By adding the prompt information of other categories, more rich and comprehensive guidance information is provided, avoiding being restricted by the segmentation reference image, and improving the segmentation accuracy of the large language model.

[0191] See Figure 7a and Figure 7b , Figure 7a shows the overall architecture diagram of an image segmentation network provided by an embodiment of this specification, as Figure 7aAs shown in the figure, the image segmentation network includes an Image Encoder, a Visual Alignment layer, a segmentation task layer, an In context fusion layer, a Text Encodr, a Large Language Model (LLM), and a Refinement Network.

[0192] Combined with Figure 7a the architecture of the image segmentation network shown in the figure, Figure 7b it shows a schematic diagram of the processing process of an image segmentation provided by an embodiment of this specification. As Figure 7a and Figure 7b shown, the processing process of the image segmentation includes the following steps:

[0193] Step 702: Assume that the task to be processed is "segment the horse in the image". First, obtain the Query Image to be segmented, the Support Image for reference segmentation, and the segmentation prompt information. The reference segmentation image is an image of a "horse", and the reference segmentation image carries the annotation data of "horse". The segmentation prompt information includes the image of a "sheep" and the contour image of a "sheep".

[0194] Step 704: Input the Query Image to be segmented, the Support Image for reference segmentation, and the segmentation prompt information into the Image Encoder to obtain the first encoded information of the Query Image to be segmented, the second encoded information of the Support Image for reference segmentation, and the third encoded information of the segmentation prompt information.

[0195] Step 706: Input the first encoded information and the second encoded information into the Visual Alignment layer for visual alignment to obtain a set of visual tokens. The visual tokens include the first encoded information of the Query Image to be segmented and the second encoded information of the Support Image for reference segmentation.

[0196] Step 708: Input the visual tokens into the segmentation task layer and the In context fusion layer to generate a segmentation task instruction and an in-context fusion instruction; and input the third encoded information of the segmentation prompt information into the In context fusion layer to generate an in-context fusion instruction.

[0197] Step 710: The segmentation task layer generates a Segmentation task instruction according to the visual tokens and the set processing rules corresponding to "horse"; and the In context fusion layer fuses the visual tokens and the third encoded information of the segmentation prompt information to generate an in-context fusion instruction, and then combines the segmentation task instruction and the in-context fusion instruction to obtain an image segmentation instruction.

[0198] Step 712: Input the image segmentation instruction into the Text Encoder to obtain the text encoding feature (Text Embedding), which includes the information of the class "horse" (Class Ec) and the segmentation task information (TaskEt), so as to convert the visual features in the image into text features by using the text editor.

[0199] Step 714: Input the text encoding feature into the large language model (LLM) to obtain the polygon vertex coordinates output by the LLM, and generate the initial binary mask of the "horse" according to the polygon vertex coordinates.

[0200] Step 716: Input the text encoding feature and the initial binary mask into the Refinement Network, and use the text encoding feature to guide the refinement of the initial binary mask to obtain the optimized binary mask. The optimized binary mask is the finally segmented area of the "horse". Applying the optimized binary mask to the image to be segmented can segment the "horse" in the image to be segmented.

[0201] Applying the solution of the embodiment of this specification, adding the segmentation prompt information of the second category object, combining the segmentation reference image of the first category object and the segmentation prompt information of the second category object, generating an image segmentation instruction, and indicating the object segmentation method of the first category object in the image to be segmented through the image segmentation instruction, so as to guide the large language model to segment the object of the first category object in the image to be segmented. By adding the prompt information of other categories, more rich and comprehensive guiding information is provided, avoiding being restricted by the segmentation reference image, and improving the segmentation accuracy of the large language model.

[0202] Corresponding to the above embodiment of the image processing method, this specification also provides an embodiment of an image processing device. Figure 8 The structural schematic diagram of an image processing device provided by an embodiment of this specification is shown. As Figure 8 shown, the device includes:

[0203] The first acquisition module 802 is configured to acquire the image to be processed, the reference image and the prompt information, where the reference image is used to provide the object reference feature of the first category object, and the prompt information is used to provide the object prompt feature of the second category object;

[0204] The first generation module 804 is configured to generate a task processing instruction according to the image to be processed, the reference image and the prompt information, where the task processing instruction is used to describe the object processing method of the first category object in the image to be processed, and the object processing method is obtained based on the object reference feature and the object prompt feature;

[0205] The first determination module 806 is configured to use an image processing model to determine an object processing result of a first category of objects in the image to be processed based on a task processing instruction.

[0206] Optionally, the prompt information includes a prompt image and a processed result image of the prompt image; the first generation module 804 is further configured to:

[0207] Use an image encoder to obtain first encoded information of the image to be processed, second encoded information of a reference image, and third encoded information corresponding to the prompt image and the processed result image;

[0208] Generate a task processing instruction according to the first encoded information, the second encoded information, and the third encoded information.

[0209] Optionally, the first generation module 804 is further configured to:

[0210] Use a visual alignment layer to visually align the image to be processed and the reference image to generate a visual alignment mark;

[0211] Generate a task processing instruction according to the prompt information and the visual alignment mark.

[0212] Optionally, the first generation module 804 is further configured to:

[0213] Generate a segmented task instruction according to the visual alignment mark and a set processing rule corresponding to the first category of objects, where the segmented task instruction is used to describe the output representation of the image processing model;

[0214] Use a context fusion layer to obtain a context fusion instruction according to the prompt information and the visual alignment mark;

[0215] Merge the segmented task instruction and the context fusion instruction to obtain a task processing instruction.

[0216] Optionally, the task processing instruction is an image segmentation instruction; the first generation module 804 is further configured to:

[0217] Determine an edge format corresponding to the first category of objects according to the visual alignment mark and a set processing rule corresponding to the first category of objects, where the edge format includes at least one of the number of vertices of the polygon forming the edge, the vertex arrangement mode, the vertex coordinate format, and the coordinate range;

[0218] Generate a segmented task instruction according to the edge format corresponding to the first category of objects.

[0219] Optionally, the first generation module 804 is further configured to:

[0220] Input the visual alignment markers and prompt information into the trained context fusion layer, and use the cross-attention mechanism in the context fusion layer to perform multi-modal fusion on the visual alignment markers and prompt information to obtain fusion features;

[0221] Generate a context fusion instruction based on the fusion features.

[0222] Optionally, the device further includes an encoding module, configured to:

[0223] Use a text editor to perform text encoding on the task processing instruction to obtain the task text encoding features corresponding to the task processing instruction;

[0224] Correspondingly, the first determination module 806 is further configured to:

[0225] Input the task text encoding features into the image processing model to obtain the object processing result of the first category of objects in the image to be processed.

[0226] Optionally, the device further includes an optimization module, configured to:

[0227] Input the task text encoding features and the object processing result into the optimization network to obtain the optimized processing result of the first category of objects.

[0228] Optionally, the optimization module is further configured to:

[0229] Select the visual features of the target layer of the optimization network;

[0230] Perform text-visual fusion on the task text encoding features and the visual features of the target layer to obtain the optimized processing result of the first category of objects.

[0231] Optionally, the image to be processed is an image to be object-segmented, and the prompt information includes a segmentation prompt image and the object segmentation image corresponding to the segmentation prompt image;

[0232] The first generation module 804 is further configured to:

[0233] Generate an image segmentation instruction according to the image to be processed, the reference image, the segmentation prompt image, and the object segmentation image.

[0234] Applying the solution of the embodiments of this specification, adding the prompt information of the second category of objects, combining the reference image of the first category of objects and the prompt information of the second category of objects, generating a task processing instruction, and indicating the object processing method of the first category of objects through the task processing instruction, so as to guide the image processing model to process the first category of objects in the image to be processed. By adding the prompt information of other categories, more rich and comprehensive guiding information is provided, avoiding being restricted by the reference image, and improving the processing accuracy of the image processing model.

[0235] The above is a schematic solution of an image processing device according to this embodiment. It should be noted that the technical solution of the image processing device belongs to the same concept as the technical solution of the above image processing method. For the details not described in the technical solution of the image processing device, reference can be made to the description of the technical solution of the above image processing method.

[0236] Corresponding to the above embodiment of the image processing model training method, this specification also provides an embodiment of an image processing model training device. Figure 9 FIG. shows a schematic structural diagram of an image processing model training device provided by an embodiment of this specification.

[0237] As Figure 9 shown, the device is applied to a cloud-side device, and the device includes:

[0238] A second acquisition module 902, configured to acquire a sample set, where the sample set includes a processed sample image, a reference sample image, and prompt sample information. The reference sample image is used to provide sample reference features of a first category of objects, the prompt sample information is used to provide sample prompt features of a second category of objects, and the processed sample image carries a processing result label;

[0239] A second generation module 904, configured to generate a sample task instruction according to the processed sample image, the reference sample image, and the prompt sample information, where the sample task instruction is used to describe the object processing method of the first category of objects in the processed sample image, and the object processing method is obtained based on the sample reference features and the sample prompt features;

[0240] A second determination module 906, configured to use the image processing model to determine an object prediction result of the first category of objects in the processed sample image based on the sample task instruction;

[0241] A training module 908, configured to adjust the model parameters of the image processing model according to the processing result label and the object prediction result, and obtain a trained image processing model.

[0242] Applying the solution of the embodiment of this specification, a loss value is calculated based on the processing result label and the object prediction result, and the loss value is compared with a preset stop condition. When the preset stop condition is not met, the image processing model is continuously trained until the preset stop condition is reached, and the training is completed to obtain the image processing model. By continuously adjusting the model parameters of the image processing model, the finally obtained image processing model can be made more accurate.

[0243] The above is a schematic solution of an image processing model training device according to this embodiment. It should be noted that the technical solution of this image processing model training device and the technical solution of the above image processing model training method belong to the same concept. For the details not described in detail in the technical solution of the image processing model training device, reference can be made to the description of the technical solution of the above image processing model training method.

[0244] Corresponding to the above embodiment of the image segmentation method, this specification also provides an embodiment of an image segmentation device. Figure 10 The following shows a schematic structural diagram of an image segmentation device provided by an embodiment of this specification. As Figure 10 shown, the device includes:

[0245] A third acquisition module 1002, configured to acquire an image to be segmented, a segmentation reference image, and segmentation prompt information, where the segmentation reference image is used to provide reference edge features of the first category of objects, and the segmentation prompt information is used to provide prompt edge features of the second category of objects;

[0246] A third generation module 1004, configured to generate an image segmentation instruction according to the image to be segmented, the segmentation reference image, and the segmentation prompt information, where the image segmentation instruction is used to describe the object segmentation method of the image to be segmented, and the object segmentation method is obtained based on the reference edge features and the prompt edge features;

[0247] A third determination module 1006, configured to use an image segmentation model to determine the target edge information of the first category of objects in the image to be segmented based on the image segmentation instruction.

[0248] Applying the solution of the embodiment of this specification, the segmentation prompt information of the second category of objects is added, and combined with the segmentation reference image of the first category of objects and the segmentation prompt information of the second category of objects, an image segmentation instruction is generated. Through this image segmentation instruction, the object segmentation method of the first category of objects is indicated, so as to guide the large language model to segment the first category of objects in the image to be segmented. By adding the prompt information of other categories, more rich and comprehensive guidance information is provided, avoiding being restricted by the segmentation reference image, and improving the segmentation accuracy of the large language model.

[0249] The above is a schematic solution of an image segmentation device according to this embodiment. It should be noted that the technical solution of this image segmentation device and the technical solution of the above image segmentation method belong to the same concept. For the details not described in detail in the technical solution of the image segmentation device, reference can be made to the description of the technical solution of the above image segmentation method.

[0250] Figure 11A structural block diagram of a computing device provided according to an embodiment of this specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 through a bus 1130, and a database 1150 is used to store data.

[0251] The computing device 1100 further includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interfaces (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0252] In an embodiment of this specification, the above components of the computing device 1100 and Figure 11 other components not shown may also be connected to each other, for example, through a bus. It should be understood that Figure 11 the shown structural block diagram of the computing device is for illustrative purposes only and is not a limitation on the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0253] The computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.) or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). The computing device 1100 can also be a mobile or stationary server.

[0254] Wherein, the processor 1120 is configured to execute the following computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above-mentioned image processing method, image processing model training method or image segmentation method are implemented.

[0255] The above is a schematic solution of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solutions of the above-mentioned image processing method, image processing model training method or image segmentation method belong to the same concept. For the details not described in detail in the technical solution of the computing device, reference can be made to the descriptions of the technical solutions of the above-mentioned image processing method, image processing model training method or image segmentation method.

[0256] An embodiment of this specification also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the above-mentioned image processing method, image processing model training method or image segmentation method are implemented.

[0257] The above is a schematic solution of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solutions of the above-mentioned image processing method, image processing model training method or image segmentation method belong to the same concept. For the details not described in detail in the technical solution of the storage medium, reference can be made to the descriptions of the technical solutions of the above-mentioned image processing method, image processing model training method or image segmentation method.

[0258] An embodiment of this specification also provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above-mentioned image processing method, image processing model training method or image segmentation method are implemented.

[0259] The above is a schematic solution of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solutions of the above image processing method, image processing model training method, or image segmentation method belong to the same concept. For the details not described in the technical solution of the computer program product, reference can be made to the descriptions of the technical solutions of the above image processing method, image processing model training method, or image segmentation method.

[0260] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0261] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. A computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, external hard drives, magnetic disks, optical discs, computer memories, read-only memories (ROMs), random access memories (RAMs), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0262] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0263] In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0264] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the embodiments of the present specification. These embodiments are selected and specifically described in the present specification to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Obtaining an image to be processed, a reference image, and prompt information, wherein the reference image is used to provide object reference features of a first category of objects, and the prompt information is used to provide object prompt features of a second category of objects; Generating a task processing instruction according to the image to be processed, the reference image, and the prompt information, wherein the task processing instruction is used to describe an object processing manner of the first category of objects in the image to be processed, and the object processing manner is obtained based on the object reference features and the object prompt features; Using an image processing model, determining an object processing result of the first category of objects in the image to be processed based on the task processing instruction.

2. The image processing method according to claim 1, wherein the prompt information includes a prompt image and a processed result image of the prompt image; the generating a task processing instruction according to the image to be processed, the reference image, and the prompt information includes: Using an image encoder to obtain first encoded information of the image to be processed, second encoded information of the reference image, and third encoded information corresponding to the prompt image and the processed result image; Generating the task processing instruction according to the first encoded information, the second encoded information, and the third encoded information.

3. The image processing method according to claim 1, wherein the generating a task processing instruction according to the image to be processed, the reference image, and the prompt information includes: Using a visual alignment layer to visually align the image to be processed and the reference image to generate a visual alignment mark; Generating the task processing instruction according to the prompt information and the visual alignment mark.

4. The image processing method according to claim 3, wherein the generating the task processing instruction according to the prompt information and the visual alignment mark includes: Generating a segmented task instruction according to the visual alignment mark and a set processing rule corresponding to the first category of objects, wherein the segmented task instruction is used to describe an output representation of the image processing model; Using a context fusion layer to obtain a context fusion instruction according to the prompt information and the visual alignment mark; Merging the segmented task instruction and the context fusion instruction to obtain the task processing instruction.

5. The image processing method according to claim 4, wherein the task processing instruction is an image segmentation instruction; The generating a segmented task instruction according to the visual alignment mark and a set processing rule corresponding to the first category of objects includes: Determining an edge format corresponding to the first category of objects according to the visual alignment mark and a set processing rule corresponding to the first category of objects, wherein the edge format includes at least one of the number of vertices of a polygon forming the edge, the vertex arrangement manner, the vertex coordinate format, and the coordinate range; Generating the segmented task instruction according to the edge format corresponding to the first category of objects.

6. The image processing method according to claim 4, wherein the using a context fusion layer to obtain a context fusion instruction according to the prompt information and the visual alignment mark includes: Input the visual alignment marker and the prompt information into the trained context fusion layer, and use the cross-attention mechanism in the context fusion layer to perform multi-modal fusion on the visual alignment marker and the prompt information to obtain fusion features; Generate the context fusion instruction based on the fusion features.

7. The image processing method according to claim 1, after generating the task processing instruction according to the to-be-processed image, the reference image, and the prompt information, further includes: Use a text editor to perform text encoding on the task processing instruction to obtain a task text encoding feature corresponding to the task processing instruction; Correspondingly, the using the image processing model to determine the object processing result of the first category of objects in the to-be-processed image based on the task processing instruction includes: Input the task text encoding feature into the image processing model to obtain the object processing result of the first category of objects in the to-be-processed image.

8. The image processing method according to claim 7, after using the image processing model to determine the object processing result of the first category of objects in the to-be-processed image based on the task processing instruction, further includes: Input the task text encoding feature and the object processing result into an optimization network to obtain an optimized processing result of the first category of objects.

9. The image processing method according to claim 8, the inputting the task text encoding feature and the object processing result into an optimization network to obtain an optimized processing result of the first category of objects includes: Select the target layer visual feature of the optimization network; Perform text-visual fusion on the task text encoding feature and the target layer visual feature to obtain an optimized processing result of the first category of objects.

10. The image processing method according to claim 1, the to-be-processed image is an image to be subjected to object segmentation, and the prompt information includes a segmentation prompt image and an object segmentation image corresponding to the segmentation prompt image; The generating the task processing instruction according to the to-be-processed image, the reference image, and the prompt information includes: Generate an image segmentation instruction according to the to-be-processed image, the reference image, the segmentation prompt image, and the object segmentation image.

11. The image processing method according to claim 1, the image processing model is a large language model, and the large language model is used to output a corresponding object processing result according to the guidance of the task processing instruction according to the image output representation, and the task processing instruction is text information describing the image output representation.

12. An image processing model training method, applied to a cloud-side device, includes: Obtain a sample set, where the sample set includes a processed sample image, a reference sample image, and prompt sample information, the reference sample image is used to provide a sample reference feature of a first category of objects, the prompt sample information is used to provide a sample prompt feature of a second category of objects, and the processed sample image carries a processing result label; Generate a sample task instruction according to the processed sample image, the reference sample image, and the hint sample information, where the sample task instruction is used to describe the object processing method of the first category of objects in the processed sample image, and the object processing method is obtained based on the sample reference feature and the sample hint feature; Use an image processing model to determine the object prediction result of the first category of objects in the processed sample image based on the sample task instruction; Adjust the model parameters of the image processing model according to the processing result label and the object prediction result to obtain a trained image processing model.

13. An image segmentation method, comprising: Obtain an image to be segmented, a segmentation reference image, and segmentation hint information, where the segmentation reference image is used to provide reference edge features of the first category of objects, and the segmentation hint information is used to provide hint edge features of the second category of objects; Generate an image segmentation instruction according to the image to be segmented, the segmentation reference image, and the segmentation hint information, where the image segmentation instruction is used to describe the object segmentation method of the image to be segmented, and the object segmentation method is obtained based on the reference edge feature and the hint edge feature; Use an image segmentation model to determine the target edge information of the first category of objects in the image to be segmented based on the image segmentation instruction.

14. A computing device, comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 11, or the image processing model training method according to claim 12, or the image segmentation method according to claim 13 are implemented.

15. A computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 11, or the image processing model training method according to claim 12, or the image segmentation method according to claim 13 is implemented.

16. A computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the image processing method according to any one of claims 1 to 11, or the image processing model training method according to claim 12, or the image segmentation method according to claim 13 is implemented.

Images