Image processing method, image processing model training method, and image segmentation method

Abstract

Embodiments of the present invention provide an image processing method, an image processing model training method, and an image segmentation method. The image processing method comprises: acquiring an image to be processed, a reference image and prompt information, wherein the reference image is used for providing an object reference feature of a first-type object, and the prompt information is used for providing an object prompt feature of a second-type object; generating a task processing instruction on the basis of the image to be processed, the reference image and the prompt information, wherein the task processing instruction is used for describing an object processing mode of the first-type object in the image to be processed, and the object processing mode is obtained on the basis of the object reference feature and the object prompt feature; and using an image processing model to determine, on the basis of the task processing instruction, an object processing result of the first-type object in the image to be processed. By newly adding prompt information of other types, richer and more comprehensive guidance information is provided, thereby avoiding being limited by the reference image, and improving the processing accuracy of the image processing model.

Description

[0001]TECHNICAL FIELD: Embodiments of the present invention relate to the field of computer technology, and more particularly to methods for image processing, image processing model training, and image segmentation. Background: With the development of computer technology, image processing has become increasingly computer-based, offering advantages such as high efficiency and a high degree of automation. Image processing, the process of performing image recognition, extraction, and other processing, has been widely used in various areas of daily life. Currently, image processing models are commonly used to perform image processing tasks, such as using image segmentation models to perform image segmentation and extract target object regions from an image. For complex image processing tasks, reference information can be used to guide image processing models. However, due to the limitations of reference information, the processing accuracy of image processing models is relatively low. Therefore, a more accurate image processing solution is urgently needed. SUMMARY OF THE INVENTION: In view of this, embodiments of the present invention provide an image processing method. One or more embodiments of the present invention simultaneously relate to an image processing model training method, an image segmentation method, an image processing apparatus, an image processing model training apparatus, an image segmentation apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present invention, an image processing method is provided, comprising: obtaining an image to be processed, a reference image, and prompt information, wherein the reference image is used to provide object reference features of objects of a first category, and the prompt information is used to provide object prompt features of objects of a second category; generating a task processing instruction based on the image to be processed, the reference image, and the prompt information, wherein the task processing instruction describes an object processing method for objects of the first category in the image to be processed, the object processing method being obtained based on the object reference features and the object prompt features; and determining an object processing result for the objects of the first category in the image to be processed using an image processing model and based on the task processing instruction.According to a second aspect of an embodiment of the present invention, there is provided an image processing model training method, applied to a cloud-side device, comprising: acquiring a sample set, wherein the sample set includes a processed sample image, a reference sample image, and prompt sample information, the reference sample image being used to provide a sample reference feature of a first category object, the prompt sample information being used to provide a sample prompt feature of a second category object, and the processed sample image carrying a processing result label; generating a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information, wherein the sample task instruction is used to describe an object processing method for a first category object in the processed sample image, the object processing method being obtained based on the sample reference feature and the sample prompt feature; determining, using an image processing model and based on the sample task instruction, an object prediction result for the first category object in the processed sample image; and adjusting model parameters of the image processing model based on the processing result label and the object prediction result to obtain a trained image processing model. According to a third aspect of an embodiment of the present invention, an image segmentation method is provided, comprising: acquiring an image to be segmented, a segmentation reference image, and segmentation hint information, wherein the segmentation reference image is used to provide reference edge features of objects of a first category, and the segmentation hint information is used to provide hint edge features of objects of a second category; generating an image segmentation instruction based on the image to be segmented, the segmentation reference image, and the segmentation hint information, wherein the image segmentation instruction is used to describe an object segmentation method for the image to be segmented, the object segmentation method being obtained based on the reference edge features and the hint edge features; and determining, using an image segmentation model and based on the image segmentation instruction, target edge information of objects of the first category in the image to be segmented. According to a fourth aspect of an embodiment of the present invention, an image processing device is provided, comprising: a first acquisition module, configured to acquire an image to be processed, a reference image and prompt information, wherein the reference image is used to provide an object reference feature of an object of a first category, and the prompt information is used to provide an object prompt feature of an object of a second category; a first generation module, configured to generate a task processing instruction based on the image to be processed, the reference image and the prompt information, wherein the task processing instruction is used to describe an object processing method for an object of the first category in the image to be processed, and the object processing method is obtained based on the object reference feature and the object prompt feature; and a first determination module, configured to determine, by using an image processing model, an object processing result of the object of the first category in the image to be processed based on the task processing instruction.According to a fifth aspect of an embodiment of the present invention, an image processing model training apparatus is provided, which is applied to a cloud-side device and includes: a second acquisition module, configured to acquire a sample set, wherein the sample set includes a processed sample image, a reference sample image, and prompt sample information, the reference sample image is used to provide a sample reference feature of a first category object, the prompt sample information is used to provide a sample prompt feature of a second category object, and the processed sample image carries a processing result label; a second generation module, configured to generate a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information, wherein the sample task instruction is used to describe an object processing method for the first category object in the processed sample image, and the object processing method is obtained based on the sample reference feature and the sample prompt feature; a second determination module, configured to use an image processing model to determine an object prediction result of the first category object in the processed sample image based on the sample task instruction; and a training module, configured to adjust model parameters of the image processing model based on the processing result label and the object prediction result to obtain a trained image processing model. According to a sixth aspect of the embodiments of the present invention, an image segmentation apparatus is provided, comprising: a third acquisition module configured to acquire an image to be segmented, a segmentation reference image, and segmentation hint information, wherein the segmentation reference image is used to provide reference edge features of objects of a first category, and the segmentation hint information is used to provide hint edge features of objects of a second category; a third generation module configured to generate an image segmentation instruction based on the image to be segmented, the segmentation reference image, and the segmentation hint information, wherein the image segmentation instruction describes an object segmentation method for the image to be segmented, the object segmentation method being obtained based on the reference edge features and the hint edge features; and a third determination module configured to determine target edge information of objects of the first category in the image to be segmented based on the image segmentation instruction using an image segmentation model. According to a seventh aspect of the embodiments of the present invention, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When executed by the processor, the computer-executable instructions implement the steps of the method provided in the first, second, or third aspects above. According to an eighth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.According to a ninth aspect of embodiments of the present invention, a computer program product is provided, comprising a computer program / instructions. When executed by a processor, the computer program / instructions implement the steps of the method provided in the first, second, or third aspects. An image processing method provided by one embodiment of the present invention obtains an image to be processed, a reference image, and prompt information, wherein the reference image is used to provide object reference features for objects of a first category, and the prompt information is used to provide object prompt features for objects of a second category; generates a task processing instruction based on the image to be processed, the reference image, and the prompt information, wherein the task processing instruction describes an object processing method for objects of the first category in the image to be processed, the object processing method being obtained based on the object reference features and the object prompt features; and determines, using an image processing model and based on the task processing instruction, an object processing result for the objects of the first category in the image to be processed. By adding prompt information for objects of the second category and combining the reference image of the first category objects with the prompt information of the second category objects, a task processing instruction is generated. The task processing instruction indicates the object processing method for the first category objects, thereby guiding the image processing model to process the objects of the first category in the image to be processed. By adding prompt information of other categories, richer and more comprehensive guidance information is provided, avoiding the limitation of the reference image and improving the processing accuracy of the image processing model. BRIEF DESCRIPTION OF THE DRAWINGS FIG1 is an architecture diagram of an image processing system provided by one embodiment of the present invention; FIG2 is an architecture diagram of another image processing system provided by one embodiment of the present invention; FIG3 is a flow chart of an image processing method provided by one embodiment of the present invention; FIG4 is a schematic diagram of an image to be processed, a reference image, and prompt information provided by one embodiment of the present invention; FIG5 is a flow chart of an image processing model training method provided by one embodiment of the present invention; FIG6 is a flow chart of the processing process of an image segmentation method provided by one embodiment of the present invention; FIG7a is an overall architecture diagram of an image segmentation network provided by one embodiment of the present invention; FIG7b is a schematic diagram of the processing process of an image segmentation provided by one embodiment of the present invention; FIG8 is a schematic diagram of the structure of an image processing device provided by one embodiment of the present invention; FIG9 is a schematic diagram of the structure of an image processing model training device provided by one embodiment of the present invention; FIG10 is a schematic diagram of the structure of an image segmentation device provided by one embodiment of the present invention; FIG11 is a block diagram of the structure of a computing device provided by one embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS The following description sets forth many specific details to facilitate a full understanding of the present invention.However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The terminology used in one or more embodiments of the present invention is for the purpose of describing specific embodiments only and is not intended to limit the present invention. As used in one or more embodiments of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that while the terms "first," "second," and so on may be used to describe various information in one or more embodiments of the present invention, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, "first" could also be referred to as "second," and similarly, "second" could also be referred to as "first," without departing from the scope of one or more embodiments of the present invention. Depending on the context, the term "if" as used herein could be interpreted as "when," "when," or "in response to determining." Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in one or more embodiments of the present invention are all authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or deny. In one or more embodiments of the present invention, a large model refers to a deep learning model with large model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. Large models can also be referred to as foundation models. These models are pre-trained using large-scale unlabeled corpora to produce pre-trained models with parameters exceeding 100 million. Such models are adaptable to a wide range of downstream tasks and have good generalization capabilities. Examples include large language models (LLMs) and multi-modal pre-training models.In practical applications, large models only require a small number of samples to fine-tune the pre-trained model and can be applied to various tasks. Large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. Key application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. First, the terms used in one or more embodiments of the present invention are explained. Semantic segmentation is an important task in computer vision, aiming to label each pixel in an image with a corresponding semantic category. However, in practical applications, it often faces a challenging problem: small-sample semantic segmentation. Small-sample semantic segmentation refers to the task of performing semantic segmentation when the training dataset has a very limited number of samples. Multimodality refers to learning by combining data from at least two modalities. In this embodiment of the present invention, learning is performed by combining data from both language and vision modalities. Image segmentation: Pixel-level classification is performed on the input image. Mask: Also known as a mask, image masks are generally used to block (either completely or partially) the image being processed to control the region or process of image processing. Image segmentation is a fundamental task in the field of computer vision and has a wide range of applications. Deep learning algorithms trained on large datasets have brought significant progress in this field. However, this training method requires significant resources. Image segmentation models trained on small amounts of data often fail to meet performance requirements due to data shortages. However, advances in large language models (LLMs) can achieve image segmentation. Specifically, large language models pre-trained on large corpora accumulate a wealth of prior knowledge, which can effectively supplement the insufficient information in reference images and provide more effective guidance for training. In few-shot semantic segmentation, the model should develop segmentation capabilities for specific categories based on the limited amount of annotated data, namely, reference images. To achieve this goal, a support feature guided framework is usually adopted, in which relevant features of the target category are extracted from annotated reference images and used as guidance information to segment the image to be segmented.However, very limited reference images contain only a small amount of incomplete and biased feature information. Therefore, frameworks that rely on this feature information for image segmentation are inherently limited by the reference image and cannot achieve sufficiently high accuracy. To address this issue, embodiments of the present invention propose a framework that introduces a large language model to solve the small-sample semantic segmentation problem. This framework incorporates additional categories in addition to the target category as hint information, and develops category-specific segmentation capabilities by combining the reference image and hint information. This framework can leverage richer and more comprehensive information to solve the visual segmentation problem. Specifically, a target image, a reference image, and hint information are obtained. The reference image provides object reference features for objects of a first category, and the hint information provides object hint features for objects of a second category. Based on the target image, the reference image, and hint information, task processing instructions are generated. The task processing instructions describe an object processing method for objects of the first category in the target image. The object processing method is derived based on the object reference features and the object hint features. An image processing model is used to determine the object processing result for the objects of the first category in the target image based on the task processing instructions. Prompt information for objects of the second category is added, and a task processing instruction is generated by combining the reference image of the first category object and the prompt information for the second category object. This task processing instruction indicates the object processing method for the first category object, thereby guiding the image processing model to process the first category object in the image to be processed. By adding prompt information of other categories, richer and more comprehensive guidance information is provided, avoiding the limitations of the reference image and improving the processing accuracy of the image processing model. The present invention provides an image processing method. One or more embodiments of the present invention also relate to an image processing model training method, an image segmentation method, an image processing device, an image processing model training device, an image segmentation device, a computing device, a computer-readable storage medium, and a computer program product, each of which is described in detail in the following embodiments.1 shows an architecture diagram of an image processing system provided according to an embodiment of the present invention. The image processing system may include a client 100 and a server 200. The client 100 is configured to send an image to be processed to the server 200. The server 200 is configured to obtain the image to be processed, a reference image, and prompt information, wherein the reference image is used to provide object reference features of objects of a first category, and the prompt information is used to provide object prompt features of objects of a second category. The server 200 generates a task processing instruction based on the image to be processed, the reference image, and the prompt information, wherein the task processing instruction describes an object processing method for objects of the first category in the image to be processed, and the object processing method is obtained based on the object reference features and the object prompt features. The server 200 determines an object processing result for the objects of the first category in the image to be processed based on the image processing model and the task processing instruction. The server 200 sends the object processing result to the client 100. The client 100 is also configured to receive the object processing result sent by the server 200. Applying the solution of an embodiment of the present invention, prompt information for objects of the second category is added. Combined with the reference image of the first category object and the prompt information for the second category object, a task processing instruction is generated. This task processing instruction indicates the object processing method for the first category object, thereby guiding the image processing model to process the first category object in the image to be processed. By adding prompt information of other categories, richer and more comprehensive guidance information is provided, avoiding the limitations of the reference image and improving the processing accuracy of the image processing model. Referring to FIG2 , FIG2 shows an architecture diagram of another image processing system provided by an embodiment of the present invention. The image processing system may include multiple clients 100 and a server 200. The clients 100 may include end-side devices, and the server 200 may include cloud-side devices. Multiple clients 100 can establish communication connections through the server 200. In image processing scenarios, the server 200 is used to provide image processing services between the multiple clients 100. The multiple clients 100 can act as senders or receivers, respectively, and communicate through the server 200. Users can interact with the server 200 through the client 100 to receive data from other clients 100 or send data to other clients 100. In an image processing scenario, a user can publish a data stream to the server 200 through the client 100. The server 200 generates object processing results based on the data stream and pushes the object processing results to other clients with which communication has been established. The connection between the client 100 and the server 200 is established via a network. The network provides the medium for the communication link between the client 100 and the server 200.The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by the client 100 may need to undergo encoding, transcoding, compression, and other processing before being published to the server 200. The client 100 can be a browser, an application (APP), a web application such as a Hypertext Markup Language 5 (H5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by the server 200, such as the Real Time Communication (RTC) SDK. The client 100 can be deployed in an electronic device and rely on the device or certain applications in the device to operate. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, or a personal computer. Electronic devices can also typically be configured with various other applications, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like. The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide backend training support for models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers or as a single server. The server can also be a server in a distributed system or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data and artificial intelligence platforms, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology. It is worth noting that the image processing method provided in the embodiment of the present invention is generally executed by the server. However, in other embodiments of the present invention, the client may also have similar functions to the server to execute the image processing method provided in the embodiment of the present invention.In other embodiments, the image processing method provided by embodiments of the present invention can also be performed jointly by a client and a server. Referring to FIG3 , FIG3 shows a flowchart of an image processing method provided by one embodiment of the present invention, specifically including steps 302 to 306: Step 302: Obtaining an image to be processed, a reference image, and prompt information, wherein the reference image is used to provide object reference features for objects of a first category, and the prompt information is used to provide object prompt features for objects of a second category. In one or more embodiments of the present invention, when performing image processing, the image to be processed, the reference image, and prompt information can be obtained to provide guidance information for objects of the first category based on the reference image, and guidance information for objects of the second category based on the prompt information, thereby guiding the processing of the objects of the first category in the image to be processed and obtaining object processing results. The objects of the first category are the categories to be processed by the processing task, and the objects of the second category are categories other than the objects of the first category. For example, the objects of the first category are "horses," and the objects of the second category are "sheep," "cows," etc. Furthermore, there can be at least one of each of the image to be processed, the reference image, and the prompt information. This means that one image to be processed can be processed at a time, or two or more images to be processed can be processed in batches. Furthermore, when processing an image to be processed, at least one reference image can provide object reference features of a first-category object, and at least one prompt information can provide object prompt features of a second-category object. If two or more prompt information provide object prompt features of a second-category object, the two or more prompt information can correspond to the same or different second-category objects. Specifically, image processing can be applied to various processing scenarios, such as image segmentation, object category recognition, and image classification. The image to be processed is an image to be processed, such as an image to be segmented. The reference image is an image with object annotation data for a first-category object, which provides object reference features of the first-category object for image processing. The prompt information includes an image of a second-category object and the corresponding processing result, which provides object prompt features of the second-category object for image processing. In actual implementation, an image with object annotation data for a first-category object can be obtained as a reference image based on the task to be processed. The annotation data refers to the processing result data of the first-category object in the image. In addition, an image including objects of the second category and a processing result corresponding to the image may be obtained as prompt information. The pending task may indicate a processing task and an object category, such as an image segmentation task for objects of the first category, a category recognition task for objects of the first category, or a classification task for images of objects of the first category.For example, in an image segmentation scenario, the task is to segment a "horse" from the image to be processed. An image containing the edge coordinates of the "horse" can be obtained as a reference image. An image containing a "sheep" and an image of the sheep's outline can also be obtained. The image and the sheep's outline can be used as prompt information. The reference image and the prompt information can then be used to guide the image processing model in determining the edge coordinates of the "horse" in the image to be processed. In an object category recognition scenario, the task is to identify whether the object in the image to be processed is a "horse." An image containing the identified "horse" can be obtained as a reference image. An image containing the identified "sheep" and an image of its local features can also be obtained. The reference image and the prompt information can then be used to guide the image processing model in identifying whether the object in the image to be processed is a horse. In this embodiment of the present invention, reference images and prompt information are used to provide different types of guidance information to guide subsequent image processing models in processing the image to be processed. Since training the image processing model requires obtaining a training set with processing results, both the reference images and prompt information can be selected from the training set. Alternatively, the reference images and prompt information can be obtained from other platforms, as long as they contain processing results. Step 304: Generate a task processing instruction based on the image to be processed, the reference image, and the prompt information. The task processing instruction describes the object processing method for the first category of objects in the image to be processed. The object processing method is obtained based on the object reference features and the object prompt features. In this embodiment of the present invention, after obtaining the image to be processed, the reference image, and the prompt information, a task processing instruction can be further generated by combining the image to be processed, the reference image, and the prompt information. The task processing instruction indicates the object processing method for the first category of objects in the image to be processed, facilitating the subsequent image processing model to process the object according to the object processing method indicated by the task processing instruction and obtain the object processing result. Because the object processing method integrates object reference features and object hint features, the addition of additional hint information provides richer and more comprehensive guidance information, avoids the limitations of the reference image, and improves the processing accuracy of the image processing model. In an optional implementation of this embodiment, taking image segmentation as an example, the image to be processed is an image for object segmentation, and the hint information includes a segmentation hint image and an object segmentation image corresponding to the segmentation hint image. Generating a task processing instruction based on the image to be processed, the reference image, and the hint information includes generating an image segmentation instruction based on the image to be processed, the reference image, the segmentation hint image, and the object segmentation image.It should be noted that the object segmentation image corresponding to the segmentation hint image refers to the contour image of the second category object in the segmentation hint image. Combining the segmentation hint image and the contour image can provide richer guidance information for image segmentation in addition to the reference image, thereby improving the accuracy of image segmentation. For example, FIG4 shows a schematic diagram of a to-be-processed image, a reference image, and hint information according to an embodiment of the present invention. As shown in FIG4 , taking an image segmentation scenario as an example, the reference image is an image containing a "horse," the to-be-processed image is also an image containing a "horse," and the task to be processed is to segment the "horse" from the to-be-processed image. The hint information includes a segmentation hint image and a contour image. The segmentation hint image is an image containing a "sheep," and the contour image is a contour image of a "sheep." In an optional implementation of this embodiment, the prompt information includes a prompt image and a processed image of the prompt image. Generating a task processing instruction based on the image to be processed, the reference image, and the prompt information includes: using an image encoder to obtain first encoding information of the image to be processed, second encoding information of the reference image, and third encoding information corresponding to the prompt image and the processed image; and generating the task processing instruction based on the first encoding information, the second encoding information, and the third encoding information. The image encoder is an image feature extractor capable of encoding visual features in an image to obtain corresponding encoding information. Furthermore, the image encoder may remain frozen during the entire image processing network training process. In actual implementation, the image to be processed, the reference image, the prompt image, and the processed result image can be used as input images and input into an image encoder to obtain first encoding information for the image to be processed, second encoding information for the reference image, and third encoding information corresponding to the prompt image and the processed result image. The first, second, and third encoding information are then combined to generate a task processing instruction. The task processing instruction incorporates the object reference features of the reference image and the object prompt features of the prompt information to indicate the object processing method for the first category of objects. In one optional embodiment, taking an image segmentation scenario as an example, the prompt image is the aforementioned segmentation prompt image, and the processed result image is the aforementioned object segmentation image, i.e., the contour image of the second category of images in the segmentation prompt image. In another optional embodiment, taking an object category recognition scenario as an example, the prompt image is an image containing the second category of objects, and the processed result image is an image of local features of the second category of objects. In embodiments of the present invention, the image encoder can be used to extract visual features from the image to be processed, the reference image, and the prompt information, so that the extracted visual features can be used to perform semantic segmentation and process objects of a specific category.In an optional implementation of this embodiment, generating a task processing instruction based on a pending image, a reference image, and prompt information includes: visually aligning the pending image with the reference image using a visual alignment layer to generate a visual alignment marker; and generating the task processing instruction based on the prompt information and the visual alignment marker. It should be noted that the objects in the pending image and the reference image are of the same category. To process the pending image based on the reference image, the pending image and the reference image may be visually aligned to generate a visual alignment marker. In actual implementation, an image encoder may be used to extract visual features from the pending image and the reference image, respectively, to obtain first and second encoded information. The first and second encoded information may then be input into a visual alignment layer, where the visual features in the first and second encoded information are visually aligned to obtain a set of aligned visual tokens. The task processing instruction is then generated by combining the prompt information with the visual alignment marker. Specifically, the visual alignment marker and the third encoded information corresponding to the prompt information may be combined to obtain the task processing instruction. The solution of the embodiments of the present invention adds a visual alignment layer, which visually aligns the encoded information of the reference image and the image to be processed. The visual alignment mark obtained through visual alignment can more accurately indicate which feature should be referenced for processing a particular visual feature, providing a corresponding visual reference and thereby improving image processing accuracy. In an optional implementation of this embodiment, generating task processing instructions based on prompt information and the visual alignment mark may include the following steps: generating segmented task instructions based on the visual alignment mark and the set processing rules corresponding to the first category of objects, where the segmented task instructions describe the output representation of the image processing model; utilizing the context fusion layer to obtain context fusion instructions based on the prompt information and the visual alignment mark; and merging the segmented task instructions with the context fusion instructions to obtain the task processing instructions. In actual implementation, a corresponding processing rule may be configured for each category. In different task scenarios, the processing rules may be the same or different. For example, in an image segmentation scenario, the set processing rule corresponding to the first category of objects may be to use a set polygon to represent the edge contour of the first category of objects; or the set processing rule corresponding to the first category of objects may also be to use a rectangular box to select the first category of objects. The embodiment of the present invention is not limited to this, and can be configured based on actual scenario requirements.Furthermore, the trained context fusion layer can be used to perform multimodal fusion of prompt information and visual alignment markers to obtain context fusion instructions. These instructions can then be used to instruct the image processing model on how to analyze and identify the visual features of the image to be processed, thereby achieving the corresponding processing results. In this embodiment of the present invention, task processing instructions consist of two parts: a segmentation task instruction, which indicates how semantic segmentation should be performed for objects in the first category and restricts the output data of the image processing model; and a context fusion instruction, which describes how objects are specifically processed in the reference image and prompt information, providing guidance to the image processing model on the processing method. The segmentation task instruction indicates the output data, while the context fusion instruction serves as an example to guide how to segment objects in the first category, thereby facilitating the subsequent use of the image processing model to achieve image segmentation of specific categories. In an optional implementation of this embodiment, taking an image segmentation scenario as an example, the task processing instruction is an image segmentation instruction; generating a segmentation task instruction based on the visual alignment mark and the set processing rules corresponding to the first category object includes: determining the edge format corresponding to the first category object based on the visual alignment mark and the set processing rules corresponding to the first category object, where the edge format includes at least one of the number of vertices, vertex arrangement, vertex coordinate format, and coordinate range of the polygons constituting the edge; and generating a segmentation task instruction based on the edge format corresponding to the first category object. In actual implementation, in the image segmentation scenario, taking the set processing rule corresponding to the first category object as an example of using set polygons to represent the edge contour of the first category object, the number of vertices, vertex arrangement, and vertex coordinate format of the polygons constituting the edge can be determined based on the set processing rules corresponding to the first category object. In addition, the visual alignment mark includes visual features of the image to be processed and the reference image. Therefore, based on the visual alignment mark, restrictions on the output data of the image processing model, such as the coordinate range, can be determined. The aligned visual alignment mark can then be used to determine the coordinate range, and the segmentation task instruction can be generated.For example, taking an image segmentation scenario as an example, the set processing rule corresponding to the first category object is to use a 16-gon polygon to represent the edge contour of the first category object. Based on the visual alignment mark, it is determined that the coordinate value of the edge contour should be within (0, 384). In this case, the edge format corresponding to the first category object includes the number of vertices of the polygon constituting the edge is 16, the vertices are arranged in a clockwise direction, the vertex coordinate format is (x, y), and the coordinate range is (0, 384). The generated segmentation task instruction is: "For each object similar to the 'first category object' in the image, output the coordinates of the 16-point polygon containing the object. These points should be arranged in a clockwise direction. The output should be (cl, c2, cn), where cn is the coordinate of the nth object. Its format should be ((xl, yl), (x2, y2), ..., (xl6, yl6)), and the coordinate value should be within (0, 384)." It should be noted that in image segmentation scenarios, the output representation of an image processing model can be described based on edge formats such as the number of vertices, vertex arrangement, vertex coordinate format, and coordinate range of the polygons that constitute the edge. This guides the image processing model to output corresponding processing results based on the output representation, enabling the image processing model to analyze and recognize visual features and achieve image segmentation. In an optional implementation of this embodiment, taking image segmentation as an example, the aforementioned use of the context fusion layer to obtain context fusion instructions based on prompt information and visual alignment marks can include the following steps: inputting the visual alignment marks and prompt information into a trained context fusion layer; utilizing the cross-attention mechanism in the context fusion layer to perform multimodal fusion on the visual alignment marks and prompt information to obtain fused features; and generating context fusion instructions based on the fused features. The context fusion layer can be trained in the following manner: obtaining a processed sample image, a reference sample image, and prompt sample information, wherein the prompt sample information includes a prompt sample image and a corresponding sample contour; using an image encoder to obtain first sample encoding information of the processed sample image, second sample encoding information of the reference sample image, and third sample encoding information of the prompt sample image; inputting the visually aligned first sample encoding information, second sample encoding information, and third sample encoding information into the context fusion layer, and using the cross-attention mechanism in the context fusion layer to obtain predicted fusion features; multiplying the predicted fusion features and the third sample encoding information to obtain a predicted contour of the prompt sample image; calculating a loss function according to the predicted contour and the sample contour, adjusting the parameters of the context fusion layer based on the loss function, and obtaining a trained context fusion layer.In actual implementation, the visually aligned first, second, and third sample encoding information are input into the context fusion layer. A cross-attention mechanism in the context fusion layer is used to obtain predicted fused features. This can include the following steps: randomly masking the first, second, and third sample encoding information in the context fusion layer to obtain first, second, and third masked encoding information; then, multimodally fusing the first, second, and third masked encoding information using the cross-attention mechanism to obtain fused features. For example, the input to the context fusion layer is the first sample encoding information visual token Fq of the image to be processed, the second sample encoding information visual token Fs of the reference sample image, the third sample encoding information visual token Fi of the hint sample image, the hint sample image, and the corresponding sample contour G. The training process of the context fusion layer is: first, randomly mask Fs, Fq, and Fi, that is, randomly replace some of their features (tokens) with fixed values ​​(such as 0); then, use the cross-attention mechanism to perform multimodal fusion on the masked three, and obtain it using the following formula (1). Among them, Atn{Q,KA) is the fusion feature; Q is the third sample coding information visual token Fi of the prompt sample image; K is the second sample coding information visual token Fs of the reference sample image; V is the first sample coding information visual token Fq of the image to be processed o It should be noted that the training process of the context fusion layer and the training of the entire image processing network can be performed simultaneously, using the same or different training samples. The entire image processing network may include a visual alignment layer, a context fusion layer, an image processing model, and an optimization network. In actual implementation, the context fusion layer can fuse the attribute-region correspondence information for the first category of objects indicated by the reference prompt information and the contour segmentation method indicated by the prompt information to obtain a fused feature, and then generate a context fusion instruction. The attribute-region correspondence information for the first category of objects can indicate the regions in the reference image corresponding to different attributes of the first category of objects. Specifically, the context fusion layer can first use artificial intelligence to extract the detailed attributes of the first category objects. Then, using the guidance of the contour segmentation method in the prompt information, different regions in the reference image are divided and searched for regions corresponding to the detailed attributes of the first category objects within each region. The attribute-region correspondence information of the first category objects generated based on this is integrated with the object contours of the second category objects in the prompt information. This serves as a demonstration example and can subsequently guide image processing models on how to identify categories in images in a more human-like and fine-grained manner to perform semantic segmentation. For example, a set of visual alignment marks (first encoding information of the image to be processed, second encoding information of the reference image), and third encoding information of the prompt information can be input into a trained context fusion layer. The context fusion layer uses the object contours of the second category objects as guidance information to fuse and obtain the correspondence between attributes and regions under the first category objects, such as "sleek coat - ((al, b2), ..., (an, bn))", "eyes - ((cl, d2), ..., (cn, dn))", "mane - ((el, f2), ..., (en, fn))", and "pointed ears - ((gl, h2), ..., (gn, hn))". In this embodiment of the present invention, the object reference features of the first category objects provided by the reference image and the object prompt features of the second category objects provided by the prompt information are fused to perform multimodal fusion and generate context fusion instructions, thereby splicing the segmented task instructions and the context fusion instructions. A task processing instruction is obtained, and the object processing method for the first category of objects is indicated by the task processing instruction, thereby guiding the image processing model to process the first category of objects in the image to be processed. By adding prompt information of other categories, richer and more comprehensive guidance information is provided, avoiding the limitation of the reference image, and improving the processing accuracy of the image processing model.Continuing with the above example, the generated task processing instructions could be "For each object in the image that resembles a "horse," output the coordinates of a 16-point polygon containing the object. These points should be arranged in a clockwise direction. The output should be (cl, c2, cn), where cn is the coordinate of the nth object. Its format should be ((xl, yl), (x2, y2), ..., (xl6, yl6)), and the coordinate values ​​should be within (0, 384). To complete this task, you can refer to the following attributes of "horse": sleek coat, eyes, mane, and pointed ears. For example, for the reference image, the output should be ((ml, u2), >, (ml6, ul6)), because in these areas ((al, b2), >, (an, bn)) is a sleek coat, ((cl, d2), , (cn, dn)) is a sleek coat, and ((cl, d2)), , (cn, dn) ) are eyes, ( (el , f2) , > , (en, fn) ) are manes, and ( (gl , h2) , > , (gn, hn) ) are pointed ears. So, for the image to be processed, what is the output? " . In an optional implementation of this embodiment, after generating a task processing instruction based on the image to be processed, the reference image, and the prompt information, the method further includes: using a text editor to perform text encoding on the task processing instruction to obtain task text encoding features corresponding to the task processing instruction. It should be noted that the task processing instruction generated by fusing the image to be processed, the reference image, and the prompt information describes the visual features of the image. Therefore, the task processing instruction can also be text-encoded using a text editor to obtain task text encoding features corresponding to the task processing instruction. The task text encoding features obtained by encoding can include the class feature "Class Ec" of the first category object and the task feature "Task Et" of the task to be processed. Subsequently, the task text encoding features can be semantically segmented using an image processing model to obtain corresponding processing results. In this embodiment of the present invention, a text editor is added to the image processing network. This text editor converts the visual features of an image into text features, facilitating subsequent analysis and processing by the image processing model, thereby improving image processing accuracy. Step 306: Using the image processing model and based on the task processing instructions, determine object processing results for objects of the first category in the image to be processed. In an optional implementation of this embodiment, the image processing model is a large language model. The large language model is configured to output corresponding object processing results according to an image output representation, guided by the task processing instructions, which are textual information describing the image output representation.It should be noted that the image processing model can be a large language model (LLM) pre-trained on a large-scale corpus, which has strong text feature analysis capabilities. The LLM performs semantic segmentation on the task processing instructions, labels each pixel in the image with a corresponding semantic category, and outputs the corresponding object processing result. This enables the LLM to analyze and recognize visual features, enabling image processing and effectively supplementing the insufficient information in the reference image by leveraging the extensive prior knowledge accumulated by the LLM. In practical implementation, taking image segmentation as an example, the task processing instruction is an image segmentation instruction, and the object processing result for objects of the first category in the processed image is the polygon vertices of the objects of the first category in the processed image. Taking object category recognition as an example, the task processing instruction is an object category recognition instruction, and the object processing result for objects of the first category in the processed image is whether the processed image contains objects of the first category. Applying the solution of the embodiment of the present invention, prompt information for objects of the second category is added. The reference image of the first category object and the prompt information for the second category object are combined to generate a task processing instruction. This task processing instruction indicates the object processing method for the first category object, thereby guiding the image processing model to process the first category object in the image to be processed. The addition of prompt information of other categories provides richer and more comprehensive guidance information, avoids the limitations of the reference image, and improves the processing accuracy of the image processing model. In an optional implementation of this embodiment, if the task processing instruction is text-encoded to obtain corresponding task text encoding features, then using the image processing model to determine the object processing result for the first category object in the image to be processed based on the task processing instruction may include the following steps: inputting the task text encoding features into the image processing model to obtain the object processing result for the first category object in the image to be processed. It should be noted that the task text encoding features may include the class features "Class Ec" of the first category of objects and the task features "Task Et" of the task to be processed. In the large language model, which serves as the image processing model, these class features "Class Ec" and "Task Et" are processed together, integrating the task and class information. Specifically, the large language model processes the task text encoding features using a multi-layer self-attention network. The large language model performs semantic segmentation on the input task text encoding features, labels each pixel in the image with a corresponding semantic category, and outputs the corresponding object processing result, thus enhancing the large language model's image processing capabilities.In actual implementation, for image segmentation scenarios, after the image processing model outputs the vertex coordinates of a polygon containing objects of the first category, the area enclosed by the polygon can be filled with "1" and the area outside the polygon can be filled with "0" to obtain a binary segmentation mask for the objects of the first category. Because the accuracy of the polygon vertex coordinates output by the image processing model may be poor, resulting in uneven edges for the objects of the first category, an optimization model can be further introduced to correct the segmentation results of the image processing model. In an optional implementation of this embodiment, after determining the object processing results for the objects of the first category in the image to be processed based on the task processing instructions using the image processing model, the method further includes: inputting the task text encoding features and the object processing results into an optimization network to obtain optimized processing results for the objects of the first category. Specifically, the optimization network can be any segmentation network including a pixel decoder and a mask converter. The pixel decoder gradually increases the size of the feature map of the image to be processed, and the mask converter is used to refine the mask. It should be noted that the task text encoding features and the object processing results output by the image processing model can be jointly input into the optimization network. The optimization network then fuses the task text encoding features and the image processing model to optimize the object processing results for the first category of objects, thereby obtaining optimized processing results for the first category of objects. In actual implementation, taking an image segmentation scenario as an example, a binary mask can be constructed based on the vertex coordinates output by the image processing model. This binary mask and the task text encoding features are then used as inputs to the optimization network, which generates an optimized segmentation mask, i.e., the optimized processing results for the first category of objects. In this embodiment of the present invention, the task text encoding features are used as guidance information for the optimization network to guide the optimization network in optimizing the object processing results of the image processing model, thereby improving the optimization effect of the optimization network and thereby increasing the accuracy of the processing results. In an optional implementation of this embodiment, the aforementioned inputting of the task text encoding features and the object processing results into the optimization network to obtain optimized processing results for the first category of objects may include the following steps: selecting target-layer visual features from the optimization network; and performing text-visual fusion on the task text encoding features and the target-layer visual features to obtain optimized processing results for the first category of objects. In actual implementation, the object processing results obtained by the image processing model represent image information, namely, visual features, while the task text encoding features are text features describing the object processing results. It should be noted that the object processing results obtained by the image processing model can be input into the optimization network. The optimization network is a segmentation network consisting of multiple layers of visual feature analysis. This network can gradually increase the mapping size of the input image features, thereby gradually refining the segmentation of the input mask image and optimizing the subdivided regions to achieve a refined mask.Specifically, the target layer visual features of the optimization network can be selected. The target layer visual features refer to the mask information extracted by the target visual feature analysis layer after refining the segmentation of the input mask image. Then, using the cross-attention mechanism, the task text encoding features and the target layer visual features are fused for text-visual fusion, combining the text features and the video features, and using the result of the multimodal fusion to optimize the processing result. In actual implementation, the visual features of the Z-th layer of the optimization network (visual feature), denoted as \(v_z\), are input into the task text encoding features of the optimization network, Text Embedding, denoted as \(t\). Then, a fusion process of "text-visual interaction" is carried out. Specifically, it can be calculated using the cross-attention mechanism, as shown in the following formula (2): Among them, \(\mathrm{MLP}_1\), \(\mathrm{MLP}_2\), and \(\mathrm{MLP}_3\) represent three MLP layers (Multilayer Perceptron), and \(\sqrt{d_k}\) represents a scaling factor By applying the solution of the embodiment of the present invention, the task text encoding features can be used as guidance information for network optimization, the text features and visual features can be integrated, and the output results of the image processing model can be optimized to obtain the optimized processing results for the first category of objects, thereby greatly improving the accuracy of image processing. In an optional implementation of this embodiment, before determining the object processing result of the first category object in the to-be-processed image based on the task processing instruction using the image processing model, the method further includes: acquiring a sample set, wherein the sample set includes a processed sample image, a reference sample image, and prompt sample information, wherein the reference sample image is used to provide a sample reference feature of the first category object, the prompt sample information is used to provide a sample prompt feature of the second category object, and the processed sample image carries a processing result label; generating a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information, wherein the sample task instruction is used to describe an object processing method for the first category object in the processed sample image, and the object processing method is obtained based on the sample reference feature and the sample prompt feature; determining, using the image processing model and based on the sample task instruction, an object prediction result of the first category object in the processed sample image; and adjusting model parameters of the image processing model based on the processing result label and the object prediction result to obtain a trained image processing model. Specifically, the image processing model is trained using supervised training based on prompt learning. Reference sample images and prompt sample information in the sample set provide guidance for processing the sample images. The processed sample images carry true sample labels, i.e., processing result labels, to guide the training of the image processing model. The sample set can be acquired by reading a large number of reference sample images, prompt sample information, and processed sample images with processing result labels from other data acquisition devices or databases. Alternatively, a large number of reference sample images, prompt sample information, and processed sample images with processing result labels input by a user can be received. The method for acquiring the sample set is selected based on actual circumstances and is not limited in this embodiment of the present invention. It should be noted that the processing result labels are the true processing results of the processed sample images. Specifically, sample task instructions can be generated based on the processed sample images, reference sample images, and prompt sample information to guide the image processing model to output object prediction results for objects of the first category. The processing result labels and object prediction results are then used to calculate a loss value and adjust the model parameters of the image processing model to obtain a trained image processing model.It is worth noting that the implementation of "generating a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information" is similar to the implementation of "generating a task processing instruction based on the image to be processed, the reference image, and the prompt information" described above, and will not be further described in this embodiment of the present invention. The implementation of "using the image processing model to determine the object prediction result for the first category of objects in the processed sample image based on the sample task instruction" is similar to the implementation of "using the image processing model to determine the object processing result for the first category of objects in the image to be processed based on the task processing instruction" described above, and will not be further described in this embodiment of the present invention. In actual applications, when adjusting the model parameters of the image processing model based on the processing result labels and object prediction results, a loss value can be calculated based on the processing result labels and object prediction results, and the model parameters of the image processing model can be adjusted based on the loss value until a preset stopping condition is met, thereby obtaining a trained image processing model. The preset stopping condition includes, but is not limited to, the loss value being less than or equal to a preset threshold and the number of iterations reaching a preset number of iterations. Using the solution of the embodiments of the present invention, a loss value is calculated based on the processing result labels and object prediction results. This loss value is then compared with a preset stopping condition. If the preset stopping condition is not met, the image processing model is trained until the preset stopping condition is met, completing the training and obtaining the image processing model. By continuously adjusting the model parameters of the image processing model, the resulting image processing model can be made more accurate. Furthermore, if the image processing network also includes an optimization network, optimizing the object processing results of the image processing model is a non-differentiable process. Because constructing a binary mask using the vertex coordinates output by the image processing model and then using it as input to the optimization network is a non-differentiable process, two independent loss functions can be used to train the optimization network and the image processing model, respectively. FIG5 shows a flowchart of an image processing model training method according to an embodiment of the present invention. The image processing model training method is applied to a cloud-side device and specifically includes the following steps: Step 502: Acquire a sample set, wherein the sample set includes a processed sample image, a reference sample image, and prompt sample information. The reference sample image is used to provide sample reference features of objects of the first category, the prompt sample information is used to provide sample prompt features of objects of the second category, and the processed sample image carries a processing result label. Step 504: Generate a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information. The sample task instruction describes an object processing method for processing objects of the first category in the sample image. The object processing method is obtained based on the sample reference features and the sample prompt features.Step 506: Using the image processing model, based on the sample task instructions, determine the object prediction result for the first category of objects in the sample image. Step 508: Adjusting the model parameters of the image processing model based on the processing result label and the object prediction result to obtain a trained image processing model. It should be noted that the implementation of steps 502 to 508 is detailed in the image processing model training method described above, and this embodiment of the present invention does not impose any limitations thereon. In practical applications, after obtaining the trained image processing model, the model parameters of the trained image processing model can be sent to the client device, allowing the user to locally construct the image processing model based on the model parameters to complete the image processing task. By applying the solution of the embodiment of the present invention, the model parameters of the image processing model are adjusted based on the processing result label and the object prediction result to obtain a trained image processing model. By continuously adjusting the model parameters of the image processing model, the resulting image processing model can be made more accurate. The image processing method provided by the present invention will be further described below, using the application of the image processing method provided by the present invention in an image segmentation scenario as an example, with reference to FIG6 . FIG6 shows a flowchart of a processing process of an image segmentation method provided by one embodiment of the present invention, which specifically includes the following steps. Step 602: Obtain an image to be segmented, a segmentation reference image, and segmentation hint information. The segmentation reference image is used to provide reference edge features for objects of the first category, and the segmentation hint information is used to provide hint edge features for objects of the second category. Step 604: Generate an image segmentation instruction based on the image to be segmented, the segmentation reference image, and the segmentation hint information. The image segmentation instruction describes an object segmentation method for the image to be segmented, and the object segmentation method is obtained based on the reference edge features and the hint edge features. Step 606: Determine target edge information for objects of the first category in the image to be segmented using an image segmentation model and based on the image segmentation instruction. It should be noted that the implementation of steps 602 to 606 is the same as the implementation of steps 302 to 306 described above, and this embodiment of the present invention does not impose any limitation thereto. By applying the solution of the embodiments of the present invention, segmentation prompt information for objects of the second category is added. The segmentation reference image for objects of the first category and the segmentation prompt information for the second category are combined to generate an image segmentation instruction. The image segmentation instruction indicates the object segmentation method for objects of the first category, thereby guiding the large language model to segment objects of the first category in the image to be segmented. The addition of prompt information of other categories provides richer and more comprehensive guidance information, avoids the limitations of the segmentation reference image, and improves the segmentation accuracy of the large language model.Refer to FIGS. 7a and 7b. FIG. 7a shows the overall architecture diagram of an image segmentation network provided by an embodiment of the present invention. As shown in FIG. 7a, the image segmentation network includes an image encoder (Image Encoder), a visual alignment layer (Visual Alignment), a segmentation task layer, an in-context fusion layer (In context fusion), a text encoder (Text Encodr), a large language model (LLM), and a refinement network (Refinement Network). Combining the architecture of the image segmentation network shown in FIG. 7a, FIG. 7b shows a schematic diagram of the processing process of an image segmentation provided by an embodiment of the present invention. As shown in FIGS. 7a and 7b, the processing process of the image segmentation includes the following steps: Step 702: Assume that the task to be processed is "segment the horse in the image". First, obtain the query image (Query Image), the support image (Support Image), and the segmentation prompt information. The support image is an image of a "horse", and the support image carries the annotation data of "horse". The segmentation prompt information includes an image of a "sheep" and a contour image of the "sheep". Step 704: Input the query image, the support image, and the segmentation prompt information into the image encoder (Image Encoder) to obtain the first encoded information of the query image, the second encoded information of the support image, and the third encoded information of the segmentation prompt information. Step 706: Input the first encoded information and the second encoded information into the visual alignment layer (Visual Alignment) for visual alignment to obtain a set of visual tokens (visual tokens). The visual tokens include the first encoded information of the query image and the second encoded information of the support image. Step 708: Input the visual tokens into the segmentation task layer and the in-context fusion layer (In context fusion) to generate a segmentation task instruction and an in-context fusion instruction; and input the third encoded information of the segmentation prompt information into the in-context fusion layer to generate an in-context fusion instruction. Step 710: The segmentation task layer generates a segmentation task instruction (Segmentation task instruction) according to the visual tokens and the set processing rules corresponding to "horse"; and the in-context fusion layer fuses the visual tokens and the third encoded information of the segmentation prompt information to generate an in-context fusion instruction, and then combines the segmentation task instruction and the in-context fusion instruction to obtain an image segmentation instruction.Step 712: Input the image segmentation instruction into the text encoder (Text Encoder) to obtain text embedding features (Text Embedding). This text embedding feature includes information about the "horse" class (CI Ass Ec) and segmentation task information (Task Et). The text editor then converts the visual features in the image into text features. Step 714: Input the text embedding features into the Large Language Model (LLM) to obtain the polygon vertex coordinates output by the LLM. Based on these polygon vertex coordinates, an initial binary mask for the "horse" image is generated. Step 716: Input the text embedding features and the initial binary mask into the Optimization Network (Refinement Network). The text embedding features are used to guide the refinement of the initial binary mask to obtain an optimized binary mask. This optimized binary mask represents the final segmented "horse" region. Applying this optimized binary mask to the image to be segmented will segment the "horse" image. By applying the solution of an embodiment of the present invention, segmentation hint information for objects of the second category is added. The segmentation reference image for objects of the first category and the segmentation hint information for objects of the second category are combined to generate an image segmentation instruction. This image segmentation instruction indicates the object segmentation method for the first category objects, thereby guiding the large language model to segment the objects of the first category in the image to be segmented. The addition of hint information of other categories provides richer and more comprehensive guidance information, avoids the limitations of the segmentation reference image, and improves the segmentation accuracy of the large language model. Corresponding to the above-mentioned image processing method embodiment, the present invention also provides an image processing device embodiment. Figure 8 shows a schematic structural diagram of an image processing device provided by one embodiment of the present invention. As shown in Figure 8, the apparatus includes: a first acquisition module 802, configured to acquire an image to be processed, a reference image, and prompt information, wherein the reference image is used to provide object reference features of objects of the first category, and the prompt information is used to provide object prompt features of objects of the second category; a first generation module 804, configured to generate a task processing instruction based on the image to be processed, the reference image, and the prompt information, wherein the task processing instruction is used to describe an object processing method for objects of the first category in the image to be processed, and the object processing method is obtained based on the object reference features and the object prompt features; and a first determination module 806, configured to use an image processing model and, based on the task processing instruction, determine an object processing result for objects of the first category in the image to be processed.Optionally, the prompt information includes a prompt image and a processed result image of the prompt image; the first generation module 804 is further configured to: utilize an image encoder to obtain first encoding information of the image to be processed, second encoding information of the reference image, and third encoding information corresponding to the prompt image and the processed result image; and generate a task processing instruction based on the first, second, and third encoding information. Optionally, the first generation module 804 is further configured to: utilize a visual alignment layer to visually align the image to be processed and the reference image to generate a visual alignment marker; and generate a task processing instruction based on the prompt information and the visual alignment marker. Optionally, the first generation module 804 is further configured to: utilize the visual alignment marker and a set processing rule corresponding to the first category of objects to generate a segmentation task instruction, wherein the segmentation task instruction is used to describe the output representation of the image processing model; utilize a context fusion layer to obtain a context fusion instruction based on the prompt information and the visual alignment marker; and combine the segmentation task instruction and the context fusion instruction to obtain the task processing instruction. Optionally, the task processing instruction is an image segmentation instruction; the first generation module 804 is further configured to: determine the edge format corresponding to the first category object based on the visual alignment mark and the set processing rules corresponding to the first category object, wherein the edge format includes at least one of the number of vertices of the polygon constituting the edge, the vertex arrangement, the vertex coordinate format, and the coordinate range; and generate a segmentation task instruction based on the edge format corresponding to the first category object. Optionally, the first generation module 804 is further configured to: input the visual alignment mark and prompt information into a trained context fusion layer, utilize the cross-attention mechanism in the context fusion layer to perform multimodal fusion on the visual alignment mark and prompt information to obtain fusion features; and generate the context fusion instruction based on the fusion features. Optionally, the device further includes an encoding module configured to: perform text encoding on the task processing instruction using a text editor to obtain task text encoding features corresponding to the task processing instruction; accordingly, the first determination module 806 is further configured to: input the task text encoding features into the image processing model to obtain object processing results for the first category object in the image to be processed. Optionally, the apparatus further includes an optimization module configured to: input the task text encoding features and the object processing results into the optimization network to obtain optimization processing results for the first category of objects. Optionally, the optimization module is further configured to: select target layer visual features from the optimization network; and perform text-visual fusion on the task text encoding features and the target layer visual features to obtain optimization processing results for the first category of objects.Optionally, the image to be processed is an image to be segmented, and the prompt information includes a segmentation prompt image and an object segmentation image corresponding to the segmentation prompt image. The first generation module 804 is further configured to generate an image segmentation instruction based on the image to be processed, the reference image, the segmentation prompt image, and the object segmentation image. Applying the solution of this embodiment of the present invention, prompt information for objects of the second category is added. Combined with the reference image of the first category object and the prompt information for the second category object, a task processing instruction is generated. The task processing instruction indicates the object processing method for the first category object, thereby guiding the image processing model to process the first category object in the image to be processed. The addition of prompt information of other categories provides richer and more comprehensive guidance information, avoids the limitations of the reference image, and improves the processing accuracy of the image processing model. The above is a schematic diagram of an image processing device according to this embodiment. It should be noted that the technical solution of this image processing device and the technical solution of the aforementioned image processing method are based on the same concept. For details not described in detail in the technical solution of the image processing device, reference can be made to the description of the technical solution of the aforementioned image processing method. Corresponding to the above-mentioned image processing model training method embodiment, the present invention also provides an image processing model training device embodiment. FIG9 shows a schematic structural diagram of an image processing model training device provided by one embodiment of the present invention. As shown in FIG9 , the apparatus is applied to a cloud-side device, and includes: a second acquisition module 902, configured to acquire a sample set, wherein the sample set includes a processed sample image, a reference sample image, and prompt sample information, the reference sample image is used to provide a sample reference feature of a first category object, the prompt sample information is used to provide a sample prompt feature of a second category object, and the processed sample image carries a processing result label; a second generation module 904, configured to generate a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information, wherein the sample task instruction is used to describe an object processing method for the first category object in the processed sample image, and the object processing method is obtained based on the sample reference feature and the sample prompt feature; a second determination module 906, configured to use an image processing model to determine an object prediction result of the first category object in the processed sample image based on the sample task instruction; and a training module 908, configured to adjust model parameters of the image processing model based on the processing result label and the object prediction result to obtain a trained image processing model. By applying the solution of the embodiment of the present invention, a loss value is calculated based on the processing result label and the object prediction result. The loss value is compared with a preset stopping condition. If the preset stopping condition is not met, the image processing model is trained continuously until the preset stopping condition is met, thereby completing the training and obtaining the image processing model.By continuously adjusting the model parameters of the image processing model, the resulting image processing model can be made more accurate. The above is a schematic diagram of an image processing model training device according to this embodiment. It should be noted that the technical solution of this image processing model training device and the technical solution of the aforementioned image processing model training method share the same concept. For details not described in detail in the technical solution of the image processing model training device, please refer to the description of the technical solution of the aforementioned image processing model training method. Corresponding to the above-described image segmentation method embodiment, the present invention also provides an image segmentation device embodiment. Figure 10 shows a schematic structural diagram of an image segmentation device according to one embodiment of the present invention. As shown in FIG10 , the apparatus includes: a third acquisition module 1002 configured to acquire an image to be segmented, a segmentation reference image, and segmentation prompt information, wherein the segmentation reference image is used to provide reference edge features of objects of the first category, and the segmentation prompt information is used to provide prompt edge features of objects of the second category; a third generation module 1004 configured to generate an image segmentation instruction based on the image to be segmented, the segmentation reference image, and the segmentation prompt information, wherein the image segmentation instruction is used to describe an object segmentation method for the image to be segmented, and the object segmentation method is obtained based on the reference edge features and the prompt edge features; and a third determination module 1006 configured to determine target edge information of objects of the first category in the image to be segmented based on the image segmentation instruction using an image segmentation model. Applying the solution of an embodiment of the present invention, segmentation hint information for objects of the second category is added. The segmentation hint information for objects of the first category is combined with the segmentation hint information for objects of the second category to generate an image segmentation instruction. This image segmentation instruction indicates the object segmentation method for the first category objects, thereby guiding the large language model to segment the first category objects in the image to be segmented. By adding hint information of other categories, richer and more comprehensive guidance information is provided, avoiding the limitations of the segmentation reference image and improving the segmentation accuracy of the large language model. The above is a schematic diagram of an image segmentation device according to this embodiment. It should be noted that the technical solution of this image segmentation device and the technical solution of the aforementioned image segmentation method share the same concept. For details not described in detail in the technical solution of the image segmentation device, please refer to the description of the technical solution of the aforementioned image segmentation method. Figure 11 shows a block diagram of a computing device according to one embodiment of the present invention. Components of computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. Processor 1120 and memory 1110 are connected via a bus 1130. A database 1150 is used to store data.The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface. In one embodiment of the present invention, the aforementioned components of the computing device 1100 and other components not shown in FIG. 11 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG. 11 is for illustrative purposes only and does not limit the scope of the present invention. Those skilled in the art may add or replace other components as needed. Computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1100 may also be a mobile or stationary server.The processor 1120 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned image processing method, image processing model training method, or image segmentation method. The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device shares the same concept as the technical solution of the aforementioned image processing method, image processing model training method, or image segmentation method. For details not described in detail in the technical solution of the computing device, reference can be made to the description of the technical solution of the aforementioned image processing method, image processing model training method, or image segmentation method. An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned image processing method, image processing model training method, or image segmentation method. The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the aforementioned image processing method, image processing model training method, or image segmentation method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the aforementioned image processing method, image processing model training method, or image segmentation method. One embodiment of the present invention also provides a computer program product, comprising a computer program / instructions. When executed by a processor, the computer program / instructions implement the steps of the aforementioned image processing method, image processing model training method, or image segmentation method. The above is an illustrative embodiment of a computer program product of this embodiment. It should be noted that the technical solution of this computer program product is based on the same concept as the technical solution of the aforementioned image processing method, image processing model training method, or image segmentation method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the aforementioned image processing method, image processing model training method, or image segmentation method. The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multi-tasking and parallel processing are possible or may be advantageous. Computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form.Computer-readable media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media. It should be noted that the content of computer-readable media may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals. It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously, depending on the embodiments of the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily required for the embodiments of the present invention. In the above embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments of the present invention disclosed above are intended only to illustrate the present invention. The alternative embodiments do not describe all details in detail, nor do they limit the present invention to the specific implementations described. Obviously, many modifications and variations are possible based on the content of the embodiments of the present invention. These embodiments are selected and described in detail to better explain the principles and practical applications of the embodiments of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

22 Claims 1. An image processing method, wherein: include: An image to be processed, a reference image, and prompt information are obtained, wherein the reference image is used to provide object reference features of objects of a first category, and the prompt information is used to provide object prompt features of objects of a second category; a task processing instruction is generated according to the image to be processed, the reference image, and the prompt information, wherein the task processing instruction is used to describe an object processing method for objects of the first category in the image to be processed, and the object processing method is obtained based on the object reference features and the object prompt features; and an object processing result for the objects of the first category in the image to be processed is determined based on the task processing instruction using an image processing model.

2. The image processing method according to claim 1, wherein the prompt information includes a prompt image and a processing result image of the prompt image; and generating a task processing instruction based on the image to be processed, the reference image, and the prompt information comprises: Using an image encoder, obtaining first encoding information of the image to be processed, second encoding information of the reference image, and third encoding information corresponding to the prompt image and the processing result image; Generate the task processing instruction according to the first coding information, the second coding information and the third coding information.

3. The image processing method according to claim 1, wherein generating a task processing instruction according to the image to be processed, the reference image, and the prompt information comprises: Using a visual alignment layer, visually aligning the image to be processed and the reference image to generate a visual alignment mark; The task processing instruction is generated according to the prompt information and the visual alignment mark.

4. The image processing method according to claim 3, wherein generating the task processing instruction according to the prompt information and the visual alignment mark comprises: Generate a segmented task instruction based on the visual alignment mark and the set processing rules corresponding to the first category object, wherein the segmented task instruction is used to describe the output representation of the image processing model; utilize a context fusion layer to obtain a context fusion instruction based on the prompt information and the visual alignment mark; and merge the segmented task instruction and the context fusion instruction to obtain the task processing instruction.

5. The image processing method according to claim 4, wherein the task processing instruction is an image segmentation instruction; and generating a segmentation task instruction according to the set processing rules corresponding to the visual alignment mark and the first category object comprises: According to the set processing rules corresponding to the visual alignment mark and the first category object, the edge format corresponding to the first category object is determined, wherein the edge format includes at least one of the number of vertices of the polygon constituting the edge, the vertex arrangement method, the vertex coordinate format and the coordinate range; according to the edge format corresponding to the first category object, the segmented task instruction is generated.

6. The image processing method according to claim 4, wherein the utilizing the context fusion layer to obtain the context fusion instruction according to the prompt information and the visual alignment mark comprises: Inputting the visual alignment mark and the prompt information into the trained context fusion layer, and performing multimodal fusion on the visual alignment mark and the prompt information using the cross-attention mechanism in the context fusion layer to obtain a fusion feature; The context fusion instruction is generated based on the fusion feature.

7. The image processing method according to claim 1, after generating a task processing instruction according to the image to be processed, the reference image and the prompt information, further comprising: Using a text editor, the task processing instruction is text-encoded to obtain a task text encoding feature corresponding to the task processing instruction; accordingly, using an image processing model to determine an object processing result of the first category of objects in the image to be processed based on the task processing instruction includes: inputting the task text encoding feature into the image processing model to obtain an object processing result of the first category of objects in the image to be processed.

8. The image processing method according to claim 7, after determining the object processing result of the first category of objects in the image to be processed based on the task processing instruction using the image processing model, further comprising: The task text encoding features and the object processing results are input into the optimization network to obtain the optimization processing results of the first category objects.

9. The image processing method according to claim 8, wherein the step of inputting the task text encoding feature and the object processing result into an optimization network to obtain the optimization processing result of the first category of objects comprises: Selecting target layer visual features of the optimization network; Perform text-visual fusion on the task text encoding features and the target layer visual features to obtain an optimization processing result of the first category object.

10. The image processing method according to claim 1, wherein the image to be processed is an image to be segmented, and the prompt information includes a segmentation prompt image and an object segmentation image corresponding to the segmentation prompt image; and generating a task processing instruction based on the image to be processed, the reference image, and the prompt information comprises: An image segmentation instruction is generated according to the image to be processed, the reference image, the segmentation prompt image, and the object segmentation image.

11. The image processing method according to claim 1, wherein the image processing model is a large language model, and the large language model is used to output a corresponding object processing result according to an image output representation under the guidance of the task processing instruction, and the task processing instruction is text information describing the image output representation.

12. A method for training an image processing model, applied to a cloud-side device, wherein: include: Acquire a sample set, wherein the sample set includes a processed sample image, a reference sample image, and prompt sample information, the reference sample image is used to provide a sample reference feature of a first category object, the prompt sample information is used to provide a sample prompt feature of a second category object, and the processed sample image carries a processing result label; generate a sample task instruction based on the processed sample image, the reference sample image, and the prompt sample information, wherein the sample task instruction is used to describe an object processing method for the first category object in the processed sample image, and the object processing method is obtained based on the sample reference feature and the sample prompt feature; utilize an image processing model and, based on the sample task instruction, determine an object prediction result for the first category object in the processed sample image; adjust model parameters of the image processing model according to the processing result label and the object prediction result to obtain a trained image processing model.

13. An image segmentation method, wherein: include: Acquire an image to be segmented, a segmentation reference image, and segmentation prompt information, wherein the segmentation reference image is used to provide reference edge features of objects of the first category, and the segmentation prompt information is used to provide prompt edge features of objects of the second category; An image segmentation instruction is generated according to the image to be segmented, the segmentation reference image, and the segmentation prompt information, wherein the image segmentation instruction is used to describe an object segmentation method for the image to be segmented, and the object segmentation method is obtained based on the reference edge features and the prompt edge features; and target edge information of the first category of objects in the image to be segmented is determined based on the image segmentation instruction using an image segmentation model.

14. A computing device, wherein: include: memory and processor; The memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the image processing method according to any one of claims 1 to 11, the image processing model training method according to claim 12, or the image segmentation method according to claim 13 are implemented.

15. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the image processing method according to any one of claims 1 to 11, the image processing model training method according to claim 12, or the image segmentation method according to claim 13.

16. A computer program product, wherein: The method comprises a computer program / instruction, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 11, the image processing model training method according to claim 12, or the image segmentation method according to claim 13.

Images