Image processing method, commodity recommendation method and training method of image processing model

Abstract

The invention discloses an image processing method, a commodity recommendation method and a training method of an image processing model, and relates to the field of image processing. The method comprises the steps that in response to a received image processing instruction, a to-be-processed image and to-be-processed elements corresponding to the image processing instruction are determined, and the to-be-processed elements are used for representing image elements needing to be processed in the to-be-processed image; the to-be-processed image is input to the target image processing model, the target image processing model is used for processing the to-be-processed image according to the to-be-processed elements, an image processing result is generated, and the target image processing model is used for processing the to-be-processed image through at least one image sample and multiple image features and multiple description texts corresponding to the at least one image sample. The initial image processing model is trained; and outputting an image processing result. The technical problem that the processing result of the to-be-processed image cannot meet the user demand in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to an image processing method, a product recommendation method, and a training method for an image processing model. Background Art

[0002] With the rapid development of Internet and artificial intelligence technologies, multi-modal information processing has become an indispensable key technology in many fields. For example, in application scenarios such as e-commerce, social media, and search engines, the processing ability of multi-modal data using an image processing model usually directly affects the user experience. Therefore, how to accurately process multi-modal data using an image processing model to meet the needs of users is the main research direction of the image processing model.

[0003] For the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of the present application provide an image processing method, a product recommendation method, and a training method for an image processing model, so as to at least solve the technical problem that the result of processing an image to be processed in related technologies cannot meet the user's needs.

[0005] According to one aspect of the embodiments of the present application, an image processing method is provided, including: in response to receiving an image processing instruction, determining an image to be processed and an element to be processed corresponding to the image processing instruction, where the element to be processed is used to represent an image element in the image to be processed that needs to be processed; inputting the image to be processed into a target image processing model, and using the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result, where the target image processing model is obtained by training an initial image processing model through at least one image sample, and a plurality of image features and a plurality of description texts corresponding to the at least one image sample, the plurality of image features are used to represent the features of a plurality of image elements included in the at least one image sample, and the plurality of description texts are used to represent texts for describing the plurality of image elements generated by using the initial image processing model; outputting the image processing result.

[0006] According to another aspect of the embodiments of the present application, there is also provided a product recommendation method, including: in response to a product recommendation instruction, determining at least one product to be recommended and product recommendation elements corresponding to the product recommendation instruction, where the product recommendation elements are used to characterize the product elements that need to be processed in at least one product to be recommended; inputting the product images of at least one product to be recommended into a product recommendation model, and using the product recommendation model to select a target recommended product from at least one product to be recommended according to the product recommendation elements, where the product recommendation model is obtained by training an initial recommendation model through at least one product sample, as well as a plurality of product features and a plurality of product texts corresponding to at least one product sample, the plurality of product features are used to characterize the features of a plurality of product elements included in at least one product sample, and the plurality of product texts are used to characterize the texts generated by using the initial recommendation model to describe the plurality of product elements; outputting the target recommended product.

[0007] According to another aspect of the embodiments of the present application, there is also provided another image processing method, including: obtaining a to-be-processed image and to-be-processed elements corresponding to an image processing instruction by calling a first interface, where the first interface includes a first parameter, and the parameter value of the first parameter includes the to-be-processed image and the to-be-processed elements, and the to-be-processed elements are used to characterize the image elements that need to be processed in the to-be-processed image; inputting the to-be-processed image into a target image processing model, and using the target image processing model to process the to-be-processed image according to the to-be-processed elements to generate an image processing result, where the target image processing model is obtained by training an initial image processing model through at least one image sample, as well as a plurality of image features and a plurality of description texts corresponding to at least one image sample, the plurality of image features are used to characterize the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to characterize the texts generated by using the initial image processing model to describe the plurality of image elements; outputting the image processing result by calling a second interface, where the second interface includes a second parameter, and the parameter value of the second parameter includes the image processing result.

[0008] According to another aspect of the embodiments of the present application, there is also provided a training method for an image processing model, including: obtaining training data; using a target training strategy and the training data to adjust a pre-trained model to obtain an image processing model, where the image processing model includes: a target image processing model and a product recommendation model; deploying the image processing model to an execution device, where the image processing model is called by the execution device to perform the method according to any one of the above.

[0009] According to one aspect of the embodiments of the present application, there is also provided an image processing apparatus, including: a first determination module, configured to determine a to-be-processed image and a to-be-processed element corresponding to an image processing instruction in response to receiving the image processing instruction, where the to-be-processed element is used to characterize an image element that needs to be processed in the to-be-processed image; a first processing module, configured to input the to-be-processed image into a target image processing model, and use the target image processing model to process the to-be-processed image according to the to-be-processed element to generate an image processing result, where the target image processing model is obtained by training an initial image processing model through at least one image sample, and a plurality of image features and a plurality of description texts corresponding to at least one image sample, the plurality of image features are used to characterize the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to characterize texts generated by using the initial image processing model to describe the plurality of image elements; a first output module, configured to output the image processing result.

[0010] According to another aspect of the embodiments of the present application, there is also provided a product recommendation apparatus, including: a second determination module, configured to determine at least one to-be-recommended product and a product recommendation element corresponding to a product recommendation instruction in response to receiving the product recommendation instruction, where the product recommendation element is used to characterize a product element that needs to be processed in at least one to-be-recommended product; a second processing module, configured to input the product images of at least one to-be-recommended product into a product recommendation model, and use the product recommendation model to select a target recommended product from at least one to-be-recommended product according to the product recommendation element, where the product recommendation model is obtained by training an initial recommendation model through at least one product sample, and a plurality of product features and a plurality of product texts corresponding to at least one product sample, the plurality of product features are used to characterize the features of a plurality of product elements included in at least one product sample, and the plurality of product texts are used to characterize texts generated by using the initial recommendation model to describe the plurality of product elements; a second output module, configured to output the target recommended product.

[0011] According to another aspect of the embodiments of the present application, another image processing apparatus is further provided, including: a first calling module, configured to obtain an image to be processed and elements to be processed corresponding to an image processing instruction by calling a first interface, where the first interface includes a first parameter, and the parameter value of the first parameter includes the image to be processed and the elements to be processed, and the elements to be processed are used to represent the image elements in the image to be processed that need to be processed; a third processing module, configured to input the image to be processed into a target image processing model, and use the target image processing model to process the image to be processed according to the elements to be processed to generate an image processing result, where the target image processing model is obtained by training an initial image processing model through at least one image sample, and a plurality of image features and a plurality of description texts corresponding to at least one image sample, the plurality of image features are used to represent the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to represent the texts for describing the plurality of image elements generated by using the initial image processing model; a second calling module, configured to output the image processing result by calling a second interface, where the second interface includes a second parameter, and the parameter value of the second parameter includes the image processing result.

[0012] According to another aspect of the embodiments of the present application, a training apparatus for an image processing model is further provided, including: a data acquisition module, configured to acquire training data; a model adjustment module, configured to adjust a pre-trained model by using a target training strategy and the training data to obtain an image processing model, where the image processing model includes: a target image processing model and a commodity recommendation model; a model deployment module, configured to deploy the image processing model to an execution device, where the image processing model is called by the execution device to perform the method according to any one of the above.

[0013] According to another aspect of the embodiments of the present application, an image processing system is further provided, including: a client, configured to send an image processing instruction; a server, connected to the client, configured to determine an image to be processed and elements to be processed corresponding to the image processing instruction, where the elements to be processed are used to represent the image elements in the image to be processed that need to be processed; input the image to be processed into a target image processing model, and use the target image processing model to process the image to be processed according to the elements to be processed to generate an image processing result, where the target image processing model is obtained by training an initial image processing model through at least one image sample, and a plurality of image features and a plurality of description texts corresponding to at least one image sample, the plurality of image features are used to represent the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to represent the texts for describing the plurality of image elements generated by using the initial image processing model; the client is further configured to output the image processing result.

[0014] According to another aspect of the embodiments of the present application, there is also provided a computing device, including: a memory storing an executable program; a processor configured to run the program, wherein when the program runs, it executes the methods in various embodiments of the present application.

[0015] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory storing an executable program; a processor connected to the memory through a bus and configured to run the program, wherein when the program runs, it executes the methods in various embodiments of the present application.

[0016] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, which includes a stored executable program, wherein when the executable program runs, it controls the device where the computer-readable storage medium is located to execute the methods in various embodiments of the present application.

[0017] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a computer program that implements the methods in various embodiments of the present application when executed by a processor.

[0018] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a non-volatile computer-readable storage medium storing a computer program that implements the methods in various embodiments of the present application when executed by a processor.

[0019] According to another aspect of the embodiments of the present application, there is also provided a computer program that implements the methods in various embodiments of the present application when executed by a processor.

[0020] In the embodiments of the present application, the following method is adopted: determining a to-be-processed image and to-be-processed elements corresponding to an image processing instruction; inputting the to-be-processed image into a target image processing model, and using the target image processing model to process the to-be-processed image according to the to-be-processed elements to generate an image processing result; outputting the image processing result. By using an initial image processing model to describe the elements included in an image sample, a corresponding plurality of description texts are generated, and then, based on these plurality of description texts and combined with a plurality of image features corresponding to the image sample, the initial image processing model is trained, which can improve the processing ability of the trained target image processing model for the to-be-processed image. Moreover, when processing the to-be-processed image, further introducing the to-be-processed elements corresponding to the image processing instruction can further improve the accuracy of the generated image processing result, thereby ensuring that the image processing result can largely meet the user's requirements for processing the to-be-processed image, and further solving the technical problem that the result of processing the to-be-processed image in the related art cannot meet the user's needs.

[0021] It is easy to notice that the above general description and the following detailed description are only for exemplifying and explaining the present application, and do not constitute a limitation to the present application. Description of the Drawings

[0022] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0023] Figure 1 is a schematic diagram of an image processing scenario shown according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of an image processing method shown according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of an image processing model training framework shown according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of the training and application process of an image processing model shown according to an embodiment of the present application;

[0027] Figure 5 is a flowchart of a product recommendation method shown according to an embodiment of the present application;

[0028] Figure 6 is a flowchart of another image processing method shown according to an embodiment of the present application;

[0029] Figure 7 is a flowchart of a training method of an image processing model shown according to an embodiment of the present application;

[0030] Figure 8 is a structural block diagram of an image processing device shown according to an embodiment of the present application;

[0031] Figure 9 is a structural block diagram of a product recommendation device shown according to an embodiment of the present application;

[0032] Figure 10 is a structural block diagram of another image processing device shown according to an embodiment of the present application;

[0033] Figure 11 is a structural block diagram of a training device of an image processing model shown according to an embodiment of the present application;

[0034] Figure 12 is a structural block diagram of an image processing system shown according to an embodiment of the present application;

[0035] Figure 13 is a structural block diagram of a computing device shown according to an embodiment of the present application;

[0036] Figure 14 is a structural block diagram of an electronic device shown according to an embodiment of the present application. Detailed implementation manners

[0037] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar elements, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0039] The technical solutions provided by the present application are mainly implemented using large language model technology. Here, the large language model refers to a deep learning model with a large number of model parameters, usually including hundreds of millions, tens of billions, hundreds of billions, trillions or even more than one trillion model parameters. The large language model can also be called a foundation model. Through the pre-training of the large language model with a large amount of unlabeled corpus, a pre-trained model with more than one billion parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLMs), multi-modal pre-training models, etc.

[0040] It should be noted that when large language models are applied in practice, the pre-trained models can be fine-tuned with a small number of samples, enabling the large language models to be applied to different tasks. For example, large language models can be widely applied in fields such as Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to tasks in the field of computer vision such as Visual Question Answering (VQA), Image Captioning (IC), and image generation, and can also be widely applied to tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of large language models include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiments of this application, the data processing through the target image processing model and the product recommendation model in the image processing scenario is taken as an example for explanation.

[0041] First, some nouns or terms that appear during the description of the embodiments of this application are applicable to the following explanations:

[0042] Vision-Language Models: VLMs, models used to process visual and text information simultaneously.

[0043] Text-to-Image Retrieval: A cross-modal task that can retrieve the image semantically most matching the given text query. This task can be used to measure the similarity between text and image.

[0044] According to the embodiments of this application, an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0045] Considering the large number of model parameters of large language models and the limited computing resources of mobile terminals, the above method provided by the embodiments of this application can be applied to the Figure 1 application scenarios shown, but not limited thereto. Figure 1 is a schematic diagram of an image processing scenario shown according to the embodiments of this application. In Figure 1In the application scenario shown, the large language model is deployed in server 10. Server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. Here, the client devices 20 can include, but are not limited to: smartphones, tablets, laptops, palmtop computers, personal computers, smart home devices, in-vehicle devices, etc. The client device 20 can interact with the user through a graphical user interface to implement the invocation of the large language model, and further implement the method provided in the embodiments of the present application.

[0046] In the embodiments of the present application, the system composed of the client device and the server can perform the following steps: The client device executes to obtain the image to be processed and the element to be processed corresponding to the image processing instruction; outputs the image processing result. The server executes to input the image to be processed into the target image processing model, and uses the target image processing model to process the image to be processed according to the element to be processed, and generates the image processing result.

[0047] It should be noted that with the rapid development of high-performance computing units, in other application scenarios, the above method provided in the embodiments of the present application can also be applied to a model all-in-one machine. In an optional embodiment, multiple models are built into the model all-in-one machine, and the user can select and adjust one model according to needs to obtain the user's own model. Thus, the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above method provided in the embodiments of the present application. In another optional embodiment, a trained model is built into the large language model all-in-one machine. Thus, the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above method provided in the embodiments of the present application.

[0048] Furthermore, when the user needs to train their own model, they can also upload their own dataset through the client. The dataset is sent from the client to the server, enabling the server to adjust the pre-trained model with the dataset to obtain the user's own model, and then deploy it to the production environment. To facilitate the user's model adjustment requirements, the server can provide complete adjustment tools, development frameworks, and processes, and can support multiple adjustment strategies, enabling the adjusted model to better adapt to different field applications and achieve high customization.

[0049] In the above operating environment, the present application provides the Figure 2 image processing method as shown. Figure 2 is a flowchart of an image processing method shown according to the embodiments of the present application. As Figure 2 shown, the method can include the following steps:

[0050] Step S202: In response to receiving an image processing instruction, determine the image to be processed and the element to be processed corresponding to the image processing instruction.

[0051] The element to be processed is used to represent the image element in the image to be processed that needs to be processed.

[0052] The above-mentioned image processing instruction can be an instruction for processing an image generated according to user requirements, and can include, but is not limited to: instructions such as image cropping, feature description, and element recognition. The above-mentioned element to be processed can be an image element in the image to be processed that needs to be processed according to the image processing instruction. For example, if the image processing instruction is an image cropping instruction, the element to be processed can be the element that needs to be cropped currently; if the image processing instruction is an image description instruction, the element to be processed can be the element that needs to be described currently; if the image processing instruction is an element recognition instruction, the element to be processed can be the element that needs to be recognized currently. It should be noted that the image processing instructions mentioned here are only for exemplary display and are not specifically limited.

[0053] In an alternative solution of this embodiment, in order to improve the efficiency of processing the image to be processed and ensure the accuracy of the processing result, the user can input the corresponding image processing instruction in a pre-set user operation interface. Correspondingly, when receiving the image processing instruction, the image processing system can first determine the image to be processed that needs to be processed currently according to the received image processing instruction, and determine the element to be processed that needs to be processed currently from the image to be processed according to the image processing instruction, so as to ensure that the image processing system processes the image to be processed according to the image processing instruction, such as the accuracy of operations such as regional cropping, feature description, and element recognition.

[0054] Step S204: Input the image to be processed into the target image processing model, and use the target image processing model to process the image to be processed according to the element to be processed, and generate an image processing result.

[0055] The target image processing model is obtained by training an initial image processing model through at least one image sample, as well as a plurality of image features and a plurality of description texts corresponding to at least one image sample. The plurality of image features are used to represent the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to represent the texts for describing the plurality of image elements generated by using the initial image processing model. The plurality of image elements include the element to be processed.

[0056] The above-mentioned target image processing model can be a model that can process different types of data simultaneously. Different types of data can include: data such as image type, text type, audio type, or video type.

[0057] In an alternative solution of this embodiment, considering that various types of data may be involved in the process of processing the image to be processed according to the processing requirements of the user. For example, the user can input voice or prompt word text for processing a specified element in the image to be processed to prompt the image processing system with information such as the processing method and display method of the image to be processed. Correspondingly, when the image processing system processes the image to be processed, it needs to consider various types of data. Therefore, in order to meet the processing requirements of the user, the image processing system can process the image to be processed by introducing the above-mentioned target image processing model. Correspondingly, the image processing system can input the image to be processed into the target image processing model to use the target image processing model to process the image to be processed based on the above-mentioned element to be processed. For example, according to the user's needs, the element to be processed is cropped to obtain a corresponding cropped image, and the element to be processed is described to obtain a corresponding description text, etc., to obtain an image processing result that can meet the user's needs, thereby ensuring the user's usage experience.

[0058] Furthermore, considering that the current image processing model is mainly trained by relying on a large amount of data sets of single image-text pairs, this method usually lacks rich context information and complex semantics, which limits the ability of the image processing model to process long text inferences. And because the diversity and details of images are difficult to be comprehensively expressed by a single description text, it may also cause the image processing model to be difficult to capture fine visual features. Therefore, in order to improve the accuracy of the trained image processing model in processing the image to be processed to meet the user's needs in complex scenarios, the image processing system can train the initial image processing model by using multiple image features corresponding to at least one image sample, that is, the features of multiple image elements included in at least one image sample, and multiple description texts corresponding to these multiple image elements, to obtain a target image processing model with better processing ability. Among them, in order to avoid the influence of introducing other models on the accuracy of the trained target image processing model, the image processing system can input multiple image elements included in at least one image sample into the initial image processing model to generate corresponding multiple description texts by using the initial image processing model.

[0059] Step S206, output the image processing result.

[0060] In an alternative solution of this embodiment, after processing the image to be processed using the target image processing model to obtain the corresponding image processing result, the image processing system can output the corresponding image processing result in the user operation interface for the user to view conveniently. Considering that the user has a certain degree of subjectivity when viewing the image processing result, and the processing process of the pre-trained target image processing model is relatively fixed, the image processing result obtained by processing through the target image processing model may not meet the processing requirements of different users for the image to be processed. Therefore, in order to further improve the user's satisfaction with the output image processing result, the image processing system can also configure a corresponding processing result evaluation area in the user operation interface. The user can evaluate and adjust the image processing result output by the image processing system in this processing result evaluation area. Correspondingly, the image processing system can adjust the model parameters of the target image processing model according to the evaluation result and adjustment result input by the user, combined with the user identity information of this user, so as to construct an image processing model that can further meet the personalized needs of the user, thereby improving the user experience when using the target image processing model.

[0061] In the embodiment of this application, the method includes: determining the image to be processed and the element to be processed corresponding to the image processing instruction; inputting the image to be processed into the target image processing model, and using the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result; outputting the image processing result. By using the initial image processing model to describe the elements included in the image sample to generate corresponding multiple description texts, and then based on these multiple description texts, combined with multiple image features corresponding to the image sample, training the initial image processing model can improve the processing ability of the trained target image processing model for the image to be processed. Moreover, further introducing the element to be processed corresponding to the image processing instruction when processing the image to be processed can further improve the accuracy of the generated image processing result, so as to ensure that the image processing result can largely meet the user's needs for processing the image to be processed, thereby solving the technical problem that the result of processing the image to be processed in the related art cannot meet the user's needs.

[0062] In the embodiment of this application, the image processing result includes: an image description text corresponding to the element to be processed; inputting the image to be processed into the target image processing model, and using the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result, including: obtaining a first prompt word; using an image encoder to encode the image to be processed according to the element to be processed to obtain an image encoding feature corresponding to the element to be processed; extracting features from the first prompt word to obtain description text parameters; generating an image description text based on the first image encoding feature and the description text parameters.

[0063] The above first prompt word can be a prompt word input by the user for guiding the target image processing model to process the image to be processed. For example, the first prompt word may include information such as the element to be processed currently, the processing method for processing the element to be processed, and the display method for displaying the image processing result. The image processing system can process the image to be processed more precisely based on this information to improve the user's satisfaction with the image processing result. The above image description text can be text used to describe the characteristics of the element to be processed. Through the image description text.

[0064] In an alternative solution of this embodiment, the obtained image processing result may at least include the image description text corresponding to the element to be processed. Correspondingly, when using the target image processing model to process the image to be processed, the image processing system may first use an image encoder to encode the image to be processed according to the determined element to be processed to obtain the image encoding features corresponding to the element to be processed. For example, features such as the texture, color, and pose of the element to be processed itself, as well as the spatial position, size ratio, and interaction relationship of the element to be processed relative to the image to be processed. After encoding to obtain the corresponding image encoding features, the image processing system may further use a pre-constructed text generator to generate a description text that matches the element features of the element to be processed based on the image encoding features, so as to facilitate the user to understand the information possessed by the element to be processed. Considering that the reading habits of different users may be different, to improve the user experience when viewing the image description text, when generating the description text, the image processing system may also obtain the first prompt word input by the user and determine the display method of the image processing result required by the user according to the first prompt word. For example, determine information such as the description style, detail level, or abstraction level of the generated description text, and introduce the above first prompt word when generating the corresponding description text based on the image encoding features to obtain an image description text with accurate semantic description and strong expressive ability, thereby improving the user's satisfaction when viewing the image description text.

[0065] In the embodiment of the present application, in response to receiving an image processing instruction, determining the element to be processed corresponding to the image processing instruction includes: obtaining the first prompt word and the instruction type corresponding to the image processing instruction; determining the element to be processed based on the instruction type and the first prompt word.

[0066] The above instruction types can reflect the current processing focus that the image processing system needs to pay attention to. Image processing instructions of different instruction types have different processing focuses. For example, the above instruction types can include, but are not limited to, information retrieval type and detail analysis type. For the information retrieval type, the image processing system can pay more attention to the core elements contained in the image to be processed, such as the element with the largest spatial proportion or the largest number of elements in the element to be processed, or the target element specified by the user. For the detail analysis type, the image processing system can pay more attention to the element features of the image to be processed, such as the background information of the image to be processed, the states and details of different elements in the image to be processed, the emotional color or style of the image to be processed, etc.

[0067] In an alternative solution of this embodiment, in order to improve the accuracy of processing the image to be processed, when determining the element to be processed corresponding to the image processing instruction, the image processing system can first obtain the first prompt word input by the user and the instruction type corresponding to the image processing instruction. According to the first prompt word and the instruction type, the image processing system can accurately determine the element to be processed that needs to be processed currently from the image to be processed, thereby improving the accuracy of the image processing result. For example, assume that the above first prompt word is "describe the background content of the image" and the above instruction type is "detail analysis type". Then the image processing system can identify the elements contained in the image to be processed and divide these elements into foreground and background parts according to the relative positions of the elements. Then, the elements belonging to the background part are determined as the above elements to be processed. Assume that the above first prompt word is "introduce the breed of the puppy in the image" and the above instruction type is "information retrieval type". Then the image processing system can identify the elements contained in the image to be processed and determine the elements that match "puppy" among these elements as the elements to be processed.

[0068] In the embodiment of the present application, the above method further includes: encoding at least one image sample to obtain a plurality of image features; inputting at least one image sample into an initial image processing model, and using the initial image processing model to generate a plurality of description texts; constructing a feature loss function corresponding to the initial image processing model based on the plurality of image features and the plurality of text features corresponding to the plurality of description texts, where the feature loss function is used to characterize the similarity between different image features and different text features; adjusting the model parameters of the initial image processing model based on the feature loss function to obtain a target image processing model.

[0069] In an alternative solution of this embodiment, in order to improve the accuracy of the trained target image processing model, the image processing system may first obtain at least one image sample for model training, and then train the initial image processing model based on multiple image features and multiple description texts corresponding to the at least one image sample, so as to obtain a target image processing model with strong image understanding ability, strong robustness and high accuracy. Specifically, the image processing system may first encode at least one image sample to obtain the features of multiple elements included in the at least one image sample, that is, the above-mentioned multiple image features, and input the at least one image sample into the initial image processing model to generate texts for describing the multiple elements included in the image sample by using the initial image processing model, that is, the above-mentioned multiple description texts. Then, according to the determined multiple image features and multiple text features, a feature loss function corresponding to the initial image processing model is constructed. Through this feature loss function, the image processing system can determine the similarity between the multiple description texts currently generated by using the initial image processing model and the multiple image features actually possessed by the multiple elements in the image sample through the method of text-image retrieval. Based on this, the image processing system can determine the current image processing ability of the initial image processing model according to the feature loss function, and appropriately adjust the model parameters of the initial image processing model based on the feature loss function, so as to obtain a target image processing model with better processing ability.

[0070] In the embodiment of the present application, encoding at least one image sample to obtain multiple image features includes: obtaining element classification identifiers corresponding to different image elements in at least one image sample; based on the element classification identifiers, using a first image encoder to encode at least one image sample to obtain multiple first features of the at least one image sample; based on a multi-layer perceptron, using a second image encoder to encode at least one image sample to obtain multiple second features of the at least one image sample; and summarizing the multiple first features and the multiple second features to obtain multiple image features.

[0071] The above-mentioned element classification identifiers may refer to classification identifiers used to guide the image encoder to focus on the classification of multiple elements included in the image to be processed during the encoding process of the image to be processed, and may include, but are not limited to, classification identifiers such as background identifier, foreground identifier, element 1 identifier, element 2 identifier, color identifier, texture identifier, etc. Different element classification identifiers may correspond to different elements or the same element. Specifically, the element classification identifiers corresponding to different elements may be set by the user himself or obtained by the image processing system through identifying the features of the elements, and no specific limitation is made.

[0072] In an alternative solution of this embodiment, in order to extract and express the content features of an image from different perspectives and depths, so as to achieve a more comprehensive and fine-grained cross-modal semantic alignment, when encoding at least one image sample, the image processing system may first obtain the element classification identifiers corresponding to different image elements in at least one image sample, and on the basis of the element classification identifiers, use a first image encoder to encode at least one image sample to obtain multiple first features of at least one sample image, and through a multi-layer perceptron, use a second image encoder to encode at least one image sample to obtain multiple second features of at least one sample image. Finally, by summarizing the obtained multiple first features and multiple second features, multiple image features corresponding to the image to be processed can be obtained.

[0073] In the embodiment of the present application, encoding at least one image sample by using a first image encoder based on the element classification identifier to obtain multiple first features of at least one image sample includes: extracting features from at least one image sample to obtain multiple initial image features; adding the element classification identifier to the multiple initial image features to obtain multiple element classification features; performing self-attention processing on the multiple element classification features based on the self-attention mechanism to obtain multiple first features.

[0074] In an alternative solution of this embodiment, in order to be able to fully capture different types of image features of at least one image sample, when encoding at least one image sample on the basis of using the element classification identifier, the graphics processing system may first extract features from at least one image sample to obtain the image features corresponding to different elements in at least one image sample, that is, obtain the above-mentioned multiple initial image features, and then add the obtained element classification identifier to the different initial image features to obtain multiple element classification features carrying the element classification identifier. Finally, perform self-attention processing on these multiple element classification features through the self-attention mechanism to obtain the above-mentioned multiple first features. By introducing these multiple first features into the training of the initial image processing model, it can be ensured that the trained target image processing model can more accurately focus on multiple levels of the image to be processed, such as background, elements, details, etc., realize more fine-grained feature extraction and cross-modal alignment, thereby improving the performance of the target image processing model in complex scenarios.

[0075] In the embodiments of the present application, based on a multi-layer perceptron, at least one image sample is encoded by a second image encoder to obtain multiple second features of at least one image sample, including: extracting features from at least one image sample to obtain multiple initial image features; performing feature transformation on the multiple initial image features based on a multi-layer perceptron to obtain multiple perceptual features; performing attention processing on the multiple perceptual features based on an attention mechanism to obtain multiple attention features; and classifying the multiple attention features based on an element classification identifier to obtain multiple second features.

[0076] In an alternative solution of this embodiment, when encoding at least one image sample based on a multi-layer perceptron, the image processing system may first extract features from at least one image sample to obtain multiple initial image features of at least one image sample, and then use the multi-layer perceptron to perform feature transformation on the multiple initial image features to obtain multiple perceptual features, and then perform attention processing on the multiple perceptual features through an attention mechanism to obtain corresponding multiple attention features. Finally, the multiple attention features are classified using the obtained element classification identifier to obtain the above-mentioned multiple second features. By introducing these multiple second features into the training of the initial image processing model, the non-linear expression ability of the trained target image processing model can be enhanced, the diversity of feature transformation can be improved, thereby promoting the alignment and fusion of multi-modal features, and improving the adaptability of the model in complex and diverse tasks.

[0077] In the embodiments of the present application, a feature loss function corresponding to an initial image processing model is constructed based on multiple image features and multiple text features corresponding to multiple description texts, including: mapping the multiple image features to a preset semantic space to obtain multiple first mapped features, and mapping the multiple text features to a preset semantic space to obtain multiple second mapped features; calculating the similarity between the multiple first mapped features and the multiple second mapped features to obtain multiple feature similarities; constructing an initial loss function between different first mapped features and different second mapped features based on the multiple feature similarities; and fusing the initial loss function to obtain a feature loss function.

[0078] In an alternative solution of the embodiment of the present application, in order to accurately construct the above-mentioned feature loss function, the image processing system may first map a plurality of encoded image features into a preset semantic space to obtain corresponding first mapped features, and map a plurality of text features into the preset semantic space as well to obtain corresponding second mapped features, so as to implement the operation of feature alignment. After obtaining the plurality of first mapped features and the plurality of second mapped features, the image processing system may further calculate the similarity between the plurality of first mapped features and the plurality of second mapped features to obtain the similarity between different first mapped features and different second mapped features, that is, the above-mentioned plurality of feature similarities. Through these plurality of feature similarities, the image processing system may construct an initial loss function between different first mapped features and different second mapped features. Finally, by fusing the constructed initial loss function, for example, determining the importance of different elements according to parameters such as the size and quantity of different elements in at least one image sample, then assigning corresponding weights to the image features and text features corresponding to different elements according to the determined importance, and finally performing weighted summation on the constructed initial loss function according to the assigned weights to obtain the above-mentioned feature loss function. Or, in order to improve the efficiency of constructing the feature loss function, the image processing system may also directly perform normalization processing on the constructed initial loss function, and then add the loss function after normalization processing to obtain the corresponding feature loss function. Or, the image processing system may also calculate parameters such as the average value and maximum value of the constructed initial loss function to be used as the above-mentioned feature loss function.

[0079] In the embodiment of the present application, inputting at least one image sample into an initial image processing model, and using the initial image processing model to generate a plurality of description texts, including: obtaining a second prompt word, where the second prompt word is used to prompt the image elements for which corresponding description texts need to be generated; performing semantic segmentation on at least one image sample based on the second prompt word to obtain a plurality of segmentation regions, where different segmentation regions contain different elements; extracting features from the plurality of segmentation regions to obtain a plurality of region features corresponding to different segmentation regions; summarizing the plurality of region features based on the element description identifiers corresponding to different image elements to obtain element features corresponding to different image elements; and generating description texts corresponding to different image elements based on the element features.

[0080] In an alternative solution of this embodiment, when generating the description texts corresponding to different elements in at least one image sample, the image processing system may first obtain the second prompt word input by the user to determine the image elements for which the corresponding description texts need to be generated, and then perform semantic segmentation on the at least one image sample according to the second prompt word to obtain multiple segmentation regions containing different elements. Then, feature extraction is performed on these multiple segmentation regions, and the image processing system can obtain multiple region features corresponding to different segmentation regions. After obtaining the multiple region features, the image processing system can further summarize these multiple region features according to the element description identifiers corresponding to different image elements to obtain the element features corresponding to different image elements, and then input the obtained element features into a preset text generator to generate the description texts corresponding to different image elements.

[0081] In an embodiment of the present application, the above method further includes: encoding the multiple description texts by using a text encoder to obtain multiple embedding vectors; performing self-attention processing on the multiple embedding vectors to obtain multiple self-attention vectors; and performing feed-forward processing on the multiple self-attention vectors to obtain multiple text features.

[0082] In an alternative solution of this embodiment, the image processing system may first encode the multiple description texts by using a text encoder to obtain the vectors corresponding to different description texts, that is, the above multiple embedding vectors, and then perform self-attention processing on these multiple embedding vectors to obtain the corresponding multiple self-attention vectors. Finally, by performing feed-forward processing on the obtained multiple self-attention vectors, the image processing system can obtain the corresponding multiple text features.

[0083] For ease of understanding, Figure 3 is a schematic diagram of an image processing model training framework shown according to an embodiment of the present application. As Figure 3 shown, in order to ensure the processing ability of the trained target image processing model, the initial image processing model can be used to describe the multiple elements included in the image sample to obtain the corresponding multiple description texts, and then, based on the multiple text features corresponding to the multiple description texts, the initial image processing model is trained in combination with the multiple image features of the pattern image sample to obtain the corresponding target image processing model. Based on this, the image processing system first inputs the image sample into the initial image processing model to use the initial image processing model to generate the description texts corresponding to different elements in the image sample, thereby obtaining the above multiple description texts, and encodes these multiple description texts by using a text encoder to obtain the corresponding multiple text features. At the same time, the image processing system can first perform image encoding on the image sample to obtain the corresponding multiple image features. In order to ensure the integrity of the encoded multiple image features, the image processing system can use the element classification identifiers corresponding to different image elements in the image sample, that is, Figure 3Based on the CLS token, the first image encoder is used to encode the image sample to obtain corresponding multiple first features. At the same time, based on the multi-layer perceptron, the second image encoder is used to encode the image sample to obtain corresponding multiple second features. Then, by summarizing the multiple first features and the multiple second features, the corresponding multiple image features can be obtained. In order to fully extract the image features contained in the image sample, the image processing system can apply the attention mechanism to the last three layers of the second image encoder. When using the second image encoder to encode the image sample, the image processing system can first use the attention mechanism to perform attention processing on the multiple perceptual features obtained by the feature transformation of the multi-layer perceptron to obtain corresponding multiple attention features, and then sequentially use the linear layer and the activation layer, as well as the multi-layer linear layer to process the multiple attention features to obtain sufficient and accurate multiple second features.

[0084] Figure 4 is a schematic diagram of the training and application process of an image processing model shown according to an embodiment of the present application, as Figure 4As shown, in the training stage, the image processing system can first obtain image samples, using an image sample showing a pet dog in an outdoor scene, and use a text generator, such as a vision-language model deployed in the initial image processing model, to generate multi-perspective and multi-granularity descriptive texts for the image samples to fully display the element information of different elements in the image samples, such as information on the pet dog, outdoor scene, background, style features, detailed descriptions, etc., thus forming a "multi-text - multi-image" paired training set containing rich semantic information. At the same time, the image processing system can perform image encoding on the image samples through an image encoder to generate multi-branch embedding representations, that is, generate multiple image features, with different branches focusing on different features of the image, such as background, subject, details, etc. During the process of image encoding, a multi-layer attention mechanism can be used in the image encoder to ensure that each embedding branch can focus on the key information points of the image and text. While performing image encoding, the image processing system can use a text encoder to encode the multiple descriptive texts generated above to convert the multiple descriptive texts into cross-modal embedding representations and obtain corresponding multiple text features. After obtaining the multiple text features and multiple text features, the image processing system can embed the image features and text features into a preset space to obtain corresponding first mapping features and second mapping features, and perform forward propagation on each text feature during the process of embedding the text features to enhance the interpretability of the embedded second mapping features. After embedding to obtain the first mapping features and second mapping features, the image processing system can calculate the similarity between these two mapping features to construct a corresponding feature loss function, and adjust the model parameters of the initial image processing model by minimizing the loss function to ensure the maximum semantic similarity between the image and text, thus achieving a more accurate cross-modal semantic pair, and thereby training a target image processing model with better processing capabilities.

[0085] In the application stage, the image processing system can first determine the elements to be processed in the image to be processed according to the prompt words input by the user, and then, based on the elements to be processed, input the image to be processed into the target image processing model to process the image to be processed by using the target image processing model, so as to obtain an accurate image processing result. For example, if the image to be processed shows a flower and the prompt words input by the user are "What kind of flower is this", the image processing system can first perform text encoding on the prompt words to obtain the corresponding prompt word features, and at the same time perform image encoding on the image to be processed to obtain the corresponding image features to be processed, and then match the image features to be processed and the prompt word features to determine information such as the elements to be processed and the processing method, that is, determine that the current element to be processed is the "flower" shown in the image to be processed. The corresponding image processing system can screen out the target features related to the "flower" from the image features to be processed, and based on the screened target features, generate the corresponding description text to reply to the prompt words input by the user, thereby improving the user experience.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0087] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0088] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of this application.

[0089] According to an embodiment of the present application, a product recommendation method is further provided. Figure 5 It is a flowchart of a product recommendation method shown according to an embodiment of the present application. As Figure 5 shown, the method may include the following steps:

[0090] Step S502, in response to a product recommendation instruction, determine at least one product to be recommended and product recommendation elements corresponding to the product recommendation instruction.

[0091] Among them, the product recommendation elements are used to represent product elements that need to be processed in at least one product to be recommended.

[0092] Step S504, input the product images of at least one product to be recommended into the product recommendation model, and use the product recommendation model to select a target recommended product from at least one product to be recommended according to the product recommendation elements.

[0093] Among them, the product recommendation model is obtained by training an initial recommendation model through at least one product sample, as well as multiple product features and multiple product texts corresponding to at least one product sample. The multiple product features are used to represent the features of multiple product elements included in at least one product sample, and the multiple product texts are used to represent texts generated by using the initial recommendation model to describe the multiple product elements.

[0094] Step S506, output the target recommended product.

[0095] In an alternative solution of this embodiment, when receiving a product recommendation instruction, the product recommendation system may first determine at least one product to be recommended and product recommendation elements corresponding to the product recommendation instruction, that is, product elements that need to be processed in at least one product to be recommended, and then input the product images of at least one product to be recommended into the product recommendation model, so as to use the product recommendation model to select a target recommended product from at least one product to be recommended according to the product recommendation elements, and input the target recommended product in a preset operation interface for the user to view conveniently.

[0096] In an alternative solution of this embodiment, in order to ensure the accuracy of the recommended target recommended product, the above product recommendation model may be a model obtained by training an initial recommendation model through at least one product sample, as well as multiple product features and multiple product texts corresponding to at least one product sample. The corresponding multiple product features may refer to the features of multiple product elements included in at least one product sample, and the multiple product texts may refer to texts generated by using the initial recommendation model to describe the multiple product elements.

[0097] It should be noted that the above process of product recommendation may refer to the previous image processing process and will not be elaborated here.

[0098] According to an embodiment of the present application, another image processing method is also provided. Figure 6 It is a flowchart of another image processing method shown according to an embodiment of the present application. As Figure 6 shown, the method may include the following steps:

[0099] Step S602, obtain the image to be processed and the element to be processed corresponding to the image processing instruction by calling the first interface.

[0100] Among them, the first interface includes a first parameter, and the parameter value of the first parameter includes the image to be processed and the element to be processed. The element to be processed is used to represent the image element that needs to be processed in the image to be processed;

[0101] Step S604, input the image to be processed into the target image processing model, and use the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result.

[0102] Among them, the target image processing model is obtained by training an initial image processing model through at least one image sample, as well as a plurality of image features and a plurality of description texts corresponding to at least one image sample. The plurality of image features are used to represent the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to represent the texts for describing the plurality of image elements generated by using the initial image processing model. The plurality of image elements include the element to be processed.

[0103] Step S606, output the image processing result by calling the second interface.

[0104] Among them, the second interface includes a second parameter, and the parameter value of the second parameter includes the image processing result.

[0105] In an alternative solution of this embodiment, when performing image processing, the image processing system may first obtain the first parameter by calling the first interface, that is, obtain the image to be processed and the element to be processed corresponding to the image processing instruction, so as to determine the image element that needs to be processed in the image to be processed, and then input the image to be processed into the target image processing model, so as to use the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result, and finally output the second parameter by calling the second interface, that is, output the image processing result.

[0106] In an alternative solution of this embodiment, to ensure the accuracy of processing the image to be processed, the above-mentioned target image processing model can be a model obtained by training an initial image processing model with at least one image sample, as well as multiple image features and multiple description texts corresponding to at least one image sample. The above-mentioned multiple image features can refer to the features of multiple image elements included in at least one image sample, and the multiple description texts can refer to the texts generated by using the initial image processing model to describe the multiple image elements. The multiple image elements include the elements to be processed.

[0107] It should be noted that the above-mentioned image processing process can be referred to as shown in the previous text and will not be elaborated here.

[0108] According to an embodiment of the present application, there is also provided a method for training an image processing model. Figure 7 It is a flowchart of a method for training an image processing model shown according to an embodiment of the present application, as Figure 7 shown, and this method can include the following steps:

[0109] Step S702, obtain training data.

[0110] Step S704, use the target training strategy and training data to adjust the pre-trained model to obtain an image processing model. Among them, the image processing model includes: a target image processing model and a commodity recommendation model.

[0111] Step S706, deploy the image processing model to an execution device.

[0112] Among them, the image processing model is called by the execution device to execute the method of any one of the above.

[0113] In an alternative solution of this embodiment, when training multiple models mentioned in the present application, such as the target image processing model and the commodity recommendation model, the training system can first obtain training data, and then adjust the pre-trained model according to the corresponding target training strategy and training data to obtain the corresponding image processing model, that is, obtain the above-mentioned target image processing model and commodity recommendation model, and then deploy the image processing model to the execution device for the convenience of users. It should be noted that the above-mentioned target training strategy and training data can be referred to as shown in the previous text and will not be elaborated here.

[0114] According to an embodiment of the present application, there is also provided an image processing device for implementing the above-mentioned image processing method. Figure 8 It is a structural block diagram of an image processing device shown according to an embodiment of the present application, as Figure 8 shown, and this device includes: a first determination module 802, a first processing module 804, and a first output module 806.

[0115] Among them, the first determination module 802 is configured to determine a to-be-processed image and a to-be-processed element corresponding to the image processing instruction in response to receiving the image processing instruction, where the to-be-processed element is used to characterize the image element in the to-be-processed image that needs to be processed; the first processing module 804 is configured to input the to-be-processed image into the target image processing model, and use the target image processing model to process the to-be-processed image according to the to-be-processed element to generate an image processing result, where the target image processing model is obtained by training an initial image processing model through at least one image sample, as well as a plurality of image features and a plurality of description texts corresponding to at least one image sample, the plurality of image features are used to characterize the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to characterize the texts for describing the plurality of image elements generated by using the initial image processing model; the first output module 806 is configured to output the image processing result.

[0116] It should be noted here that the above-mentioned first determination module 802, first processing module 804, and first output module 806 correspond to steps S202 to S206 in the above embodiment. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above-mentioned module or unit can be a hardware component or a software component stored in the memory and processed by one or more processors, and the above-mentioned module can also be a part of the device and can run in the server 10 provided in the above embodiment.

[0117] According to an embodiment of the present application, there is also provided a product recommendation device for implementing the above product recommendation method, Figure 9 which is a structural block diagram of a product recommendation device shown according to an embodiment of the present application, as Figure 9 shown. The device includes: a second determination module 902, a second processing module 904, and a second output module 906.

[0118] Among them, the second determination module 902 is configured to determine at least one product to be recommended and product recommendation elements corresponding to the product recommendation instruction in response to the product recommendation instruction, where the product recommendation elements are used to represent product elements that need to be processed among the at least one product to be recommended; the second processing module 904 is configured to input product images of the at least one product to be recommended into the product recommendation model, and use the product recommendation model to select a target recommended product from the at least one product to be recommended according to the product recommendation elements, where the product recommendation model is obtained by training an initial recommendation model through at least one product sample, as well as a plurality of product features and a plurality of product texts corresponding to the at least one product sample, the plurality of product features are used to represent the features of a plurality of product elements included in the at least one product sample, and the plurality of product texts are used to represent texts generated by the initial recommendation model for describing the plurality of product elements; the second output module 906 is configured to output the target recommended product.

[0119] It should be noted here that the above-mentioned second determination module 902, second processing module 904, and second output module 906 correspond to steps S502 to S506 in the above embodiment. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be part of a device and can run in the server 10 provided in the above embodiment.

[0120] According to an embodiment of the present application, there is also provided another image processing device for implementing the above image processing method. Figure 10 It is a structural block diagram of another image processing device shown according to an embodiment of the present application, as Figure 10 shown. The device includes: a first call module 1002, a third processing module 1004, and a second call module 1006.

[0121] Among them, the first calling module 1002 is used to obtain the image to be processed and the element to be processed corresponding to the image processing instruction by calling the first interface. The first interface includes a first parameter, and the parameter value of the first parameter includes the image to be processed and the element to be processed. The element to be processed is used to represent the image element in the image to be processed that needs to be processed. The third processing module 1004 is used to input the image to be processed into the target image processing model, and use the target image processing model to process the image to be processed according to the element to be processed, and generate an image processing result. The target image processing model is obtained by training an initial image processing model through at least one image sample, as well as a plurality of image features and a plurality of description texts corresponding to at least one image sample. The plurality of image features are used to represent the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to represent the texts for describing the plurality of image elements generated by using the initial image processing model. The second calling module 1006 is used to output the image processing result by calling the second interface. The second interface includes a second parameter, and the parameter value of the second parameter includes the image processing result.

[0122] It should be noted here that the above first calling module 1002, third processing module 1004, and second calling module 1006 correspond to steps S602 to S606 in the above embodiment. The instances and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be part of a device and can run in the server 10 provided in the above embodiment.

[0123] According to an embodiment of the present application, there is also provided an image processing model training device for implementing the above image processing model training method. Figure 11 It is a structural block diagram of an image processing model training device shown according to an embodiment of the present application, as Figure 11 shown. The device includes: a data acquisition module 1102, a model adjustment module 1104, and a model deployment module 1106.

[0124] Among them, the data acquisition module 1102 is used to acquire training data; the model adjustment module 1104 is used to adjust a pre-trained model by using a target training strategy and training data to obtain an image processing model, where the image processing model includes: a target image processing model and a commodity recommendation model; the model deployment module 1106 is used to deploy the image processing model to an execution device, where the image processing model is called by the execution device to perform the method of any one of the above.

[0125] It should be noted here that the above data acquisition module 1102, model adjustment module 1104, and model deployment module 1106 correspond to steps S702 to S706 in the above embodiments. The instances and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of a device and can run in the server 10 provided in the above embodiments.

[0126] According to an embodiment of the present application, there is also provided an image processing system. Figure 12 It is a structural block diagram of an image processing system shown according to an embodiment of the present application. As Figure 12 shown, it includes: a client 1202 and a server 1204.

[0127] Among them, the client 1202 is used to send an image processing instruction; the server 1204 is connected to the client and is used to determine the image to be processed and the element to be processed corresponding to the image processing instruction. Among them, the element to be processed is used to represent the image element that needs to be processed in the image to be processed; input the image to be processed into the target image processing model, and use the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result. Among them, the target image processing model is obtained by training an initial image processing model through at least one image sample, as well as a plurality of image features and a plurality of description texts corresponding to at least one image sample. The plurality of image features are used to represent the features of a plurality of image elements included in at least one image sample, and the plurality of description texts are used to represent the texts generated by using the initial image processing model to describe the plurality of image elements; the client 1202 is also used to output the image processing result.

[0128] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in the above embodiments, but are not limited to the schemes provided in the above embodiments.

[0129] An embodiment of the present application may provide a computing device. Figure 13 It is a structural block diagram of a computing device shown according to an embodiment of the present application. As Figure 13 shown, the computing device 1300 may include: one or more (only one is shown in the figure) processors 1302, a memory 1304, a storage controller, and a peripheral interface.

[0130] The above-mentioned computing device can be understood as an integrated intelligent terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), model all-in-ones, etc. Moreover, the models in the above-mentioned embodiments of the present application can be pre-installed in the computing device.

[0131] Specifically, the computing device can pre-install various types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multi-modal task processing, etc., so as to provide diverse model selections. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of various types of models such as discriminative and generative models), model version control (supporting the control of different model versions), model evaluation (evaluating the performance and effect of the model based on model evaluation tools), etc. In other product forms, the computing device can also create applications based on the model, provide API invocation capabilities, and can call the model into the created application through the API interface, and at the same time provide application management tools to realize the control of the application.

[0132] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning data sets), a training center (providing rich training resources to help users learn and master AI technologies), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive and integrated AI development, training, deployment, and application device is provided.

[0133] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the methods in the above-mentioned embodiments. The memory can include high-speed random access memory, and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory can further include memories remotely set relative to the processor, and these remote memories can be connected to terminal A through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranets, local area networks, mobile communication networks, and their combinations.

[0134] The processor can call the executable program stored in the memory through the transmission device to execute the method in any one of the above-mentioned embodiments.

[0135] Embodiments of the present application can provide an electronic device.Figure 14 It is a structural block diagram of an electronic device shown according to an embodiment of the present application. As shown in the figure, the electronic device may include: an input / output device 1402; a memory 1404 and a processor 1406, where the processor 1406 is connected to the input / output device 1402 and the memory 1404 through a bus 1408.

[0136] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely provided with respect to the processor, and these remote memories can be connected to the terminal A through a network. Examples of the above networks include, but are not limited to, the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] The processor can call the executable program stored in the memory through a transmission device to execute the method of any one of the above embodiments.

[0138] Those of ordinary skill in the art can understand that Figure 14 the structure shown is only schematic, and the computing device can also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a Mobile Internet Device (MID), a PAD, etc. The Figure 14 It does not limit the structure of the above computing device. For example, the computing device 100 may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that Figure 14 shown.

[0139] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware of the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc, etc.

[0140] The embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the above computer-readable storage medium can be used to save the program code executed by the method provided in the above embodiment.

[0141] Optionally, in this embodiment, the above storage medium may be located in a computing device.

[0142] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, and when the executable program runs, it controls the device where the computer-readable storage medium is located to execute the method according to any one of the above embodiments.

[0143] The embodiment of the present application also provides a computer program product. Optionally, in this embodiment, the above computer program product may include a computer program, and when the computer program is executed by a processor, it implements the method provided by the above embodiment.

[0144] The embodiment of the present application also provides a computer program product. Optionally, the above computer program product may include a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium may be used to store a computer program, and when the computer program is executed by a processor, it implements the method provided by the above embodiment.

[0145] The embodiment of the present application also provides a computer program. Optionally, in this embodiment, when the above computer program is executed by a processor, it implements the method provided by the above embodiment.

[0146] In the above embodiments of the present application, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0147] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of units or modules can be in electrical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0151] The above is only the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: In response to receiving an image processing instruction, determining an image to be processed and an element to be processed corresponding to the image processing instruction, wherein the element to be processed is used to represent an image element that needs to be processed in the image to be processed; Inputting the image to be processed into a target image processing model, and using the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result, wherein the target image processing model is obtained by training an initial image processing model through at least one image sample, and a plurality of image features and a plurality of description texts corresponding to the at least one image sample, the plurality of image features being used to characterize the features of a plurality of image elements contained in the at least one image sample, and the plurality of description texts being used to characterize the texts generated by using the initial image processing model and describing the plurality of image elements; The image processing result is outputted.

2. The method according to claim 1, characterized in that: The image processing result includes: image description text corresponding to the element to be processed; inputting the image to be processed into a target image processing model, and using the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result, including: Get the first prompt word; Encoding the image to be processed according to the element to be processed by using an image encoder to obtain image coding features corresponding to the element to be processed; Performing feature extraction on the first prompt word to obtain description text parameters; The image description text is generated based on the first image coding feature and the description text parameter.

3. The method according to claim 1, characterized in that The step of determining, in response to receiving the image processing instruction, the to-be-processed element corresponding to the image processing instruction comprises: Obtaining the first prompt word and the instruction type corresponding to the image processing instruction; The element to be processed is determined based on the instruction type and the first prompt word.

4. The method according to claim 1, characterized in that: The method further comprises: Encoding the at least one image sample to obtain the plurality of image features; Inputting the at least one image sample into an initial image processing model, and generating the plurality of description texts using the initial image processing model; Based on the multiple image features and the multiple text features corresponding to the multiple description texts, construct a feature loss function corresponding to the initial image processing model, wherein the feature loss function is used to characterize the similarity between different image features and different text features; The model parameters of the initial image processing model are adjusted based on the feature loss function to obtain a target image processing model.

5. The method according to claim 4, characterized in that The encoding of the at least one image sample to obtain the plurality of image features includes: Obtaining element classification identifiers corresponding to different image elements in the at least one image sample; Based on the element classification identifier, using a first image encoder to encode the at least one image sample to obtain a plurality of first features of the at least one image sample; Based on a multi-layer perceptron, using a second image encoder to encode the at least one image sample to obtain a plurality of second features of the at least one image sample; The multiple first features and the multiple second features are aggregated to obtain the multiple image features.

6. The method according to claim 5, characterized in that The step of encoding the at least one image sample by using a first image encoder based on the element classification identifier to obtain a plurality of first features of the at least one image sample includes: Performing feature extraction on the at least one image sample to obtain a plurality of initial image features; Adding the element classification identifier to the multiple initial image features to obtain multiple element classification features; Based on the self-attention mechanism, self-attention processing is performed on the multiple element classification features to obtain the multiple first features.

7. The method according to claim 5, characterized in that The method of encoding the at least one image sample by using a second image encoder based on a multi-layer perceptron to obtain a plurality of second features of the at least one image sample comprises: Performing feature extraction on the at least one image sample to obtain a plurality of initial image features; Performing feature transformation on the multiple initial image features based on the multi-layer perceptron to obtain multiple perceptual features; Performing attention processing on the multiple perception features based on an attention mechanism to obtain multiple attention features; The multiple attention features are classified based on the element classification identifier to obtain the multiple second features.

8. The method according to claim 4, characterized in that The step of constructing a feature loss function corresponding to the initial image processing model based on the multiple image features and the multiple text features corresponding to the multiple description texts includes: Mapping the plurality of image features to a preset semantic space to obtain a plurality of first mapping features, and mapping the plurality of text features to the preset semantic space to obtain a plurality of second mapping features; Calculating similarities between the plurality of first mapping features and the plurality of second mapping features to obtain a plurality of feature similarities; Based on the multiple feature similarities, constructing an initial loss function between different first mapping features and different second mapping features; The initial loss function is fused to obtain the feature loss function.

9. The method according to claim 4, characterized in that The step of inputting the at least one image sample into an initial image processing model and using the initial image processing model to generate the plurality of description texts comprises: Acquire a second prompt word, wherein the second prompt word is used to prompt an image element for which a corresponding description text needs to be generated; Performing semantic segmentation on the at least one image sample based on the second prompt word to obtain a plurality of segmented regions, wherein different segmented regions contain different elements; Performing feature extraction on the multiple segmented regions to obtain multiple region features corresponding to the different segmented regions; Summarizing the plurality of regional features based on element description identifiers corresponding to different image elements to obtain element features corresponding to different image elements; Based on the element features, description texts corresponding to different image elements are generated.

10. The method according to any one of claims 4, characterized in that: The method further comprises: Encoding the multiple description texts using a text encoder to obtain multiple embedding vectors; Performing self-attention processing on the multiple embedding vectors to obtain multiple self-attention vectors; The multiple self-attention vectors are feed-forward processed to obtain the multiple text features.

11. A product recommendation method, characterized in that: include: In response to a product recommendation instruction, determining at least one product to be recommended and a product recommendation element corresponding to the product recommendation instruction, wherein the product recommendation element is used to represent a product element that needs to be processed in the at least one product to be recommended; Inputting the product image of the at least one product to be recommended into a product recommendation model, and selecting a target recommended product from the at least one product to be recommended according to the product recommendation element using the product recommendation model, wherein the product recommendation model is obtained by training an initial recommendation model using at least one product sample, and multiple product features and multiple product texts corresponding to the at least one product sample, wherein the multiple product features are used to characterize the features of multiple product elements contained in the at least one product sample, and the multiple product texts are used to characterize texts generated using the initial recommendation model and describing the multiple product elements; The target recommended product is output.

12. An image processing method, characterized in that: include: Acquire the image to be processed and the element to be processed corresponding to the image processing instruction by calling the first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the image to be processed and the element to be processed, and the element to be processed is used to represent the image element to be processed in the image to be processed; Inputting the image to be processed into a target image processing model, and using the target image processing model to process the image to be processed according to the element to be processed to generate an image processing result, wherein the target image processing model is obtained by training an initial image processing model through at least one image sample, and a plurality of image features and a plurality of description texts corresponding to the at least one image sample, the plurality of image features being used to characterize the features of a plurality of image elements contained in the at least one image sample, and the plurality of description texts being used to characterize the texts generated by using the initial image processing model and describing the plurality of image elements; The image processing result is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the image processing result.

13. A training method for an image processing model, characterized in that: include: Get training data; The pre-trained model is adjusted using the target training strategy and the training data to obtain an image processing model, wherein the image processing model includes: a target image processing model and a product recommendation model; The image processing model is deployed to an execution device, wherein the image processing model is called by the execution device to execute the method described in any one of claims 1 to 12.

14. An image processing system, characterized in that: include: Client, used to send image processing instructions; A server is connected to the client and is used to determine an image to be processed and an element to be processed corresponding to the image processing instruction, wherein the element to be processed is used to characterize the image element to be processed in the image to be processed; the image to be processed is input into a target image processing model, and the image to be processed is processed according to the element to be processed by using the target image processing model to generate an image processing result, wherein the target image processing model is obtained by training an initial image processing model through at least one image sample, and multiple image features and multiple description texts corresponding to the at least one image sample, the multiple image features are used to characterize the features of multiple image elements contained in the at least one image sample, and the multiple description texts are used to characterize the text generated by using the initial image processing model to describe the multiple image elements; The client is also used to output the image processing result.

15. A computing device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 12 when running.

16. An electronic device, characterized in that: include: A memory storing an executable program; A processor, connected to the memory via a bus, and configured to run the program, wherein the program executes the method described in any one of claims 1 to 12 when running.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 12.

18. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images