Image processing method and apparatus, device, medium and program product

Abstract

Disclosed in embodiments of the present application are an image processing method and apparatus, a device and a medium. The method comprises: acquiring a target image to be processed; inputting the target image into a pre-trained image-text model, wherein model loss of the image-text model comprises image loss, and the image loss is constructed on the basis of a first sample image and a second sample image obtained by converting a first sample text used for describing the first sample image; and acquiring a target text which is generated by the image-text model and is used for describing the target image. According to the technical solution of the embodiments of the present application, the generated target text can accurately describe the target image as much as possible, and the accuracy of the target text is ensured.

Description

Image processing method, device, equipment, medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 20, 2023, with application number 2023108940974 and application name “Image processing method, device, equipment and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of video and image processing technology, and in particular, to image processing. Background Art

[0003] In the field of images, the commonly used basic methods for generating image description text include question-and-answer format and picture description format. However, the description text generated by the question-and-answer format is prone to oversimplification or omission of key information, resulting in a large difference between the description text and the original image content. In addition, the description text generated by the picture description format is easily interfered by irrelevant or minor content in the image, which also leads to a large difference between the generated text and the image content.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, an image processing device, an electronic device, a computer-readable storage medium, and a computer program product, which can enable the generated target text to describe the target image as accurately as possible, thereby ensuring the accuracy of the target text.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0007] According to one aspect of an embodiment of the present application, an image processing method is provided, comprising: obtaining a target image to be processed; inputting the target image into a pre-trained graph-text model, wherein the model loss of the graph-text model includes an image loss, and the image loss is constructed based on a first sample image and a second sample image converted from a first sample text for describing the first sample image; and obtaining a target text generated by the graph-text model for describing the target image.

[0008] According to one aspect of an embodiment of the present application, an image processing device is provided, including: an acquisition module for acquiring a target image to be processed; an input module for inputting the target image into a pre-trained graph-text model, the model loss of the graph-text model including an image loss, the image loss being constructed based on a first sample image and a second sample image converted from a first sample text for describing the first sample image; the acquisition module is further used to acquire a target text generated by the graph-text model for describing the image content of the target image.

[0009] According to one aspect of an embodiment of the present application, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the image processing method described above.

[0010] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the image processing method as described above.

[0011] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the image processing method as described above.

[0012] In the technical solution provided in the embodiments of the present application, the target image to be processed is input into the graph-text model. Since the model loss of the graph-text model includes image loss, and the image loss is constructed based on the first sample image and the second sample image converted from the first sample text used to describe the first sample image, the second sample image that has undergone graph-to-text and text-to-graph can better reflect the consistency of the image content in the conversion process. Therefore, the image loss constructed based on the first sample image in the graph-to-text stage and the second sample image in the text-to-graph stage enables the graph-text model obtained through image loss training to improve the loss of content information in the conversion process and ensure the consistency of image content. Therefore, the target text generated by the graph-text model can describe the target image as much as possible, thereby ensuring the accuracy of the target text.

[0013] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a schematic diagram of an implementation environment involved in this application;

[0015] FIG2 is a flow chart of an image processing method shown in an exemplary embodiment of the present application;

[0016] FIG3 is a flow chart of another image processing method shown in an exemplary embodiment of the present application;

[0017] FIG4 is a flow chart of another image processing method shown in an exemplary embodiment of the present application;

[0018] FIG5 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0019] FIG6 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0020] FIG7 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0021] FIG8 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0022] FIG9 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0023] FIG10 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0024] FIG11 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0025] FIG12 is a flowchart of an image processing method shown in an exemplary embodiment of the present application;

[0026] FIG13 is a flowchart of another image processing method shown in an exemplary embodiment of the present application;

[0027] FIG14 is a schematic diagram of the structure of a model to be trained shown in an exemplary embodiment of the present application;

[0028] FIG15 is a processing flow chart of a reverse module according to another exemplary embodiment of the present application;

[0029] FIG16 is a schematic diagram of forward diffusion and reverse diffusion according to an exemplary embodiment of the present application;

[0030] FIG17 is a schematic diagram of a denoising process according to an exemplary embodiment of the present application;

[0031] FIG18 is a processing flow chart of a caption module according to another exemplary embodiment of the present application;

[0032] FIG19 is a structural block diagram of an image processing apparatus shown in an exemplary embodiment of the present application;

[0033] FIG20 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0037] It should also be noted that the term "plurality" used in this application refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0038] The technical solutions of the embodiments of the present application relate to the field of artificial intelligence (AI) technology. Before introducing the technical solutions of the embodiments of the present application, a brief introduction to AI technology is given. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. AI is also about studying the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0039] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of AI and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0040] The technical solution of the embodiment of the present application specifically relates to machine learning technology in AI, specifically, a graphic model is obtained by pre-training based on machine learning technology, thereby realizing image-to-text conversion. The technical solution of the embodiment of the present application is described in detail below:

[0041] Please refer to FIG1 , which is a schematic diagram of an implementation environment involved in this application. The implementation environment includes a terminal 10 and a server 20 .

[0042] The terminal 10 is used to send the target image to be processed to the server 20 .

[0043] The server 20 is used to input the target image into a pre-trained graph-text model, and then obtain the target text generated by the graph-text model to describe the target image, wherein the model loss of the graph-text model includes image loss, which is constructed based on the first sample image and the second sample image converted from the first sample text used to describe the first sample image.

[0044] In some embodiments, the server 20 may send the target text to the terminal 10, and the terminal 10 may perform downstream tasks based on the target text, such as training a downstream model, or generating a new image based on the target text.

[0045] In some embodiments, the server 20 may also obtain the target image to be processed by itself, and then input it into the graphic model, and then obtain the target text generated after the graphic model processes the target image, so as to perform subsequent processing based on the target text.

[0046] In some embodiments, the terminal 10 may also independently implement image processing, that is, the terminal 10 obtains a target image to be processed, and then inputs it into the graphic model, and then obtains a target text generated after the graphic model processes the target image.

[0047] Among them, the aforementioned terminal 10 can be any electronic device that can obtain target video and images to be processed, such as a smart phone, tablet, laptop, computer, intelligent voice interaction device, smart home appliance, vehicle-mounted terminal, aircraft, etc. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms. This is not restricted here.

[0048] The terminal 10 and the server 20 establish a communication connection in advance through a network, so that the terminal 10 and the server 20 can communicate with each other through the network. The network can be a wired network or a wireless network, and there is no limitation here.

[0049] It should be noted that the embodiments of the present application can perform image-text conversion on various images and can be applied to various scenarios, including but not limited to images in various scenarios such as cloud technology, AI (Artificial Intelligence), smart transportation, assisted driving, etc., or it can also be used to convert images in image processing applications.

[0050] Specifically, if the technical solution of the embodiment of the present application is applied in a smart transportation scenario, the terminal can be a vehicle-mounted terminal. The vehicle-mounted terminal uses the image taken by the driving recorder as the target image to be processed, inputs the target image to be processed into the graphic model, and then obtains the target text describing the target image generated by the graphic model, and then plays the target text through the player, so that even if the driver does not observe relevant events on the road, he can still obtain various information on the road.

[0051] For example, the technical solution of the embodiment of the present application is applied to an image processing application. The server can be an image processing server. For example, the image processing server obtains the target image to be processed uploaded by the object, inputs the target image into the graphic model, and then obtains the target text generated by the graphic model to describe the target image. The server performs downstream tasks based on the target text, such as training a language model.

[0052] It should be noted that in the specific implementation of this application, the target image is object-related. When the embodiment of this application is applied to a specific product or technology, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0053] The following describes in detail the various implementation details of the technical solutions of the embodiments of the present application:

[0054] As shown in FIG2 , FIG2 is a flowchart of an image processing method according to an embodiment of the present application. The method can be applied to the implementation environment shown in FIG1 . The method can be executed by a computer device, such as the aforementioned terminal or server. Specifically, the method can be executed by the terminal or the server, or by both the terminal and the server. In the embodiment of the present application, the method is executed by the server as an example. The image processing method may include S210 to S230, which are described in detail as follows:

[0055] S210: Acquire a target image to be processed.

[0056] In the embodiment of the present application, the target image to be processed can be any image with image content, such as a landscape painting, a portrait, etc.

[0057] In one example, the target image to be processed may be a frame image in a video, that is, the process of obtaining the target image to be processed is: reading each frame image in the video at a specified frame rate or time interval, and randomly selecting a frame or a key frame from the read frame images as the target image.

[0058] In one example, the target image to be processed may be transmitted from other devices, such as a receiving image acquisition device; the target image may also be downloaded directly from the network or uploaded by an object.

[0059] It should be noted that in the specific implementation of this application, the target image obtained involves information related to the object. When the embodiments of this application are applied to specific products or technologies, any one of them needs to obtain the object's permission or consent separately, and the collection, use and processing of relevant object information need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0060] For example, if the target image is a portrait, before obtaining and processing the portrait, the information processing rules corresponding to the object contained in the portrait will be informed, such as the face recognition, face feature extraction and other processing rules of the object, and the separate consent of the target object will be sought. The relevant information will be processed in strict accordance with the requirements of laws and regulations and the personal information processing rules, and technical measures will be taken to ensure the security of relevant data.

[0061] S220. Input the target image into a pre-trained graph-text model. The model loss of the graph-text model includes image loss. The image loss is constructed based on the first sample image and the second sample image converted from the first sample text used to describe the first sample image.

[0062] In an embodiment of the present application, the image-text model is a trained model used to perform image-text conversion on images, that is, converting images into text, such as converting an image of traffic on the road taken by a camera into a text message such as "vehicles are constantly moving on the road". Therefore, after the target image is input into the image-text model, the target text used to describe the image content of the target image can be obtained.

[0063] It is worth noting that the model loss of the graph-text model includes image loss. Therefore, during the training phase, the graph-text model can be trained based on the image loss. The image loss is constructed based on the first sample image and the second sample image converted from the first sample text used to describe the first sample image. For example, there is a first sample image A1 and a first sample text B1 used to describe the first sample image A1. The first sample text B1 is converted to obtain the second sample image A2. The image loss is constructed based on the first sample image A1 and the second sample image A2. In one example, the first sample text B1 can be converted from the first sample image A1.

[0064] As described above, the first sample text is used to describe the first sample image, and the content expressed by the first sample text and the first sample image is the same, but the second sample image is obtained by converting the first sample text, wherein conversion between image and text is performed, so that the image content of the first sample image and the image content of the second sample image may be the same or different, and then the image content of the first sample image and the image content of the second sample image can be used to construct image loss to characterize the consistency of content information in the conversion process, and then the image-text model obtained through image loss training can improve the loss of content information in the conversion process, so that after the image-text conversion, the text generated by the image-text model can describe the image as accurately as possible.

[0065] S230: Obtain target text generated by the image-text model and used to describe the target image.

[0066] In an embodiment of the present application, after the target image is input into the graph-text model, the graph-text model can be directly obtained to perform graph-text conversion on the target image to obtain the target text. Since the graph-text model is constructed based on image loss, the image content contained in the target image will not be lost, and the meaning is the same as that expressed by the target text, thereby achieving the most accurate description of the image content with text.

[0067] In an embodiment of the present application, the target image to be processed is input into the graph-text model. Since the model loss of the graph-text model includes image loss, and the image loss is constructed based on the first sample image and the second sample image converted from the first sample text used to describe the first sample image, the second sample image that has undergone image-to-text and text-to-image conversion can better reflect the consistency of the image content in the conversion process. Therefore, the image loss constructed based on the first sample image and the second sample image enables the graph-text model obtained through image loss training to improve the loss of content information in the conversion process and ensure the consistency of image content. Therefore, the target text generated by the graph-text model can describe the target image as much as possible, thereby ensuring the accuracy of the target text.

[0068] In one embodiment of the present application, another image processing method is provided. The image processing method can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or can be executed by both the terminal and the server. In the embodiment of the present application, the method is described by taking the method executed by the server as an example. As shown in FIG3 , the image processing method is based on S210 to S230 shown in FIG2 , and adds a training process of an image-text model before S220 in FIG2 , including steps S310 to S350. S310 to S350 are described in detail as follows:

[0069] S310: Obtain a model to be trained.

[0070] In an embodiment of the present application, the model to be trained may be a complete neural network model of a certain type, or it may be a neural network model composed of a plurality of neural networks of different types, which is not limited here.

[0071] S320: Acquire an initial sample text for describing the image content, and generate a first sample image for describing the initial sample text based on the model to be trained and the initial sample text.

[0072] In an embodiment of the present application, an initial sample text is obtained, which is used to describe the image content. The image content can be any legal and compliant content and is not limited here. The acquisition method can be extracted from the network or uploaded by the object.

[0073] After the initial sample text is obtained, a first sample image is generated according to the initial sample text. The first sample image is used to describe the initial sample text, that is, to express the content of the text in the form of an image.

[0074] In one example, the initial sample text can be input into the model to be trained, which includes a network structure that can realize a text-generated graph. The model to be trained can then generate an image that matches the description from the text description, and then generate a first sample image, for example, converting the text into an image space, and at the same time linking visual features with language information to achieve mapping between text and image.

[0075] S330 : Based on the model to be trained, generate a first sample text according to the first sample image, and generate a second sample image for describing the first sample text according to the first sample text.

[0076] In one example, after obtaining the first sample image, the first sample image can be input into the model to be trained. The training model includes a network structure that can realize image-to-text generation. Then, the first sample text can be generated by the model to be trained, and the first sample image can be described by the first sample text.

[0077] In an embodiment of the present application, a second sample image is generated based on a first sample text, and the first sample text is described by the second sample image. It should be noted that, in order to better construct image loss later, the generation method of the first sample image based on the initial sample text is different from the generation method of the second sample image based on the first sample text. For example, the initial sample text is converted to the first sample image by implementing a first network structure of the Vincent graph, and the first sample text is converted to the second sample image by implementing a second network structure of the Vincent graph. The first network structure and the second network structure are different, so that the different network structures can reflect the difference in the processing results of the content information.

[0078] S340: Construct an image loss according to the difference between the first sample image and the second sample image, and generate a model loss according to the image loss.

[0079] In an embodiment of the present application, from the initial sample text to the first sample image, from the first sample image to the first sample text, and then from the first sample text to the second sample image, through text-to-image, image-to-text, and text-to-image, during this period, the image content of the described sample text may have changed. In order for the trained model to better learn the mapping relationship between image and text, the image loss is constructed through the difference between the first sample image and the second sample image, where the difference between the first sample image and the second sample image refers to the similarity between the first sample image and the second sample image.

[0080] After generating the image loss, the model loss is generated according to the image loss. For example, the model loss is generated after processing the image loss, such as setting the weight.

[0081] S350: Adjust the model parameters of the model to be trained according to the model loss to obtain a graph-text model.

[0082] The model parameters of the model to be trained are adjusted according to the model loss until the network of the model to be trained converges, and a trained graph-text model is obtained.

[0083] It should be noted that for other detailed descriptions of S210 to S230 shown in FIG3 , please refer to S210 to S230 shown in FIG2 , which will not be repeated here.

[0084] In an embodiment of the present application, a first sample image is generated by an initial sample text that describes the image content, that is, a first sample image and a second sample image are obtained through a conversion process of text-to-image, image-to-text, and text-to-image. Then, image loss is constructed based on the difference between the first sample image and the second sample image, so that the image loss can better measure the consistency of the image content. Subsequently, during model training, the model can avoid the loss of image content, so that the generated text can describe the image as accurately as possible.

[0085] The present application provides another image processing method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In the present application, the method is described by taking the server as an example. As shown in FIG4 , the image processing method is based on FIG3 , and S340 shown in FIG3 is expanded to S410 to S420. S410 to S430 are described in detail as follows:

[0086] S410 , performing feature extraction on the first sample image and the second sample image respectively to obtain a first sample image feature of the first sample image and a second sample image feature of the second sample image.

[0087] In an embodiment of the present application, feature extraction is performed on the first sample image and the second sample image respectively, and the feature extraction methods may be the same or different. For example, the first sample image feature is extracted by calculating the gradient direction and intensity of each pixel point in the image to obtain the first sample image feature, and the second sample image feature is obtained by performing multi-scale analysis on the second sample image to extract features with rotation invariance and scale invariance.

[0088] It is understandable that if the feature formats of the first sample image feature and the second sample image feature are different, the first sample image feature and the second sample image feature need to be normalized to ensure consistency in feature format, so as to facilitate subsequent construction of image loss.

[0089] S420: Construct an image loss according to the distance between the first sample image feature and the second sample image feature.

[0090] In the embodiment of the present application, the difference between the image features is reflected by the distance between the first sample image feature and the second sample image feature. The closer the distance, the smaller the difference. The similarity between the first sample image feature and the second sample image feature can be calculated, and the similarity is used as the distance between the first sample image feature and the second sample image feature.

[0091] In one example, the similarity between the first sample image feature and the second sample image feature may be calculated using cosine similarity or Euclidean distance.

[0092] S430: Generate model loss according to the image loss.

[0093] It should be noted that, for the detailed introduction of S210 to S230, S310 to S330, and S350 shown in FIG4 , please refer to S210 to S230, S310 to S330, and S350 shown in FIG3 , which will not be repeated here.

[0094] In the embodiment of the present application, the distance between the first sample image feature of the first sample image and the second sample image feature of the second sample image can reflect the similarity between the first sample image and the second sample image, thereby ensuring the accuracy of the constructed image loss.

[0095] The present application also provides another image processing method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In the present application, the method is described by taking the server as an example. As shown in FIG5 , based on FIG3 , S340 shown in FIG3 is expanded to S510 to S530. S510 to S530 are described in detail as follows:

[0096] S510: Construct an image loss according to the difference between the first sample image and the second sample image.

[0097] S520: Construct a text loss according to the difference between the initial sample text and the first sample text.

[0098] In the embodiment of the present application, text loss is used to characterize the loss of text content during the image-to-text conversion process.

[0099] As described above, before the first sample text is generated, it goes through text-to-image and image-to-text, and the text semantics describing the image content may also change. In order for the trained model to better learn the mapping relationship between image and text, the text loss is constructed through the difference between the initial sample text and the first sample text, where the difference between the initial sample text and the first sample text refers to the similarity between the initial sample text and the first sample text.

[0100] S530: Generate model loss according to text loss and image loss.

[0101] In one example, the sum of text loss and image loss can be used as the model loss.

[0102] In one example, weights may be configured for text loss and image loss, and the weighted sum of the text loss and image loss may be used as the model loss.

[0103] It should be noted that for other detailed descriptions of S210 to S230 , S310 to S330 , and S350 shown in FIG5 , please refer to S210 to S230 , S310 to S330 , and S350 shown in FIG3 , which will not be repeated here.

[0104] In the embodiment of the present application, not only the consistency of the image content during the conversion process is considered, but also the consistency of the text description during the conversion process is considered. The text loss is constructed by the initial sample text and the first sample text, so that the model is subject to multiple constraints of text and image, ensuring that the subsequent model can avoid information loss of image content and text description.

[0105] In one embodiment of the present application, another image processing method is provided. The image processing method can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In this embodiment of the present application, the method is described by taking the method executed by the server as an example. As shown in FIG6 , the image processing method is based on the method shown in FIG5 , but S520 is expanded to S610 to S620. S610 to S620 are described in detail as follows:

[0106] S610: Acquire initial sample text features corresponding to valid words and sentences with semantic information in the initial sample text, and first sample text features corresponding to valid words and sentences in the first sample text.

[0107] In an embodiment of the present application, there may be differences in the words in the initial sample text and the first sample text. First, it is necessary to obtain valid words and sentences in the initial sample text. The valid words and sentences include words or sentences with semantic information, such as "dance", while groups of words such as "of" and "is" are non-valid words; among them, sentences in which the relationship between elements is unclear, or lack of context or context, making it impossible for people to understand their meaning are sentences without semantics, for example, the single sentence "That incident is very serious" is a sentence without semantics.

[0108] Similarly, valid words and sentences are extracted from the first sample text, and then feature extraction is performed on the valid words and sentences in the initial sample text to obtain initial sample text features, and feature extraction is performed on the valid words and sentences in the first sample text to obtain first sample text features. The feature extraction methods for the initial sample text and the first sample text can be the same or different. For example, the valid words and sentences are converted into vector representations, and the distributed representation of the valid words and sentences is learned by linking the valid words and sentences with the valid words and sentences in the surrounding context to obtain the initial sample text features; for example, by analyzing the structure of the valid words and sentences, grammatical information such as nouns, verbs, adjectives, etc. in the sentence is extracted as the first sample text features.

[0109] It is understandable that if the feature forms of the initial sample text feature and the first sample text feature are different, the initial sample feature and the first sample text feature need to be processed to unify the feature forms to facilitate subsequent construction of text loss.

[0110] In some examples, in order to improve the efficiency of feature extraction, after extracting valid words and sentences from the sample text, words with distinguishing ability can be screened out from the valid words and sentences for feature extraction. For example, if a word or phrase appears frequently in one article and rarely appears in other articles, it is considered that this word or phrase has good category distinguishing ability.

[0111] S620: Construct a text loss according to the distance between the initial sample text feature and the first sample text feature.

[0112] In the embodiment of the present application, the difference between the text features is reflected by the distance between the initial sample text feature and the first sample text feature. The smaller the distance, the smaller the difference. The similarity between the initial sample text feature and the first sample text feature can be calculated, and the similarity is used as the distance between the initial sample text feature and the first sample text feature.

[0113] In one example, the similarity between text features can be calculated using cosine similarity or Euclidean distance.

[0114] It should be noted that for other detailed introductions of S210~S230, S310~S330, S510, S530 and S350 shown in Figure 6, please refer to S210~S230, S310~S330, S510, S530 and S350 shown in Figure 5, which will not be repeated here.

[0115] In an embodiment of the present application, the distance between the initial sample text features corresponding to the valid words and sentences with semantic information in the initial sample text and the first sample text features corresponding to the valid words and sentences in the first sample text can reflect the similarity between the initial sample text and the first sample text, thereby ensuring the accuracy of the constructed text loss.

[0116] It is worth noting that in one embodiment of the present application, another image processing method is also provided. The image processing method can be applied to the implementation environment shown in Figure 1. The method can be executed by a terminal or a server, or by both the terminal and the server. In the embodiment of the present application, the method is described as being executed by a server. As shown in Figure 7, the image processing method is based on that shown in Figure 5, and adds a process of realizing a text-based image through a text-based model after S230, including S710 to S720. Among them, S710 to S720 are described in detail as follows:

[0117] S710: Acquire supplementary text for supplementing the target text, and generate a text to be processed according to the supplementary text and the target text.

[0118] It should be noted that the image-text model in the embodiment of the present application can realize text generation from images or image generation from text. After obtaining the target text describing the image content of the target image to be processed, the target text can be supplemented. This means that on the basis of the target text, new information, opinions, data and other content are added to make the text more complete and accurate. For example, the detailed content of the event or object can be supplemented based on the context of the target text to obtain supplementary text.

[0119] After obtaining the supplementary text, the target text and the supplementary text can be modified and polished to obtain the text to be processed, making the text to be processed more fluent, readable and logical.

[0120] S720: Input the text to be processed into the image-text model to obtain an image generated by the image-text model to describe the text to be processed.

[0121] The text to be processed is input into the graph-text model. The model loss of the graph-text model includes text loss and image loss, so that the graph-text model can fully learn the mapping relationship between image and text, and then obtain the image generated by the graph-text model to describe the text to be processed.

[0122] It should be noted that for other detailed introductions of S210 to S230, S310 to S330, S510 to S530, and S350 shown in FIG7 , please refer to S210 to S230, S310 to S330, S510 to S530, and S350 shown in FIG5 , which will not be repeated here.

[0123] In the embodiment of the present application, the image-text model can be applied not only to the application scenario of image-to-text, but also to the application scenario of text-to-image. It can be supplemented by the target text to generate the text to be processed, and then generate a more detailed image based on the text to be processed, which can meet the needs of various re-drawing images.

[0124] In one embodiment of the present application, another image processing method is provided. The image processing method can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In this embodiment of the present application, the method is described by taking the method executed by the server as an example. As shown in FIG8 , the image processing method is based on the method shown in FIG3 , but S320 is expanded to S810 to S830. S810 to S830 are described in detail as follows:

[0125] S810: Acquire initial sample text for describing image content, and perform feature extraction on the initial sample text based on the model to be trained to obtain an initial sample text vector.

[0126] In an embodiment of the present application, feature extraction is performed on the initial sample text, and the input initial sample text can be encoded into a representation vector through the Text Encoder module, that is, the initial sample text vector is obtained.

[0127] S820: Obtain a randomly noisy sample image.

[0128] In the embodiment of the present application, the randomly noisy sample image refers to a sample image using random Gaussian noise. In one example, the randomly noisy sample image is in a latent space.

[0129] S830: Based on the model to be trained, denoise the randomly noisy sample image according to the initial sample text vector to obtain a first sample image.

[0130] In an embodiment of the present application, a denoising process is performed on a randomly noisy sample image based on an initial sample text vector, and the semantics of the initial sample text is injected step by step to obtain a first sample image describing the initial sample text.

[0131] In one example, the denoising process includes: inputting the initial sample text vector and the randomly noisy sample image into the denoising module in the model to be trained, the denoising module iteratively denoising the randomly noisy sample image while taking the initial sample text vector as a condition, wherein the denoising module can predict the noise based on the randomly noisy sample image and the initial sample text vector, and then subtract the predicted noise from the randomly noisy sample image to obtain a predicted denoised image representation, and then predict the noise again based on the predicted denoised image representation and the initial sample text vector, and subtract the predicted noise again from the predicted denoised image representation, and iterate multiple times; wherein in the denoising process, the noise can be predicted by the injected initial sample text vector, and then the predicted noise is gradually subtracted to obtain a first sample image that conforms to the text corresponding to the initial sample text vector, and the first sample image can accurately describe the initial sample text.

[0132] It should be noted that for other detailed descriptions of S210 to S230, S310, and S330 to S350 shown in FIG8 , please refer to S210 to S230, S310, and S330 to S350 shown in FIG3 , which will not be repeated here.

[0133] In an embodiment of the present application, feature extraction is performed on the initial sample text to obtain an initial sample text vector, and the randomly noisy sample image is denoised based on the initial sample text vector, that is, semantic information is injected during the denoising process to ensure that the generated first sample image can accurately describe the initial sample text.

[0134] In one embodiment of the present application, another image processing method is provided. The image processing method can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In this embodiment of the present application, the method is described by taking the method executed by the server as an example. As shown in FIG9 , the image processing method is based on the method shown in FIG3 , but S330 is expanded to S910 to S940. S910 to S940 are described in detail as follows:

[0135] S910: Based on the model to be trained, perform image encoding on the first sample image to obtain an image feature vector.

[0136] In an embodiment of the present application, the first sample image can be input into an image encoder to perform image encoding on the first sample image, wherein the image encoder is used to extract feature vectors from the image, and quantize the extracted feature vectors and then perform compression encoding; after image encoding, all feature vectors are extracted from the image encoder to obtain image feature vectors, wherein the image feature vectors include but are not limited to pixel values, edges and color features in the image.

[0137] S920 . Based on the model to be trained, obtain a target feature vector according to the image feature vector and the query vector previously learned through text information. The target feature vector is used to represent image information related to the text information in the first sample image.

[0138] In an embodiment of the present application, it is necessary to first obtain a query vector learned in advance through text information, and the query vector is obtained by learning a visual-linguistic feature vector based on preset text information and an image set; based on the query vector, all image information related to the text in the image feature vector can be learned, thereby extracting features related to the text and obtaining a target feature vector, so that the target feature vector can represent the image information related to the text information in the first sample image.

[0139] S930: Generate sample image text based on the target feature vector based on the model to be trained.

[0140] In an embodiment of the present application, the target feature vector can be directly input into a language model, which processes the target feature vector to generate sample image text. In one example, the language model can be an LLM (Large Language Model) model, which has a neural network model for generating text, a text encoder-decoder, etc.

[0141] In one example, since the target feature vector is used to represent image information related to text information in the first sample image, a target sample language label can be generated based on the target feature vector and a preset language label. For example, a feature vector corresponding to key text information is first extracted from the target feature vector, including features such as the shape, color, and size of the object. The feature vector corresponding to the proposed key text information is compared with the preset language label to obtain the most matching label, namely the target sample language label. The target sample language label can be at least one of a word, a short sentence, and an entire sentence. The grammar and semantics of the text can then be predicted based on the target sample language label. Thereafter, a text description is generated based on the predicted grammar and semantics of the text to realize the conversion of the target feature vector into sample image text.

[0142] S940: Based on the model to be trained, generate a first sample text according to the sample image text, and generate a second sample image for describing the first sample text according to the first sample text.

[0143] In one example, the sample image text may be directly used as the first sample text.

[0144] In one example, the sample image text may be processed, such as text amplification, to generate a first sample text based on the processed text.

[0145] For the process of generating the second sample image based on the first sample text, please refer to the subsequent embodiments.

[0146] It should be noted that for other detailed descriptions of S210 to S230, S310 to S320, and S340 to S350 shown in FIG9 , please refer to S210 to S230, S310 to S320, and S340 to S350 shown in FIG3 , which will not be repeated here.

[0147] In an embodiment of the present application, an image feature vector of the first sample image is obtained through image encoding, and then a target feature vector for characterizing image information related to text information in the first sample image is obtained based on the image feature vector and the learned query vector, and then a sample image text is generated, so that the obtained sample image text can accurately describe the sample image.

[0148] In one embodiment of the present application, another image processing method is provided. The image processing method can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In this embodiment of the present application, the method is described by taking the method executed by the server as an example. As shown in FIG10 , the image processing method is based on the method shown in FIG9 , but S940 is expanded to S1010 to S1020. S1010 to S1020 are described in detail as follows:

[0149] S1010 , based on the model to be trained, perform text amplification processing on the sample image text according to the semantic information of the sample image text to obtain amplified sample text.

[0150] In an embodiment of the present application, in order for the model to be trained to fully learn various text forms that describe image content, after generating sample image text based on the first sample image, the sample image text is subjected to text amplification processing, wherein the text amplification processing is performed based on the semantic information of the sample image text to ensure that the meaning of the description of the image content is the same.

[0151] In one example, some words in the sample image text can be replaced with synonyms based on the semantic information of the sample image text, that is, some words in the original text can be replaced with words with similar meanings to the original text to expand the vocabulary and obtain augmented sample text; the sentences in the sample image text can also be reorganized to generate new sentences, but the semantic information of the new sentences is the same as the semantic information of the sentences in the sample image text to obtain augmented sample text.

[0152] In one example, the language form in the sample image text is expanded according to the semantic information of the sample image text to obtain an extended sample text. For example, the sample image text in Chinese is expanded into sample image text in other language forms, such as English sample image text, Japanese sample image text, etc.

[0153] In one example, text amplification processing is performed on the sample image text, such as randomly inserting some words or short sentences into the sample image text, or randomly deleting some words or short sentences to increase the complexity of the text, to obtain an amplified sample text, and the expanded sample text has the same semantic information as the sample image text.

[0154] S1020: Based on the model to be trained, normalize the amplified sample text and the sample image text to obtain a first sample text, and generate a second sample image for describing the first sample text according to the first sample text.

[0155] In an embodiment of the present application, in order to ensure that the formats of the amplified sample text and the sample image text are consistent, the amplified sample text and the sample image text are normalized, and the statistical distribution of the unified samples is summarized to obtain the first sample text. It can be understood that the first sample text includes the sample image text and the amplified sample text.

[0156] Here, a second sample image is generated to describe the first sample text. For the specific process, please refer to the subsequent embodiments.

[0157] It should be noted that for other detailed introductions of S210~S230, S310~S320, S910~S930, S340~S350 shown in Figure 10, please refer to S210~S230, S310~S320, S910~S930, S340~S350 shown in Figure 9, which will not be repeated here.

[0158] In an embodiment of the present application, text amplification is performed on the sample image text using the semantic information of the sample image text, and then based on the amplified sample text and the sample image text, the generated first sample text has rich meaning and can be applied to various scenarios, thereby ensuring better robustness during subsequent model training.

[0159] The present application provides another image processing method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In the present application, the method is described by taking the method executed by the server as an example. As shown in FIG11 , the image processing method is based on the method shown in FIG3 , but S330 is expanded to S1110 to S1140. S1110 to S1140 are described in detail as follows:

[0160] S1110 : Based on the model to be trained, generate a first sample text according to the first sample image, and perform feature extraction on the first sample text to obtain a first sample text vector.

[0161] In the embodiment of the present application, feature extraction is performed on the first sample text, and the input first sample text can be encoded into a representation vector through a text encoder, that is, a first sample text vector is obtained.

[0162] S1120. Based on the model to be trained, the randomly noisy sample image is denoised one by one according to the first sample text vector and the preset noise sampling step number to obtain multiple noisy sample images, wherein the noise intensity corresponding to each denoising process is the same.

[0163] In an embodiment of the present application, a randomly noisy sample image and a preset noise sampling step number are first obtained. The preset noise sampling step number includes the number of steps for denoising. For example, if the preset noise sampling step number is 50, the randomly noisy sample image needs to be denoised 50 times, and the noise intensity corresponding to each noise sampling step number is consistent.

[0164] In one example, a random noisy sample image is subjected to successive denoising processing according to a preset number of noise sampling steps, and during the successive denoising processing, semantics are injected according to the first sample text vector. It should be noted that the successive denoising process refers to predicting the noise to be subtracted based on the randomly noisy sample image and the initial sample text vector, and subtracting the predicted noise from the randomly noisy sample image to obtain a first noise sample image, and then repeating multiple times according to the preset noise sampling steps, that is, predicting the noise to be subtracted based on the first noise sample image and the initial sample text vector, and subtracting the predicted noise from the first noise sample image to obtain a second noise sample image, and so on, until the preset noise sampling steps are reached, and multiple noise sample images are obtained, and the number of noise sample images is the same as the number of denoising processes; wherein, the noise value corresponding to the first denoising process and the noise value corresponding to the second denoising process may be different, but the noise intensity is the same, wherein the noise value refers to the amount or degree of noise existing in the image, and the noise intensity refers to the degree of influence of the noise in the image, that is, the degree of influence of the noise on the image quality, wherein the noise intensity can be measured by indicators such as signal-to-noise ratio (SNR) or peak signal-to-noise ratio (PSNR).

[0165] S1130 : Select at least two noise sample images from a plurality of noise sample images based on the model to be trained, where the at least two noise sample images include a target noise sample image corresponding to the last denoising process.

[0166] In an embodiment of the present application, the at least two noise sample images include a target noise sample image corresponding to the last denoising process, wherein the target noise sample image corresponding to the last denoising process is a sample image in which there is no noise value or the noise value is the smallest after the denoising is completed, and the target noise sample image can describe the first sample text to a certain extent.

[0167] For the other noise sample images other than the target noise sample image in the at least two noise sample images, the other noise sample images are sample images with a certain noise value, which can be randomly selected from multiple noise sample images or selected periodically according to the number of denoising processes, which is not limited here.

[0168] In one example, the number of other noise sample images can be flexibly selected according to actual conditions, for example, determined according to the number of denoising processes. The more denoising processes are performed, the more other noise sample images are selected.

[0169] S1140: Generate a second sample image based on the model to be trained and at least two noise sample images.

[0170] In an embodiment of the present application, in order to ensure that the disturbance corresponding to the first sample text is sufficiently robust, the generated second sample image is generated by combining other noise sample images with a certain noise value and a target noise sample image without a noise value. In one example, the other noise sample images and the target noise sample image can be intersection-processed to generate the second sample image.

[0171] It should be noted that for other detailed descriptions of S210 to S230, S310 to S320, and S340 to S350 shown in FIG11 , please refer to S210 to S230, S310 to S320, and S340 to S350 shown in FIG10 , which will not be repeated here.

[0172] In an embodiment of the present application, a randomly noisy sample image is successively denoised according to a first sample text vector corresponding to a first sample text and a preset number of noise sampling steps to obtain a plurality of noise sample images, from which at least two noise sample images including a target noise sample image corresponding to the last denoising process are selected to generate a second sample image, thereby ensuring the disturbance of the first sample text and making the second sample image sufficiently robust.

[0173] The present application provides another image processing method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both the terminal and the server. In the present application, the method is described by taking the method executed by the server as an example. As shown in FIG12 , the image processing method is based on FIG11 , and S1140 shown in FIG11 is expanded to S1210 to S1230. S1210 to S1230 are described in detail as follows:

[0174] S1210 : Based on the model to be trained, add values ​​of corresponding pixels of other noise sample images except the target noise sample image in at least two noise sample images to obtain an intermediate noise sample image.

[0175] As described above, the other noise sample images have a certain degree of noise value. First, the other noise sample images are added pixel by pixel, that is, the values ​​of pixels at the same position are added to obtain an intermediate noise sample image.

[0176] S1220 : Based on the model to be trained, obtain a disturbance term set for the target noise sample image, and perform disturbance processing on the target noise sample image according to the disturbance term to obtain a disturbed noise sample image.

[0177] In an embodiment of the present application, the target noise sample image does not have a noise value. In order to ensure that the second sample image has sufficient perturbation, a disturbance term pre-set for the target noise sample image is obtained, and the target noise sample image is perturbed by the disturbance term. For example, the disturbance term is multiplied by the target noise sample image to obtain a perturbed noise sample image.

[0178] S1230 : Based on the model to be trained, generate a second sample image according to the intermediate noise sample image and the disturbed noise sample image.

[0179] In an embodiment of the present application, the intermediate noise sample image and the disturbed noise sample image are intersected to generate a second sample image, that is, the values ​​of corresponding pixels of the intermediate noise sample image and the disturbed noise sample image are added to obtain the second sample image.

[0180] It should be noted that for the detailed introduction of S210~S220, S210~S230, S310~S320, S1110~S1130, S340~S350 shown in Figure 12, please refer to S210~S230, S310~S320, S1110~S1130, S340~S350 shown in Figure 11, which will not be repeated here.

[0181] In the embodiment of the present application, sufficient disturbance is added to the target noise sample image by adding pixel by pixel to the other noise sample images.

[0182] It should be noted that the embodiment of the present application also provides another image processing method, which can be applied to the implementation environment shown in Figure 1. The method can be executed by a terminal or a server, or by both the terminal and the server. In the embodiment of the present application, the method executed by the server is used as an example for explanation. As shown in Figure 13, the image processing method is expanded from S1120 to S1310 to S1330 based on the method shown in Figure 11. Among them, S1310 to S1330 are described in detail as follows:

[0183] S1310 : After performing any denoising process on the randomly noisy sample image based on the model to be trained, obtain the current noisy sample image.

[0184] S1320 : Based on the model to be trained, predict the current noise value according to the current noise sample image and the first sample text vector.

[0185] In an embodiment of the present application, after performing any denoising process on the randomly noisy sample image, the current noise sample image is obtained, and the current noise value is predicted according to the current noise sample image and the current noise sampling step number by a pre-trained denoising module.

[0186] Among them, when training, the denoising module will first perform the process of adding noise to the image, considering adding noise to a variable randomly sampled from the real data distribution. After adding N (preset noise sampling steps) times, a sequence of length N will be obtained. As N increases, the original data will lose its characteristics and become a pure Gaussian noise; and denoising is the reverse process, randomly selecting a noise sampling step n and the noise sample image corresponding to the noise sampling step n, and inputting the noise sample image corresponding to the noise sampling step n and the first sample text vector into the pre-trained denoising module. The denoising module can convert the reverse process into a forward process, and then reverse the process distribution to predict the noise from the noise sample image.

[0187] S1330. Based on the model to be trained, perform denoising processing according to the current noise sample image and the current noise value to generate the next noise sample image, until the number of denoising processes reaches the preset number of noise sampling steps, so as to obtain multiple noise sample images.

[0188] In an embodiment of the present application, the current noise value is subtracted from the current noise sample image to obtain the next noise sample image, and S1320 to S1330 are repeatedly executed, that is, the next noise value is predicted based on the next noise sample image and the first sample text vector, and then the next noise value is subtracted from the next noise sample image to generate the next noise sample image, until the number of denoising processes reaches a preset number of noise sampling steps, for example, until it reaches N times, the noise sample image generated for the Nth time is the target noise sample image corresponding to the last denoising process, and the noise sample images generated after each denoising process are collected to obtain multiple noise sample images.

[0189] It should be noted that for other detailed introductions of S210~S230, S310~S320, S1110, S1130~S1140, S340~S350 shown in Figure 13, please refer to S210~S230, S310~S320, S1110, S1130~S1140, S340~S350 shown in Figure 11, which will not be repeated here.

[0190] For ease of understanding, the embodiment of the present application also provides an image processing method, which is illustrated by a specific example. As shown in FIG14 , a model structure diagram of a model to be trained is provided, and the model to be trained includes a reverse diffusion module, a caption module, a prompt amplification module, and a reverse diffusion module with robustness constraints. The training sample of the model to be trained includes an initial sample text prompt, which is converted into a first sample image image by the reverse diffusion module, and the caption module is introduced to perform text re-parse on the first sample image. Then, the prompt amplification module is used to perform data expansion on the re-parsed text to obtain a first sample text prompt-r, and the first sample text is input into the reverse diffusion module with robustness constraints to generate a second sample image image-r, wherein the initial sample text should be as consistent as possible with the first sample text, and the first sample image should be as consistent as possible with the second sample image. Through the multiple constraints on the sample text and sample image, it is ensured that after the image is generated by the image-text model trained by the model to be trained, the text of the image re-parsed can describe the image as much as possible.

[0191] Here, the above-mentioned reverse diffusion module, caption module, prompt amplification module, and reverse diffusion module with robustness constraints are described in detail, wherein the reverse diffusion module, caption module, prompt amplification module, and reverse diffusion module with robustness constraints can be multiple different types of neural networks.

[0192] As shown in Figure 15, the reverse diffusion module includes a text feature extraction module and a denoising module. The initial sample text passes through the text feature extraction module to obtain a text embedding. The text embedding and a randomly noisy sample image (initialized using random Gaussian noise) are input into the denoising module together, and finally the first sample image is output. In one example, the text feature extraction module is a CLIP (Contrastive Language-Image Pre-Training) model.

[0193] Before introducing the denoising module, the forward diffusion process is explained first. As shown in Figure 16, the forward diffusion process is the process of adding noise to the image. Under the same noise intensity, different noise sampling steps N are used to add noise, generating different noise maps. That is, from left to right, they are step 0 (original image), step N / 2, and the difference in noise generated by step N.

[0194] The denoising module iteratively denoises the randomly noisy sample images while taking the text embedding as a condition. This can be seen as the reverse process of the denoising process, as shown in Figure 16. That is, during the denoising process of the denoising module, the text embedding vector is injected into the denoising process through the attention mechanism, resulting in different noisy sample images.

[0195] As shown in Figure 17, the denoising process includes: regarding the randomly noisy sample image as the noise image of step 0, predicting the noise value of step 1 based on the randomly noisy sample image and the text embedding vector, subtracting the noise value of step 1 from the randomly noisy sample image, similarly, using the noise sample image of step 1 and the text embedding vector to predict the noise value of step 2, reducing the noise sample image of step 1 by the noise value of step 2, and so on, until the denoising process is performed N times, obtaining the noise sample image of step N, and the noise sample image of step N is the original image after denoising.

[0196] As shown in Figure 18, the caption module includes an image encoder, a query transformation module (Q-Former), and a language model (such as an LLM model). During the training phase of the Q-Former module, the training text information is input into the second branch of the Q-Former module, and the training image set is input into the first branch of the Q-Former module via the image encoder. A set of learnable query embeddings is used as the input of the first branch. Through the interaction between the first and second branches, the learnable query embeddings interact with the features output by the image encoder, and the learnable query embeddings interact with the training text information. During the training phase, three objectives need to be jointly optimized: the Image-Text Contrastive Loss (ITC), which learns to align image and text representations to maximize their mutual information; the Image-Text Matching Loss (ITM), which learns fine-grained alignment between image and text representations; and the Image-Grounded Text Generation Loss (ITG), which trains the Q-Former to generate text given an input image as a condition. Based on these three goals, the learnable query embedding learns the feature representation between text and image, obtains the learned query vector, and then uses the learned query vector to learn text-related features from the image encoder, becoming a bridge between the image encoder and the language model.

[0197] In the application stage of the Q-Former module, the first sample image is input into the image encoder, which performs image encoding on the first sample image and outputs an image feature vector. The image feature vector is input into the first branch of the Q-Former module. Another input of the first branch of the Q-Former module is the query vector learned in the training stage, and then the first branch obtains the target feature vector through the image feature vector and the learned query vector; the target feature vector is output to the language model to obtain the sample image text; in other embodiments of the present application, the target feature vector can also be directly output to the second branch to obtain the sample image text.

[0198] The prompt amplification module is used to supplement the sample image text based on the output of the caption module to obtain a first sample text. For example, the sample image text is amplified according to the semantic information of the sample image text to obtain an amplified sample text, and the amplified sample text and the sample image text are normalized to obtain the first sample text prompt-r.

[0199] It is worth noting that the inverse diffusion module with robustness constraint includes an inverse module and a robustness constraint module, wherein the inverse module is shown in Figures 15 to 17 and will not be described in detail here. The robustness constraint module is used to ensure that the generated image has sufficient robustness for the disturbance of prompt, by considering the intersection of the noise map brought by different N under the same noise intensity and the original image after denoising. For example, in the process of randomly obtaining the noise sample images generated by steps N / 2 and N / 4 and N, the noise sample images generated by steps N / 2 and N / 4 and N are added pixel by pixel to add sufficient perturbation to the generated N-step image, such as: M+aN

[0200] Where M is the image of the N / 2 and N / 4 steps, N is the image of the N step, and α is 1.5.

[0201] The image obtained by adding pixel by pixel is used as the second sample image.

[0202] In order to ensure that the prompt after the text is generated into the image and the image after the image is generated into the text are semantically consistent, image loss and text loss are constructed.

[0203] The embedding tensor obtained by passing the first sample image and the second sample image through the text feature extraction module in the reverse diffusion module is obtained x ,e y , then the image loss is:

[0204] For the initial sample text and the first sample text, obtain the corresponding TF (Term Frequency) vectors, for example: A = (a1, a2, ..., a i ,…a n ) B=(b1,b2,…,b i ,…b n )

[0205] The Euclidean distance of the TF vector is calculated as:

[0206] The text loss is:

[0207] The model parameters of the model to be trained are adjusted through the image loss and text loss to obtain a graphic-text model.

[0208] In an embodiment of the present application, after training the image-text model, for tasks where images generate text, only the caption module in the image-text model can be used to parse the image to obtain the desired prompt; for tasks where text generates images, the entire image-text model can be used to parse the text to obtain the desired image.

[0209] The image processing method provided in the embodiment of the present application ensures that after the model generates an image, the prompt it re-parses can describe the image as much as possible through the above-mentioned multiple constraints on text and image, and after the model generates text, the image it re-parses can describe the text as accurately as possible.

[0210] Here, an embodiment of the device of the present application is introduced, which can be used to perform the image processing method in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the image processing method in the above embodiment of the present application.

[0211] An embodiment of the present application provides an image processing device, as shown in FIG19 , including:

[0212] An acquisition module 1910 is configured to acquire a target image to be processed;

[0213] An input module 1920 inputs the target image into a pre-trained image-text model, wherein the model loss of the image-text model includes an image loss, and the image loss is constructed based on a first sample image and a second sample image converted from a first sample text used to describe the first sample image;

[0214] The acquisition module 1910 is further configured to acquire the target text generated by the image-text model and used to describe the image content of the target image.

[0215] In one embodiment of the present application, based on the aforementioned scheme, the device also includes a training module for obtaining a model to be trained; obtaining an initial sample text for describing the image content, and based on the model to be trained, generating a first sample image for describing the initial sample text according to the initial sample text; based on the model to be trained, generating the first sample text according to the first sample image, and generating a second sample image for describing the first sample text according to the first sample text; constructing the image loss according to the difference between the first sample image and the second sample image, and generating the model loss according to the image loss; adjusting the model parameters of the model to be trained according to the model loss to obtain the image-text model.

[0216] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to perform feature extraction on the first sample image and the second sample image respectively to obtain first sample image features of the first sample image and second sample image features of the second sample image; and construct the image loss according to the distance between the first sample image features and the second sample image features.

[0217] In one embodiment of the present application, based on the aforementioned solution, the training module is further used to construct a text loss according to the difference between the initial sample text and the first sample text; and generate the model loss according to the text loss and the image loss.

[0218] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to obtain initial sample text features corresponding to valid words and sentences with semantic information in the initial sample text, and first sample text features corresponding to the valid words and sentences in the first sample text; and construct the text loss based on the distance between the initial sample text features and the first sample text features.

[0219] In one embodiment of the present application, based on the aforementioned scheme, the device also includes a supplementary module for obtaining supplementary text to supplement the target text, and generating a text to be processed based on the supplementary text and the target text; the input module is also used to input the text to be processed into the graphic model to obtain an image generated by the graphic model to describe the text to be processed.

[0220] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to perform feature extraction on the initial sample text based on the model to be trained to obtain an initial sample text vector; obtain a randomly noisy sample image; and based on the model to be trained, perform denoising on the randomly noisy sample image according to the initial sample text vector to obtain the first sample image.

[0221] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to perform image encoding on the first sample image based on the model to be trained to obtain an image feature vector; based on the model to be trained, obtain a target feature vector according to the image feature vector and the query vector previously learned through text information, and the target feature vector is used to characterize the image information related to the text information in the first sample image; based on the model to be trained, generate sample image text according to the target feature vector; based on the model to be trained, generate the first sample text according to the sample image text.

[0222] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to perform text amplification processing on the sample image text based on the model to be trained and according to the semantic information of the sample image text to obtain amplified sample text; based on the model to be trained, the amplified sample text and the sample image text are normalized to obtain the first sample text.

[0223] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to perform feature extraction on the first sample text based on the model to be trained to obtain a first sample text vector; based on the model to be trained, perform successive denoising processing on the randomly noisy sample image according to the first sample text vector and a preset number of noise sampling steps to obtain multiple noise sample images, wherein the noise intensity corresponding to each denoising processing is the same; based on the model to be trained, select at least two noise sample images from the multiple noise sample images, the at least two noise sample images including the target noise sample image corresponding to the last denoising processing; based on the model to be trained, generate the second sample image according to the at least two noise sample images.

[0224] In one embodiment of the present application, based on the aforementioned scheme, the training module is further configured to, based on the model to be trained, add the values ​​of corresponding pixels of the other noise sample images other than the target noise sample image in the at least two noise sample images to obtain an intermediate noise sample image; based on the model to be trained, obtain a disturbance term set for the target noise sample image, and perform disturbance processing on the target noise sample image according to the disturbance term to obtain a disturbed noise sample image; and based on the model to be trained, generate the second sample image according to the intermediate noise sample image and the disturbed noise sample image.

[0225] In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to obtain the current noise sample image obtained after performing any denoising process on the randomly noisy sample image based on the model to be trained; based on the model to be trained, predict the current noise value according to the current noise sample image and the first sample text vector; based on the model to be trained, perform denoising process according to the current noise sample image, the first sample text vector and the current noise value to generate the next noise sample image, until the number of denoising processes reaches the preset number of noise sampling steps, so as to obtain multiple noise sample images.

[0226] It should be noted that the apparatus provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.

[0227] The device provided in the above embodiment can be arranged in a terminal or in a server. The target image to be processed is input into the image-text model through the device provided in the embodiment of the present application. Since the model loss of the image-text model includes image loss, and the image loss is constructed based on the first sample image and the second sample image converted from the first sample text used to describe the first sample image, the second sample image that has undergone image-to-text and text-to-image conversion can better reflect the consistency of the image content in the conversion process. Therefore, the image loss constructed based on the first sample image and the second sample image enables the image-text model obtained through image loss training to improve the loss of content information during the conversion process and ensure the consistency of image content. Therefore, the target text generated by the image-text model can describe the target image as much as possible, thereby ensuring the accuracy of the target text.

[0228] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above image processing method.

[0229] FIG20 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.

[0230] It should be noted that the computer system 2000 of the electronic device shown in FIG20 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0231] As shown in Figure 20, computer system 2000 includes a processor (Central Processing Unit, CPU) 2001, which can perform various appropriate actions and processes according to the program stored in read-only memory (Read-Only Memory, ROM) 2002 or the program loaded from storage part 2008 into random access memory (Random Access Memory, RAM) 2003, for example, the method of executing the above embodiment. Various programs and data required for system operation are also stored in RAM 2003. CPU 2001, ROM 2002 and RAM 2003 are connected to each other via bus 2004. Input / output (I / O) interface 2005 is also connected to bus 2004.

[0232] In some embodiments, the following components are connected to the I / O interface 2005: an input section 2006 including a keyboard, a mouse, and the like; an output section 2007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 2008 including a hard disk; and a communication section 2009 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 2009 performs communication processing via a network such as the Internet. A drive 2010 is also connected to the I / O interface 2005 as needed. Removable media 2011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 2010 as needed, so that computer programs read therefrom can be installed into the storage section 2008 as needed.

[0233] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 2009, and / or installed from a removable medium 2011. When the computer program is executed by the processor (CPU) 2001, the various functions defined in the system of the present application are executed.

[0234] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0235] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

[0236] The units or modules described in the embodiments of the present application may be implemented in software or hardware, and the units or modules described may also be provided in a processor. The names of these units or modules do not, in certain circumstances, limit the units or modules themselves.

[0237] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method described above. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0238] Another aspect of the present application further provides a computer program product, the computer program product including a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the image processing method described above in each of the above embodiments.

[0239] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0240] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0241] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. An image processing method, the method being executed by a computer device, the method comprising: Obtaining a target image to be processed; Inputting the target image into a pre-trained graph-text model, wherein a model loss of the graph-text model includes an image loss, and the image loss is constructed based on a first sample image and a second sample image converted from a first sample text used to describe the first sample image; Obtain target text generated by the graphic-text model for describing the target image.

2. The method according to claim 1, before inputting the target image into the pre-trained image-text model, the method further comprises: Get the model to be trained; Acquire an initial sample text for describing the image content, and generate a first sample image for describing the initial sample text based on the model to be trained and the initial sample text; Based on the model to be trained, generating the first sample text according to the first sample image, and generating a second sample image for describing the first sample text according to the first sample text; constructing the image loss according to the difference between the first sample image and the second sample image, and generating the model loss according to the image loss; The model parameters of the model to be trained are adjusted according to the model loss to obtain the graphic model.

3. The method according to claim 2, wherein constructing the image loss according to the difference between the first sample image and the second sample image comprises: Performing feature extraction on the first sample image and the second sample image respectively to obtain a first sample image feature of the first sample image and a second sample image feature of the second sample image; The image loss is constructed according to the distance between the first sample image feature and the second sample image feature.

4. The method according to claim 2, wherein generating the model loss according to the image loss comprises: Constructing a text loss according to the difference between the initial sample text and the first sample text; The model loss is generated according to the text loss and the image loss.

5. The method according to claim 4, wherein the step of constructing a text loss according to the difference between the initial sample text and the first sample text comprises: Acquire initial sample text features corresponding to valid words and sentences with semantic information in the initial sample text, and first sample text features corresponding to the valid words and sentences in the first sample text; The text loss is constructed according to the distance between the initial sample text feature and the first sample text feature.

6. The method according to claim 4, after acquiring the target text generated by the graphic-text model for describing the target image, the method further comprises: Acquire a supplementary text that supplements the target text, and generate a text to be processed according to the supplementary text and the target text; The text to be processed is input into the graphic-text model to obtain an image generated by the graphic-text model and used to describe the text to be processed.

7. The method according to claim 2, wherein generating a first sample image for describing the initial sample text based on the model to be trained and the initial sample text comprises: Based on the model to be trained, feature extraction is performed on the initial sample text to obtain an initial sample text vector; Get a randomly noisy sample image; Based on the model to be trained, the randomly noisy sample image is denoised according to the initial sample text vector to obtain the first sample image.

8. The method according to claim 2, wherein generating the first sample text based on the to-be-trained model and the first sample image comprises: Based on the model to be trained, performing image encoding on the first sample image to obtain an image feature vector; Based on the model to be trained, a target feature vector is obtained according to the image feature vector and a query vector previously learned through text information, wherein the target feature vector is used to represent image information related to the text information in the first sample image; Based on the model to be trained, generating sample image text according to the target feature vector; Based on the model to be trained, the first sample text is generated according to the sample image text.

9. The method according to claim 8, wherein generating the first sample text based on the model to be trained and according to the sample image text comprises: Based on the model to be trained, performing text amplification processing on the sample image text according to semantic information of the sample image text to obtain an amplified sample text; Based on the model to be trained, the amplified sample text and the sample image text are normalized to obtain the first sample text.

10. The method according to claim 2, wherein generating a second sample image for describing the first sample text according to the first sample text comprises: Based on the model to be trained, feature extraction is performed on the first sample text to obtain a first sample text vector; Based on the model to be trained, the randomly noisy sample image is subjected to successive denoising processing according to the first sample text vector and a preset noise sampling step number to obtain a plurality of noisy sample images, wherein the noise intensity corresponding to each denoising processing is the same; Based on the model to be trained, selecting at least two noise sample images from the multiple noise sample images, wherein the at least two noise sample images include a target noise sample image corresponding to the last denoising process; Based on the model to be trained, the second sample image is generated according to the at least two noise sample images.

11. The method according to claim 10, wherein generating the second sample image based on the to-be-trained model and the at least two noise sample images comprises: Based on the model to be trained, adding values ​​of corresponding pixels of other noise sample images except the target noise sample image in the at least two noise sample images to obtain an intermediate noise sample image; Based on the model to be trained, a disturbance term set for the target noise sample image is obtained, and the target noise sample image is disturbed according to the disturbance term to obtain a disturbed noise sample image; Based on the model to be trained, the second sample image is generated according to the intermediate noise sample image and the disturbance noise sample image.

12. The method according to claim 10, wherein the step of performing denoising on the randomly noisy sample image one by one based on the model to be trained and according to the first sample text vector and a preset noise sampling step number to obtain a plurality of noisy sample images comprises: After performing any denoising process on the randomly noisy sample image based on the model to be trained, obtaining a current noisy sample image; Based on the model to be trained, predicting a current noise value according to the current noise sample image and the first sample text vector; Based on the model to be trained, denoising is performed according to the current noise sample image and the current noise value to generate the next noise sample image, until the number of denoising processes reaches the preset number of noise sampling steps, so as to obtain multiple noise sample images.

13. An image processing device, comprising: An acquisition module, used for acquiring a target image to be processed; An input module, used for inputting the target image into a pre-trained graph-text model, wherein the model loss of the graph-text model includes an image loss, and the image loss is constructed based on a first sample image and a second sample image converted from a first sample text used to describe the first sample image; The acquisition module is also used to acquire the target text generated by the graphic-text model and used to describe the image content of the target image.

14. An electronic device comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to execute the method according to any one of claims 1 to 12. 15 . A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of an electronic device, the electronic device executes the method according to claim 1 .

16. A computer program product comprising a computer program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 12.

Images