Image processing method and device, equipment and computer storage medium

Abstract

The embodiment of the invention provides an image processing method and device, equipment and a computer storage medium, and the method comprises the steps: obtaining a to-be-fused image and an original scene image, and enabling the to-be-fused image to comprise a to-be-fused object; determining a description text feature and a target high-dimensional semantic feature of the to-be-fused image, the description text feature being used for a semantic space of the to-be-fused image, and the target high-dimensional semantic feature being used for identifying a to-be-fused object in the to-be-fused image in the semantic space; based on the description text features, determining a mask image area used for placing a to-be-fused object in the original scene image; and redrawing the mask image area based on the description text feature, the target high-dimensional semantic feature and the original scene image to obtain a target scene image which comprises the object to be fused. According to the embodiment of the invention, the to-be-fused object can be subjected to rotation and angle adjustment operation in the semantic space of the to-be-fused image, and the quality and effect of image fusion operation are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image processing method, device, equipment and computer storage medium. Background Art

[0002] With the rapid development of image technology, the application scenarios of images are becoming more and more extensive. For example, in the application scenarios of e-commerce, product images have become a key factor in attracting consumers. High-quality product images can not only enhance the visual appeal of products, but also improve the efficiency of customer purchase decisions to a certain extent.

[0003] Currently, image editing tools can edit products into specific scene pictures that users like. Specifically, the image editing process includes: cutting out the product image, and then pasting the cutout object into the specific scene image.

[0004] However, the above-mentioned image editing operation has the following defects: the product image cannot be rotated and the angle cannot be adjusted in the scene corresponding to the scene image, so that the integration degree between the product and the scene image is very poor. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device and computer storage medium, which can enable the object to be fused to be rotated and the angle adjusted in the semantic space of the image to be fused, thereby effectively improving the quality and effect of the image fusion operation.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Acquire an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused;

[0008] Determining a descriptive text feature and a target high-dimensional semantic feature of the image to be fused, wherein the descriptive text feature is used to identify a semantic space of the image to be fused, and the target high-dimensional semantic feature is used to identify an object to be fused in the image to be fused in the semantic space;

[0009] Based on the description text features, determining a mask image region in the original scene image for placing the object to be fused;

[0010] The mask image region is redrawn based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

[0011] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0012] A first acquisition module is used to acquire an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused;

[0013] A first determination module is used to determine the descriptive text features and target high-dimensional semantic features of the image to be fused, wherein the descriptive text features are used for the semantic space of the image to be fused, and the target high-dimensional semantic features are used to identify the object to be fused in the image to be fused in the semantic space;

[0014] The first determination module is further used to determine, based on the description text features, a mask image region in the original scene image for placing the object to be fused;

[0015] The first processing module is used to redraw the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the image processing method shown in the first aspect above.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer storage medium for storing a computer program, wherein the computer program enables a computer to implement the image processing method shown in the first aspect when executed.

[0018] In a fifth aspect, an embodiment of the present invention provides a computer program product, comprising: a computer program, which, when executed by a processor of an electronic device, enables the processor to execute the image processing method shown in the first aspect above.

[0019] The image processing method, apparatus, device and computer storage medium provided in the embodiments of the present application obtain the image to be fused and the original scene image, determine the descriptive text features and the target high-dimensional semantic features of the image to be fused, and then determine the mask image area in the original scene image for placing the object to be fused based on the descriptive text features, and redraw the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain the target scene image, thereby effectively realizing efficient image processing operations, and in the process of performing an image fusion operation on the object to be fused in the image to be fused and the original scene image, the mask image area is redrawn and fused based on the descriptive text features, the target high-dimensional semantic features and the original scene image, wherein the descriptive text features can identify the semantic space of the image to be fused, and when performing the image fusion operation, the object to be fused can be rotated and angle adjusted in the semantic space corresponding to the image to be fused, thereby effectively improving the degree of fusion between the image to be fused and the original scene image, and meeting the image processing requirements of different users. When the target scene image is applied to an e-commerce platform, it is beneficial to increase the purchase rate of goods, further improving the practicality of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of the principle of an image processing method provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of determining the target high-dimensional semantic feature based on the first high-dimensional semantic feature and the second high-dimensional semantic feature provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a process for redrawing the mask image area based on the description text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image provided in an embodiment of the present application;

[0025] Figure 5A schematic diagram of the principle of a general commodity injection method based on single-graph understanding provided in the application embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0027] Figure 7 for Figure 6 A schematic structural diagram of an electronic device corresponding to the image processing device shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0030] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0031] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0032] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0033] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0034] Definition of terms:

[0035] Artificial intelligence generated content (AIGC) refers to text, images, audio, video or other types of media content generated using artificial intelligence technology. AIGC technology can imitate human creativity and automatically generate novel and creative content, such as articles, poems, works of art, and music.

[0036] Stablediffusion: is an AI-driven text-to-image generation model that uses deep learning technology to convert text descriptions into corresponding image content. This means that users can enter a text description (for example: "A cat in a space suit walking on the moon"), and the Stable Diffusion model will try to create an image that matches the description.

[0037] Mask: a black and white image, where the foreground value is 255 (white) and the background value is 0 (black). The foreground represents the area of ​​interest.

[0038] Local repainting (inpaint): refers to repainting a local area of ​​the image and changing the local content of the image.

[0039] In order to facilitate understanding of the specific implementation method and implementation effect of the technical solution provided by this embodiment, the relevant technologies are described below:

[0040] With the rapid development of e-commerce, product images have become a key factor in attracting consumers. High-quality product images can not only enhance the visual appeal of products, but also improve the efficiency of customers' purchasing decisions to a certain extent. Therefore, merchants have a growing demand for product image editing, especially editing products directly into specific background scenes that users like, which can give users the most intuitive reference for purchasing decisions and increase user purchase rates.

[0041] Related technologies provide several implementation solutions for product image editing:

[0042] Implementation plan 1: Implement image editing operations through image editing tools, where the image editing tool can edit the product into a specific scene picture that the user likes. Specifically, the image editing tool can cut out the product image and then paste the cutout object into the specific scene image.

[0043] However, the above implementation scheme has the following defects: the product image cannot be rotated and the angle cannot be adjusted in the scene corresponding to the scene picture, so that the degree of integration between the product and the scene picture is very poor.

[0044] Implementation plan 2: Use the Low-Rank Adaptation (Lora) model to train specific products. After the Lora model is trained, the image is generated in the mask smear area in the specified background image.

[0045] However, the above implementation scheme has the following defects: it usually takes more than 10 minutes to train a Lora model. Specifically, in order to train the Lora model, 10-20 product pictures need to be uploaded, and different products need to generate corresponding Lora models for training. For example, a table corresponds to a table Lora model, and a chair corresponds to a table Lora model. Due to the large number of Lora models, the training cost, storage cost and user operation cost of the Lora model are very high.

[0046] In order to solve the above technical problems, this embodiment provides an image processing method, device, equipment and computer storage medium, wherein, with reference to the attached Figure 1 As shown, the executor of the image processing method may be an image processing device 200, which is communicatively connected to a client 100. Specifically, the image processing device 200 may be implemented as any computing device with certain image processing capabilities, and the client 100 may be a mobile phone, a smart wearable device, a tablet computer, a personal computer, a setting application, and the like.

[0047] In addition, the basic structure of the client 100 may include: at least one processor. The number of processors depends on the configuration and type of the terminal. The client 100 may also include a memory, which may be volatile, such as a random access memory (RAM), or non-volatile, such as a read-only memory (ROM), flash memory, etc., or may include both types. The memory usually stores an operating system (OS), one or more application programs, and may also store program data, etc. In addition to the processing unit and the memory, the terminal also includes some basic configurations, such as a network card chip, an IO bus, a display component, and some peripheral devices, etc. Optionally, some peripheral devices may include, for example, a keyboard, a mouse, an input pen, a printer, etc. Other peripheral devices are well known in the art and will not be described in detail here.

[0048] The image processing device 200 refers to a device that can provide image processing operations in a network virtual environment, and generally refers to a device that uses the network to perform information planning and image processing operations. In physical implementation, the image processing device 200 can be any device that can provide computing services, respond to image processing requests made by the client 100, and perform image processing operations based on image processing requests, for example: it can be an edge device, a remote server, a cluster server, a conventional server, a cloud server, a cloud host, a virtual center, etc. The image processing device 200 mainly includes a processor, a hard disk, a memory, a system bus, etc., which is similar to a general computer architecture.

[0049] In the above embodiment, a network connection is established between the client 100 and the image processing device 200, and the network connection may be a wireless or wired network connection. If the client 100 and the image processing device 200 are connected by communication, the network standard of the mobile network may be any one of 2G (Global System for Mobile Communications GSM), 2.5G (General Packet Radio Service GPRS), 3G (Wideband Code Division Multiple Access WCDMA, Time Division Synchronous Code Division Multiple Access TD-SCDMA, Code Division Multiple Access 2000 CDMA2000, Universal Mobile Telecommunications System UTMS), 4G (Long Term Evolution LTE), 4G+ (Enhanced Long Term Evolution LTE+), Worldwide Interoperability for Microwave Access WiMax, 5G, 6G, etc.

[0050] In an embodiment of the present application, when a user has an image processing requirement, the image to be fused and the original scene image can be obtained through the client 100, wherein the image to be fused includes the object to be fused, and different images to be fused may include different objects to be fused. In order to realize the image processing operation, the image to be fused and the original scene image can be sent to the image processing device 200 for analysis and processing.

[0051] The image processing device 200 is used to obtain the image to be fused and the original scene image through the client 100, and analyze and process the image to be fused to determine the descriptive text features and the target high-dimensional semantic features of the image to be fused, wherein the descriptive text features identify the semantic space of the image to be fused, and the target high-dimensional semantic features are used to identify the object to be fused in the image to be fused in the semantic space; after determining the descriptive text features, the mask image area in the original scene image for placing the object to be fused can be determined based on the descriptive text features, and then the mask image area can be redrawn based on the descriptive text features, the target high-dimensional semantic features and the original scene image, so as to obtain the target scene image, which includes the object to be fused, and the scene information in the target scene image is the same as the scene information in the original scene image, thereby realizing the efficient and accurate fusion of the object to be fused in the image to be fused into the original scene image.

[0052] In the above technical scheme, efficient image processing operations are effectively implemented, and in the process of performing image fusion operations on the object to be fused in the image to be fused and the original scene image, the mask image area is redrawn and fused based on the descriptive text features, the target high-dimensional semantic features and the original scene image, wherein the descriptive text features can identify the semantic space of the image to be fused, and when performing the fusion operation, the object to be fused can be rotated and the angle adjusted in the semantic space corresponding to the image to be fused, thereby effectively improving the degree of fusion between the image to be fused and the original scene image, and meeting the image processing needs of different users. When the target scene image is applied in an e-commerce platform, it is beneficial to increase the purchase rate of goods, further improving the practicality of the method.

[0053] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0054] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application; see attached Figure 2As shown, this embodiment provides an image processing method, and the execution subject of the method can be an image processing device. In some instances, the image processing device can be implemented as software, or a combination of software and hardware. When the image processing device is implemented as hardware, it can specifically be various electronic devices with image processing operations, including but not limited to tablet computers, personal computers PCs, servers, cloud servers, etc. When the image processing device is implemented as software, it can be installed in the electronic devices listed above. Specifically, based on the above-mentioned image processing device, the image processing method in this embodiment may include the following steps:

[0055] Step S201: obtaining an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused.

[0056] Step S202: Determine the descriptive text features and target high-dimensional semantic features of the image to be fused, the descriptive text features identify the semantic space of the image to be fused, and the target high-dimensional semantic features are used to identify the object to be fused in the image to be fused in the semantic space.

[0057] Step S203: Based on the description text features, determine the mask image region in the original scene image for placing the object to be fused.

[0058] Step S204: redrawing the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

[0059] The specific implementation methods and effects of the above steps are described in detail below:

[0060] Step 201: Acquire an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused.

[0061] When the user has image processing needs or image fusion needs, in order to implement image processing operations or image fusion operations, the image processing device can obtain the image to be fused and the original scene image, wherein the image to be fused includes the object to be fused, and the objects to be fused included in the image to be fused are different in different application scenarios. In the e-commerce application scenario, the objects to be fused included in the scene to be fused are commodities; in the application scenario of character image processing, the objects to be fused included in the scene to be fused may be characters; the original scene image is used to limit or identify the background environment or scene environment in which the object to be fused is located.

[0062] In some instances, the image to be fused and the original scene image can be obtained through human-computer interaction operations. In this case, the image to be fused and the original scene image can be obtained through human-computer interaction operations. In this case, obtaining the image to be fused and the original scene image may include: displaying a human-computer interaction interface, wherein a plurality of alternative images are displayed in the human-computer interaction interface; obtaining an image selection operation input by a user in the human-computer interaction interface; and obtaining the image to be fused and the original scene image based on the image selection operation.

[0063] In other instances, the image to be fused and the original scene image can be obtained not only through human-computer interaction, but also can be determined through a preset device. In this case, obtaining the image to be fused and the original scene image can include: determining a preset device that is communicatively connected to the image processing device, wherein the preset device can store the image to be fused and the original scene image; the image to be fused and the original scene image can be actively or passively acquired through the preset device.

[0064] It should be noted that the image to be fused and the original scene image can be obtained synchronously and in the same implementation method, or can be obtained asynchronously and in different implementation methods, as long as the accuracy and reliability of obtaining the image to be fused and the original scene image can be guaranteed.

[0065] Step S202: Determine the descriptive text features and target high-dimensional semantic features of the image to be fused, the descriptive text features are used to identify the semantic space of the image to be fused, and the target high-dimensional semantic features are used to identify the object to be fused in the image to be fused in the semantic space.

[0066] After acquiring the image to be fused, in order to accurately fuse the object to be fused in the image to be fused into the original scene image, the image to be fused can be first analyzed and processed to determine the descriptive text features of the image to be fused and the target high-dimensional semantic features, wherein determining the descriptive text features of the image to be fused can include: acquiring an image inversion model for implementing an image inversion operation; performing inversion processing on the image to be fused using the image inversion model to obtain the descriptive text features corresponding to the image to be fused, and the descriptive text features are used to identify the semantic space of the image to be fused.

[0067] In order to accurately perform image processing operations on the image to be fused, it is necessary not only to determine the descriptive text features of the image to be fused, but also to determine the target high-dimensional semantic features of the image to be fused. In some instances, determining the target high-dimensional semantic features of the image to be fused may include: obtaining a neural network model for implementing image feature extraction operations; performing feature extraction operations on the image to be fused using the neural network model to obtain the target high-dimensional semantic features of the image to be fused, thereby stably determining the target high-dimensional semantic features of the image to be fused.

[0068] In other instances, the target high-dimensional semantic features can not only be determined by a neural network model, but also can be determined by a fusion operation of high-dimensional semantic features of different maintained images to be fused. At this time, determining the target high-dimensional semantic features of the images to be fused may include: determining a first high-dimensional semantic feature and a second high-dimensional semantic feature of the images to be fused, the first high-dimensional semantic feature being used to identify the image content of the images to be fused in the semantic space, and the second high-dimensional semantic feature being used to identify the differences between different objects in the images to be fused; determining the target high-dimensional semantic features based on the first high-dimensional semantic features and the second high-dimensional semantic features.

[0069] Among them, after obtaining the image to be fused, a feature extraction operation can be performed on the image to be fused, and a first high-dimensional semantic feature and a second high-dimensional semantic feature of the image to be fused can be determined. Specifically, determining the first high-dimensional semantic feature and the second high-dimensional semantic feature of the image to be fused can include: obtaining a multimodal model (for example: a multimodal deep learning model (Contrastive Language–Image Pre-training, CLIP for short)) and a self-supervised learning model (for example: a self-supervised learning model DINO); using the multimodal model to perform a feature extraction operation on the image to be fused, and obtain the first high-dimensional semantic feature of the image to be fused, and the first high-dimensional semantic feature is used to represent the image content of the image to be fused in the semantic space corresponding to the image generation; using the self-supervised learning model to perform a feature extraction operation on the image to be fused, and obtain the second high-dimensional semantic feature of the image to be fused, and the second high-dimensional semantic feature is used to identify the differences between different objects in the image to be fused.

[0070] It should be noted that after obtaining the first high-dimensional semantic feature, in order to accurately perform image processing operations, the obtained first high-dimensional semantic feature needs to be aligned with the descriptive text feature. In some instances, the specific implementation method of aligning the first high-dimensional semantic feature with the descriptive text feature may include: the difference between the cosine value of the first high-dimensional semantic feature and the cosine value of the descriptive text feature is less than or equal to a preset threshold, that is, the cosine value of the first high-dimensional semantic feature is similar to the cosine value of the descriptive text feature; or, the distance deviation between the Euclidean distance of the first high-dimensional semantic feature and the Euclidean distance of the descriptive text feature is less than or equal to the preset distance, that is, the Euclidean distance of the first high-dimensional semantic feature is similar to the Euclidean distance of the descriptive text feature.

[0071] Since the first high-dimensional semantic feature and the second high-dimensional semantic feature are obtained by performing feature extraction operations on the fused image through different network models, the obtained first high-dimensional semantic feature and the second high-dimensional semantic feature can express the fused image from different dimensions. In order to accurately determine the target high-dimensional semantic feature of the image to be fused, after obtaining the first high-dimensional semantic feature and the second high-dimensional semantic feature, the first high-dimensional semantic feature and the second high-dimensional semantic feature can be analyzed and processed to determine the target high-dimensional semantic feature.

[0072] In some instances, the target high-dimensional semantic feature can be determined by analyzing and processing the first high-dimensional semantic feature and the second high-dimensional semantic feature using a preset fusion algorithm. At this time, based on the first high-dimensional semantic feature and the second high-dimensional semantic feature, determining the target high-dimensional semantic feature may include: obtaining a preset fusion algorithm; and using the preset fusion algorithm to fuse the first high-dimensional semantic feature and the second high-dimensional semantic feature, so that the target high-dimensional semantic feature can be stably acquired.

[0073] Step S203: Based on the description text features, determine the mask image region in the original scene image for placing the object to be fused.

[0074] Since the descriptive text features are used to identify the semantic space corresponding to image generation, and the descriptive text features correspond to the image to be fused, in order to accurately fuse the object to be fused in the fused image into the original scene image, after obtaining the descriptive text features, the mask image area in the original scene image for placing the object to be fused can be determined based on the descriptive text features.

[0075] In some instances, the mask image area can be determined by a pre-trained neural network model. In this case, based on the descriptive text features, determining the mask image area in the original scene image for placing the object to be fused can include: obtaining a pre-trained neural network model; inputting the descriptive text features and the original scene image into the neural network model, and obtaining the mask image area in the original scene image output by the neural network model for placing the object to be fused, thereby effectively ensuring the accuracy and reliability of determining the mask image area.

[0076] In other instances, the mask image area can be determined not only by a preset trained neural network model, but also by the category information of the object to be fused. In this case, based on the descriptive text features, determining the mask image area in the original scene image for placing the object to be fused may include: determining the category information of the object to be fused based on the descriptive text features; determining the target fusion area in the original scene image for placing the object to be fused based on the category information; and performing mask processing on the target fusion area to obtain the mask image area.

[0077] Among them, after obtaining the descriptive text features, the descriptive text features can be analyzed and processed to determine the category information of the object to be fused. Different objects to be fused can correspond to different category information; then, based on the category information, the target fusion area in the original scene image for placing the object to be fused can be determined. In some instances, based on the category information, determining the target fusion area in the original scene image for placing the object to be fused can include: obtaining the adaptation category corresponding to each image area in the original scene image; based on the category information and each adaptation category, determining the target fusion area in the original scene image for placing the object to be fused.

[0078] Specifically, in order to accurately determine the target fusion area for placing the objects to be fused, after acquiring the original scene image, the original scene image can be analyzed and processed to determine the adaptation category corresponding to each image area in the original scene image. In some instances, the adaptation category can be determined by analyzing and processing the original scene image with a pre-trained neural network model, or obtaining the adaptation category corresponding to each image area in the original scene image can include: acquiring an image recognition model; performing a region recognition operation on the original scene image using the image recognition model to obtain the placed objects corresponding to each image area in the original scene image; determining the category corresponding to the placed objects as the adaptation category corresponding to each image area in the original scene image, for example: if the first area in the original scene image corresponds to a table, then it can be determined that the adaptation category corresponding to the first area is the table category; if the second area in the original scene image corresponds to a sofa, then it can be determined that the adaptation category corresponding to the second area is the sofa category; if the third area in the original scene image corresponds to a dining table, then it can be determined that the adaptation category corresponding to the third area is the dining table category.

[0079] After obtaining the adaptation categories corresponding to each image area in the original scene image, the category information and each adaptation category can be analyzed and processed to determine the target fusion area in the original scene image for placing the objects to be fused. In some instances, based on the category information and each adaptation category, determining the target fusion area in the original scene image for placing the objects to be fused can include: determining, in each adaptation category, a target adaptation category that matches the category information; and determining the image area corresponding to the target adaptation category as the target fusion area.

[0080] Specifically, since the number of adaptive categories can be multiple, after obtaining the adaptive categories corresponding to each image area, the target adaptive category that matches the category information can be determined in each adaptive category. In some instances, in each adaptive category, determining the target adaptive category that matches the category information can include: obtaining the similarity between the category information and each adaptive category; when the similarity is greater than or equal to a preset threshold, determining the adaptive category corresponding to the similarity as the target adaptive category.

[0081] For example 1, the various adaptation categories include adaptation category 1, adaptation category 2, adaptation category 3 and adaptation category 4, and then the similarity 1 between the category information and adaptation category 1, the similarity 2 between the category information and adaptation category 2, the similarity 3 between the category information and adaptation category 3, and the similarity 4 between the category information and adaptation category 4 can be determined. When the similarity 2 is 100%, the adaptation category 2 corresponding to the similarity 2 can be determined as the target adaptation category, which effectively ensures the accuracy and reliability of the determination of the target adaptation category.

[0082] In Example 2, each adaptive category includes adaptive category 1, adaptive category 2, adaptive category 3 and adaptive category 4, and then the similarity 1 between the category information and adaptive category 1, the similarity 2 between the category information and adaptive category 2, the similarity 3 between the category information and adaptive category 3, and the similarity 4 between the category information and adaptive category 4 can be determined. A preset threshold for judging whether it is a target adaptive category is pre-configured. For example, the preset threshold can be 95% or 90%, etc. When similarity 1 is 80%, similarity 2 is 96%, similarity 3 is 99%, and similarity is 20%, since similarity 2 and similarity 3 are both greater than the preset threshold, the adaptive category 2 and adaptive category 3 corresponding to the above similarity 2 and similarity 3 can be determined as the target adaptive category, which effectively ensures the accuracy and reliability of determining the target adaptive category.

[0083] After obtaining the target adaptation category, the image area corresponding to the target adaptation category can be determined as the target fusion area, and then the target fusion area is subjected to mask processing operations, so that the mask image area can be stably obtained, which effectively ensures the accuracy and reliability of determining the mask image area.

[0084] Step S204: redrawing the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

[0085] After obtaining the descriptive text features, the target high-dimensional semantic features and the original scene image, the mask image area can be redrawn based on the descriptive text features, the target high-dimensional semantic features and the original scene image, that is, the object to be fused in the image to be fused is fused into the original scene image, so that the target scene image fused with the object to be fused can be obtained, wherein the scene information in the target scene image is the same as the scene information in the original scene image.

[0086] The image processing method provided in this embodiment obtains the image to be fused and the original scene image, determines the descriptive text features and the target high-dimensional semantic features of the image to be fused, and then determines the mask image area in the original scene image for placing the object to be fused based on the descriptive text features, and redraws the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain the target scene image, thereby effectively realizing efficient image processing operations, and in the process of performing an image fusion operation on the object to be fused in the image to be fused and the original scene image, the mask image area is redrawn and fused based on the descriptive text features, the target high-dimensional semantic features and the original scene image, wherein the descriptive text features can identify the semantic space of the image to be fused, and when performing the fusion operation, the object to be fused can be rotated and angle-adjusted in the semantic space corresponding to the image to be fused, thereby effectively improving the degree of fusion between the image to be fused and the original scene image, and can meet the image processing needs of different users. When the target scene image is applied to an e-commerce platform, it is beneficial to increase the purchase rate of goods, further improving the practicality of the method.

[0087] Figure 3 A schematic diagram of determining a target high-dimensional semantic feature based on a first high-dimensional semantic feature and a second high-dimensional semantic feature provided in an embodiment of the present application; on the basis of the above embodiment, refer to the attached Figure 3 As shown, for the target high-dimensional semantic feature, it can be determined not only by analyzing and processing the first high-dimensional semantic feature and the second high-dimensional semantic feature through a preset fusion algorithm, but also by performing a matrix mapping operation on the first high-dimensional semantic feature and the second high-dimensional semantic feature. At this time, based on the first high-dimensional semantic feature and the second high-dimensional semantic feature, the target high-dimensional semantic feature is determined, including:

[0088] Step S301: Perform matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain a mapped semantic feature, wherein the dimension of the mapped semantic feature is greater than the dimension of the first high-dimensional semantic feature and the dimension of the second high-dimensional semantic feature.

[0089] Since the first high-dimensional semantic feature and the second high-dimensional semantic feature can express the image to be fused from different dimensions, in order to improve the accuracy of image processing and ensure the natural degree of fusion of the objects to be fused in the image to be fused into the original scene image, after obtaining the first high-dimensional semantic feature and the second high-dimensional semantic feature, a matrix mapping operation can be performed on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature.

[0090] In some instances, the matrix mapping operation can be implemented by matrix multiplication. In this case, performing matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature can include: performing matrix multiplication on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature, thereby ensuring the accuracy and reliability of determining the mapped semantic feature.

[0091] In other instances, the matrix mapping operation can be implemented not only through matrix multiplication calculation, but also through nonlinear transformation. In this case, performing matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature can include: obtaining an activation function or a kernel function for implementing the nonlinear transformation operation; performing matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature using the activation function or the kernel function to obtain the mapped semantic feature, which also ensures the accurate and reliable determination of the mapped semantic feature.

[0092] It should be noted that no matter what method is used to obtain the mapped semantic features, the dimension of the mapped semantic features obtained is greater than the dimension of the first high-dimensional semantic features, and the dimension of the mapped semantic features is greater than the dimension of the second high-dimensional semantic features. For example, when the first high-dimensional semantic features and the second high-dimensional semantic features are both 256 dimensions, the dimension of the mapped semantic features can be 2048 dimensions, and so on, thereby realizing the matrix mapping operation from low dimension to high dimension, which makes it easier to retain all information to express the fused image.

[0093] Step S302: Determine target high-dimensional semantic features based on the mapped semantic features.

[0094] After acquiring the mapped semantic features, the target high-dimensional semantic features can be determined for the mapped semantic features. In some instances, the post-mapped semantic features can be directly determined as the target high-dimensional semantic features; alternatively, based on the post-mapped semantic features, determining the target high-dimensional semantic features may include: acquiring feature dimension parameters and quantity parameters for limiting the target high-dimensional semantic features; and resampling the post-mapped semantic features based on the feature dimension parameters and quantity parameters to obtain the target high-dimensional semantic features.

[0095] Among them, in order to accurately obtain the target high-dimensional semantic features, the feature dimension parameters and quantity parameters used to limit the target high-dimensional semantic features can be obtained first. The feature dimension parameters and quantity parameters can be set through human-computer interaction, or can be determined through preset experience. For example: the feature dimension parameters can be 512 dimensions, 1024 dimensions, etc., and the quantity parameters can be 32, 64 or 128, etc. After obtaining the feature dimension parameters and quantity parameters, the mapped semantic features can be resampled based on the feature dimension parameters and quantity parameters, so as to obtain the target high-dimensional semantic features that meet the feature dimension parameters and quantity parameters, thereby ensuring the accuracy and reliability of determining the target high-dimensional semantic features.

[0096] In this embodiment, by performing matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature, the mapped semantic feature is obtained, and then the target high-dimensional semantic feature is determined based on the mapped semantic feature. This effectively ensures the accuracy and reliability of the determination of the target high-dimensional semantic feature, thereby improving the accuracy of image processing based on the target high-dimensional semantic feature.

[0097] Figure 4 A schematic diagram of a process for redrawing a mask image region based on description text features, target high-dimensional semantic features and original scene images to obtain a target scene image is provided in an embodiment of the present application; based on any one of the above embodiments, refer to the attached Figure 4 As shown, this embodiment provides a technical solution for implementing a redrawing operation on a mask image area based on an image generation model. At this time, redrawing the mask image area based on the description text features, the target high-dimensional semantic features and the original scene image, and obtaining the target scene image may include:

[0098] Step S401: Acquire an image generation model for implementing an image redrawing operation.

[0099] Step S402: Using descriptive text features, target high-dimensional semantic features, and mask image areas as injection parameters for redrawing the mask image areas, the original scene image and the mask image areas are input into the image generation model for redrawing operations to obtain the target scene image output by the image generation model.

[0100] Among them, an image generation model for realizing image redrawing operation is pre-configured, and the image generation model can be realized as a text-to-image generation model realized by a Stable Diffusion network architecture. After the image generation model is obtained, the graphic generation model can be used to perform the redrawing operation. In the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the description of text features, target high-dimensional semantic features, and mask image area are used as injection parameters for redrawing operation on the mask image area, which effectively realizes that in the process of image fusion operation, the image information of the image to be fused and the characteristics of the mask image area can be fully considered, which is conducive to improving the accuracy and naturalness of the image fusion operation.

[0101] It should be noted that the injection parameters are not the initial input parameters of the image generation model. Instead, in the process of the image generation model performing the image redrawing operation, the descriptive text features, the target high-dimensional semantic features and the mask image area can be used as reference data for the image redrawing operation. That is, the descriptive text features, the target high-dimensional semantic features and the mask image area are not directly input into the image generation model, but the descriptive text features, the target high-dimensional semantic features and the mask image area are used as one of the input parameters of other network layers in the image generation model except the first network layer.

[0102] In some instances, in order to accurately implement the image redrawing operation, the mask image area can be first encoded to obtain the encoding features, and then the encoding features, descriptive text features and target high-dimensional semantic features are used as injection parameters to perform the image redrawing operation. At this time, the descriptive text features, target high-dimensional semantic features and mask image area are used as injection parameters for redrawing the mask image area, and the original scene image and the mask image area are input into the image generation model for redrawing operation. Obtaining the target scene image output by the image generation model may include: obtaining a line drawing of the mask image area; determining the encoding features of the line drawing; using the descriptive text features, target high-dimensional semantic features and encoding features as injection parameters for redrawing the mask image area, and inputting the original scene image and the mask image area into the image generation model to obtain the target scene image output by the image generation model.

[0103] After obtaining the mask image area, a line extraction operation can be performed on the mask image area in the original scene image. Specifically, the canny edge detection algorithm can be used to perform line extraction on the mask image area in the original scene image, so that a line map of the mask image area can be stably obtained; then the line map can be analyzed and encoded to determine the encoding features of the line map, and then the descriptive text features, the target high-dimensional semantic features, and the encoding features can be used as injection parameters for redrawing the mask image area, and the original scene image and the mask image area can be input into the image generation model to obtain the target scene image output by the image generation model. This effectively realizes that in the process of redrawing using the image generation model, the line edge features of the mask image area are fully considered, which is beneficial to improve the accuracy of the image redrawing operation.

[0104] In other instances, the image generation model may include a first network layer and multiple other network layers located after the first network layer. In the process of performing a redrawing operation using the image generation model, descriptive text features, target high-dimensional semantic features, and encoding features can be used as injection parameters for all network layers and part of the network layers in other network layers to assist in the image redrawing operation. At this time, in this embodiment, descriptive text features, target high-dimensional semantic features, and mask image areas are used as injection parameters for redrawing the mask image area, and the original scene image and the mask image area are input into the image generation model for redrawing operation. Obtaining the target scene image output by the image generation model may include: determining a target network layer for injecting descriptive text features, target high-dimensional semantic features, and encoding features among multiple other network layers; in the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the descriptive text features, target high-dimensional semantic features, and encoding features are used as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model.

[0105] When the image generation model includes a first network layer and multiple other network layers located after the first network layer, for example, the multiple other network layers may include a network layer a located after the first network layer, a network layer b located after the network layer a, a network layer c located after the network layer b, and a network layer d located after the network layer c, and then a target network layer for injecting descriptive text features, target high-dimensional semantic features, and encoding features may be determined from the multiple other network layers, wherein the target network layer may be all or part of the network layers in the other network layers. When the target network layer is part of the network layers in the other network layers, the target network layer may be randomly determined from the multiple other network layers using a random algorithm, or the target network layer may be determined by the degree of influence of the network layer on the redrawing operation. At this time, determining the target network layer for injecting descriptive text features, target high-dimensional semantic features, and encoding features from the multiple other network layers may include: obtaining the degree of influence of each other network layer on the redrawing operation; when the degree of influence is greater than or equal to a preset influence threshold, determining the other network layer corresponding to the degree of influence as the target network layer.

[0106] Example 1, when the target network layer is all other network layers, the redrawing operation principle of the image generation model includes: inputting the original scene image and the mask image into the first network layer to obtain a first processing result output by the first network layer; inputting the first processing result, descriptive text features, target high-dimensional semantic features, and encoding features into network layer a for processing to obtain a second processing result output by network layer a; inputting the second processing result, descriptive text features, target high-dimensional semantic features, and encoding features into network layer b for processing to obtain a third processing result output by network layer b; inputting the third processing result, descriptive text features, target high-dimensional semantic features, and encoding features into network layer c for processing to obtain a fourth processing result output by network layer c; inputting the fourth processing result, descriptive text features, target high-dimensional semantic features, and encoding features into network layer d for processing to obtain the target scene image output by network layer d, thereby effectively ensuring the accuracy and reliability of determining the target scene image.

[0107] Example 2, when the target network layer is part of other network layers, for example: network layer b and network layer c are the target network layers, the redrawing operation principle of the image generation model includes: inputting the original scene image and the mask image into the first network layer to obtain a first processing result output by the first network layer; inputting the first processing result into network layer a for processing to obtain a second processing result output by network layer a; inputting the second processing result, descriptive text features, target high-dimensional semantic features, and encoding features into network layer b for processing to obtain a third processing result output by network layer b; inputting the third processing result, descriptive text features, target high-dimensional semantic features, and encoding features into network layer c for processing to obtain a fourth processing result output by network layer c; inputting the fourth processing result into network layer d for processing to obtain the target scene image output by network layer d, thereby effectively ensuring the accuracy and reliability of determining the target scene image.

[0108] In addition, in the process of using the descriptive text features, target high-dimensional semantic features, and encoding features as part of the input parameters of the target network layer, in order to ensure the injection quality and effect of the injection parameters, the cross-attention module can be used to perform attention calculation processing operations on the descriptive text features and the target high-dimensional semantic features, and then the results after the attention calculation processing are used as part of the input parameters of the target network layer. At this time, in the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the descriptive text features, the target high-dimensional semantic features, and the encoding features are used as part of the input parameters of the target network layer, and obtaining the target scene image output by the image generation model may include: using the first cross-attention module to process the target high-dimensional semantic features to obtain a first calculation result; using the second cross-attention module to process the descriptive text features to obtain a second calculation result; in the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the first calculation result, the second calculation result, and the encoding features are used as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model.

[0109] Specifically, a first cross-attention module for analyzing and processing the target high-dimensional semantic features and a second cross-attention module for analyzing and processing the descriptive text features are pre-configured. After acquiring the target high-dimensional semantic features and the descriptive text features, the first cross-attention module can be used to process the target high-dimensional semantic features to obtain a first calculation result corresponding to the target high-dimensional semantic features, and the second cross-attention module can be used to analyze and process the descriptive text features to obtain a second calculation result corresponding to the descriptive text features. Then, in the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the first calculation result, the second calculation result, and the encoding feature can be used as part of the input parameters of the target network layer, so that the target scene image output by the image generation model can be stably and effectively obtained, which is conducive to ensuring the accuracy and reliability of the image redrawing operation, and further improves the natural degree of fusion of the target scene image.

[0110] In this embodiment, by obtaining an image generation model for implementing the image redrawing operation, and then using the description text features, the target high-dimensional semantic features, and the mask image area as injection parameters for redrawing the mask image area, the original scene image and the mask image area are input into the image generation model for redrawing operation, so that the target scene image output by the image generation model can be obtained stably and effectively, which is conducive to ensuring the accuracy and reliability of the image redrawing operation, and further improving the natural degree of fusion of the target scene image.

[0111] For specific applications, refer to the attached Figure 5 As shown, taking a product image as an image to be fused and a product object in the product image as an object to be fused as an example, this application embodiment provides a product injection method, which can use artificial intelligence generated content (AIGC) technology to directly inject the product object into a specific scene image, wherein the scene image can correspond to a specific background scene that the user likes. Specifically, the product image can be edited into an image generation space corresponding to the specific scene image, thereby ensuring the semantic information of the product in the product image, and guiding the image redrawing operation through a line graph. At the same time, the dual cross attention technology is used to realize fast image editing and injection operations of the product image, thereby effectively ensuring the quality and efficiency of product injection. The fused image can provide users with an intuitive purchase decision reference, which is conducive to improving the user purchase rate.

[0112] Specifically, the commodity injection method may include the following steps:

[0113] Step 1: Obtain a product image, and use the first multimodal deep learning model to perform feature extraction operations on the product image to obtain a high-dimensional semantic feature A of the product image.

[0114] Specifically, users can upload product images, which will be subjected to feature extraction operations through the first multimodal deep learning model (Contrastive Language Image Pre-training, CLIP for short) to obtain high-dimensional semantic features A of the product images output by the CLIP model. Feature A is used to characterize the image content of the product image and has a higher degree of alignment with the language space corresponding to the product image.

[0115] Step 2: Use the second multimodal deep learning model to perform feature extraction operations on the product image to obtain high-dimensional semantic features B of the product image.

[0116] Specifically, after obtaining the product image, the second multimodal deep learning model (for example, the self-supervised learning model DINOv2 model) can be used to perform feature extraction operations on the product image to obtain the high-dimensional semantic feature B output by the DINOv2 model. The feature B has better visual discriminability, that is, feature B can represent the fine-grained features of the product image and can distinguish the differences and distinctions between image objects.

[0117] It should be noted that the execution order between step 1 and step 2 is not limited to the order specified by the above serial numbers. Technical personnel in this field can adjust the execution order between step 1 and step 2 according to specific application scenarios or application requirements. For example, step 2 can be executed before step 1, or step 2 and step 1 can be executed synchronously.

[0118] Step 3: Map the high-dimensional semantic feature A and the high-dimensional semantic feature B to obtain the high-dimensional semantic feature C corresponding to the product image.

[0119] Among them, a mapping model for realizing mapping calculation operations is pre-trained, and the mapping model is used to perform mapping calculations on high-dimensional semantic features A and high-dimensional semantic features B, so that the high-dimensional semantic features are mapped to the semantic space that can be understood by the product image, and the mapping result output by the mapping model is obtained, and the mapping result is the high-dimensional semantic feature C.

[0120] Specifically, the principle of using the mapping model to perform mapping calculations on high-dimensional semantic features A and high-dimensional semantic features B is as follows: perform matrix multiplication operations on features A and B to obtain calculated feature C0, wherein the dimension of the calculated feature C0 is higher than that of features A and B; after obtaining the calculated feature C0, feature parameters can be preset to perform a resampling operation on the calculated feature C0, wherein the resampling operation can be implemented in conjunction with the attention mechanism, and then a mapping result that satisfies the preset feature parameters can be obtained, for example: when the preset feature parameters include: 64 and 1024 dimensions, the obtained high-dimensional semantic feature C is represented as 64 and 1024 dimensional features, and the high-dimensional semantic feature C corresponds to the semantic space that can be understood by the product image, thereby effectively ensuring the accurate reliability of determining the high-dimensional semantic feature C.

[0121] Step 4: Use the image inversion model to perform an inversion operation on the product image to obtain the language description text D of the product image, and use the text encoder to encode the language description text D to obtain the text feature E corresponding to the product image.

[0122] It should be noted that the text feature E is used to identify the semantic space that can be understood by the product image, and the high-dimensional semantic feature A is aligned with the text feature E. In order to achieve the above-mentioned alignment of the high-dimensional semantic feature A and the text feature E, the text encoder used to encode the language description text D is the same as the encoder of the first multimodal deep learning model used to obtain the high-dimensional semantic feature A of the product image.

[0123] Step 5: Obtain a scene image, and automatically segment the area for placing commodity objects in the initial scene image according to the language description text D using the image detection model and the segmentation model, and mask the area to obtain the mask area.

[0124] Specifically, the text feature E corresponding to the language description text D is determined, and then the scene image and the text feature E are input into the image detection model for detection to obtain object detection information that matches the commodity object, which can be specifically expressed by a detection box; then the object detection information and the scene image are input into the segmentation model for segmentation processing to obtain the mask area.

[0125] It should be noted that the mask area can be realized not only through image detection models and segmentation models, but also through human-computer interaction operations, that is, displaying the scene image, determining the area position for placing commodity objects based on human-computer interaction operations, and then painting the above area position based on human-computer interaction operations to obtain the mask area.

[0126] Step 6: Extract the line drawing of the masked area in the scene image.

[0127] Specifically, a rough line drawing of the mask area in the scene image can be extracted by using a canny edge detection algorithm, thereby obtaining a line drawing of the mask area.

[0128] Step 7: Perform a local redrawing operation on the mask area corresponding to the line drawing in the scene image to obtain a redrawn scene image.

[0129] Specifically, the redrawing operation can be implemented through a text-to-image generation model (for example, a model implemented by a Stable Diffusion network architecture). At the same time, a copy of the Stable Diffusion UNet encoder is made to be able to encode the line drawing and obtain the encoding features of the line drawing. At the same time, the single cross attention layer (Cross Attention) in each network layer in the generation model is changed to two cross attention layers (Cross Attention), namely, including a first cross attention layer and a second cross attention layer. The first cross attention layer is used to process the high-dimensional semantic feature C to obtain a first calculation result, and the second cross attention layer is used to process the text feature E to obtain a second calculation result.

[0130] The encoding features, the first calculation results, and the second calculation results of the line drawing obtained by the above operation can all be injection parameters for the local redrawing operation. Specifically, the encoding features, the first calculation results, and the second calculation results of the above line drawing can be used as partial input parameters of other network layers except the first network layer. This not only increases the naturalness and accuracy of the redrawing operation, but also helps to improve the fidelity of the image fusion operation.

[0131] Step 8: The smeared area of ​​the scene image in the above process is subjected to 50 steps of denoising to obtain the target scene image.

[0132] Specifically, the 50-step denoising process may refer to repeating step 7 50 times, and then determining the final output redrawn scene image as the target scene image, the target scene image includes the commodity object, and the scene information in the target scene image is the same as the scene information in the original scene image.

[0133] The technical solution provided by the present application embodiment extracts features from the product image through the CLIP model and the DINOv2 model, and maps the extracted image features to the image generation space through the mapping model, so that the product object undergoing the fusion operation can perform angle transformation and posture adjustment operations in the image generation space corresponding to the scene image, making the fusion operation more flexible and reliable, and the training operation of the CLIP model and the DINOv2 model is easy to implement, thereby greatly saving the model training cost; in addition, when the mask area in the scene image is redrawn, the conditional information (encoding information of the line drawing), visual information (product image information), and semantic information (language description text D) of the product image are respectively injected into the entire redrawing process. Specifically, the line drawing of the mask area is encoded by a copy of the encoder of the UNet of Stable Diffusion, and the encoding result of the line drawing is added to the input of each network layer of the Unet undergoing the redrawing operation; at the same time, the original single Cross Attention of each layer of Stable Diffusion is designed as a double Cross Attention, and the above-mentioned product features C and product text E are respectively sent to two Cross In Attention, the first calculation result and the second calculation result corresponding to the product feature C and the product text feature E are obtained respectively, and then the first calculation result and the second calculation result are added to the input of each network layer of Unet, so that the product image can be fused into the scene image with high quality and can be transformed at a certain angle, which effectively realizes that the product image can be adaptively fused into the image generation space corresponding to the scene image, greatly improving the fidelity of the product image. When applied and displayed based on the target scene image, it can not only enhance the visual appeal of the product and enhance the user's application experience, but also improve the efficiency of customer purchase decisions to a certain extent, further improving the practicality of the method.

[0134] Figure 6 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application; see Attachment Figure 6 As shown, this embodiment provides an image processing device, which is used to perform the above Figure 2 The image processing method shown, wherein the image processing device may include:

[0135] A first acquisition module 11 is used to acquire an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused;

[0136] A first determination module 12 is used to determine the descriptive text features and target high-dimensional semantic features of the image to be fused, the descriptive text features identify the semantic space of the image to be fused, and the target high-dimensional semantic features are used to identify the object to be fused in the image to be fused in the semantic space;

[0137] The first determination module 12 is further used to determine the mask image area for placing the object to be fused in the original scene image based on the description text features;

[0138] The first processing module 13 is used to redraw the mask image area based on the description text features, the target high-dimensional semantic features and the original scene image to obtain the target scene image, which includes the object to be fused.

[0139] In some instances, when the first determination module 12 determines the target high-dimensional semantic features of the images to be fused, the first determination module 12 is used to perform: determining the first high-dimensional semantic features and the second high-dimensional semantic features of the images to be fused, the first high-dimensional semantic features being used to identify the image content of the images to be fused in the semantic space, and the second high-dimensional semantic features being used to identify the differences between different objects in the images to be fused; and determining the target high-dimensional semantic features based on the first high-dimensional semantic features and the second high-dimensional semantic features.

[0140] In some examples, the first high-dimensional semantic feature is aligned with the descriptive text feature.

[0141] In some examples, the difference between the cosine value of the first high-dimensional semantic feature and the cosine value of the description text feature is less than or equal to a preset threshold; or,

[0142] The distance deviation between the Euclidean distance of the first high-dimensional semantic feature and the Euclidean distance describing the text feature is less than or equal to the preset distance.

[0143] In some instances, when the first determination module 12 determines the target high-dimensional semantic feature based on the first high-dimensional semantic feature and the second high-dimensional semantic feature, the first determination module 12 is used to execute: matrix mapping the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain a mapped semantic feature, wherein the dimension of the mapped semantic feature is greater than the dimension of the first high-dimensional semantic feature and the dimension of the second high-dimensional semantic feature; and determining the target high-dimensional semantic feature based on the mapped semantic feature.

[0144] In some instances, when the first determination module 12 performs matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature, the first determination module 12 is used to execute: performing matrix multiplication calculation on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature.

[0145] In some instances, when the first determination module 12 determines the target high-dimensional semantic feature based on the mapped semantic feature, the first determination module 12 is used to execute: obtaining feature dimension parameters and quantity parameters for limiting the target high-dimensional semantic feature; resampling the mapped semantic feature based on the feature dimension parameters and quantity parameters to obtain the target high-dimensional semantic feature.

[0146] In some instances, when the first determination module 12 determines the mask image area in the original scene image for placing the object to be fused based on the descriptive text features, the first determination module 12 is used to execute: determining the category information of the object to be fused based on the descriptive text features; determining the target fusion area in the original scene image for placing the object to be fused based on the category information; and performing mask processing on the target fusion area to obtain a mask image area.

[0147] In some instances, when the first determination module 12 determines the target fusion area in the original scene image for placing the object to be fused based on the category information, the first determination module 12 is used to execute: obtaining the adaptation category corresponding to each image area in the original scene image; and determining the target fusion area in the original scene image for placing the object to be fused based on the category information and each adaptation category.

[0148] In some instances, when the first determination module 12 determines the target fusion area for placing the objects to be fused in the original scene image based on the category information and each adaptive category, the first determination module 12 is used to execute: determining the target adaptive category that matches the category information in each adaptive category; and determining the image area corresponding to the target adaptive category as the target fusion area.

[0149] In some instances, when the first determination module 12 determines a target adaptation category that matches the category information among various adaptation categories, the first determination module 12 is used to execute: obtaining the similarity between the category information and each adaptation category; when the similarity is greater than or equal to a preset threshold, determining the adaptation category corresponding to the similarity as the target adaptation category.

[0150] In some instances, when the first processing module 13 redraws the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain the target scene image, the first processing module 13 is used to: obtain an image generation model for implementing the image redrawing operation; use the descriptive text features, the target high-dimensional semantic features, and the mask image area as injection parameters for redrawing the mask image area, input the original scene image and the mask image area into the image generation model for the redrawing operation, and obtain the target scene image output by the image generation model.

[0151] In some instances, when the first processing module 13 uses descriptive text features, target high-dimensional semantic features, and mask image areas as injection parameters for redrawing the mask image area, inputs the original scene image and the mask image area into the image generation model for redrawing, and obtains the target scene image output by the image generation model, the first processing module 13 is used to: obtain a line drawing of the mask image area; determine the encoding features of the line drawing; use descriptive text features, target high-dimensional semantic features, and encoding features as injection parameters for redrawing the mask image area, input the original scene image and the mask image area into the image generation model, and obtain the target scene image output by the image generation model.

[0152] In some instances, the image generation model includes a first network layer and a plurality of other network layers located after the first network layer;

[0153] In the first processing module 13, the descriptive text features, the target high-dimensional semantic features, and the mask image area are used as injection parameters for redrawing the mask image area, and the original scene image and the mask image area are input into the image generation model for redrawing operation to obtain the target scene image output by the image generation model. The first processing module 13 is used to: determine a target network layer for injecting the descriptive text features, the target high-dimensional semantic features, and the encoding features among multiple other network layers; in the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, use the descriptive text features, the target high-dimensional semantic features, and the encoding features as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model.

[0154] In some instances, when the first processing module 13 inputs the original scene image and the mask image area into the image generation model for redrawing, the descriptive text features, the target high-dimensional semantic features, and the encoding features are used as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model. The first processing module 13 is used to: use the first cross-attention module to process the target high-dimensional semantic features to obtain a first calculation result; use the second cross-attention module to process the descriptive text features to obtain a second calculation result; and use the first calculation result, the second calculation result, and the encoding features as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model.

[0155] Figure 6 The device shown can perform Figure 1-Figure 5 For the method of the embodiment shown in the figure, the part not described in detail in this embodiment can be referred to Figure 1-Figure 5 The implementation process and technical effects of this technical solution refer to Figure 1-Figure 5 The description in the illustrated embodiment will not be repeated here.

[0156] In one possible design, Figure 6 The structure of the image processing device shown can be implemented as an electronic device, which can be a terminal, a tablet computer, a personal computer, a server, or other devices. Figure 7 As shown, the electronic device may include: a first processor 21 and a first memory 22. The first memory 22 is used to store the corresponding electronic device executing the above Figure 1-Figure 5 The program of the image processing method provided in the illustrated embodiment, the first processor 21 is configured to execute the program stored in the first memory 22 .

[0157] The program includes one or more computer instructions, wherein the one or more computer instructions can implement the following steps when executed by the first processor 21: obtaining an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused; determining a descriptive text feature and a target high-dimensional semantic feature of the image to be fused, wherein the descriptive text feature is used to identify a semantic space of the image to be fused, and the target high-dimensional semantic feature is used to identify the object to be fused in the image to be fused in the semantic space; based on the descriptive text feature, determining a mask image area in the original scene image for placing the object to be fused; redrawing the mask image area based on the descriptive text feature, the target high-dimensional semantic feature and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

[0158] Furthermore, the first processor 21 is also used to execute the aforementioned Figure 1-Figure 5 All or part of the steps in the illustrated embodiments.

[0159] The structure of the electronic device may further include a first communication interface 23 for the electronic device to communicate with other devices or a communication network.

[0160] In addition, an embodiment of the present invention provides a computer storage medium for storing computer software instructions used by electronic devices, which includes instructions for executing the above Figure 1-Figure 5 The procedures involved in the image processing method in the method embodiment shown.

[0161] In addition, this embodiment adopts a computer program product, which includes: a computer program, when the computer program is executed by a processor of an electronic device, the processor executes the above Figure 1-Figure 5 The image processing method in the method embodiment is shown.

[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0164] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processor of the computer or other programmable device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in the blocks or multiple blocks. In a typical configuration, the computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0167] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused; Determining a descriptive text feature and a target high-dimensional semantic feature of the image to be fused, wherein the descriptive text feature is used to identify a semantic space of the image to be fused, and the target high-dimensional semantic feature is used to identify an object to be fused in the image to be fused in the semantic space; Based on the description text features, determining a mask image region in the original scene image for placing the object to be fused; The mask image region is redrawn based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

2. The method according to claim 1, characterized in that: Determining the target high-dimensional semantic features of the image to be fused includes: Determine a first high-dimensional semantic feature and a second high-dimensional semantic feature of the image to be fused, wherein the first high-dimensional semantic feature is used to identify the image content of the image to be fused in the semantic space, and the second high-dimensional semantic feature is used to identify the difference between different objects in the image to be fused; The target high-dimensional semantic feature is determined based on the first high-dimensional semantic feature and the second high-dimensional semantic feature.

3. The method according to claim 2, characterized in that Determining the target high-dimensional semantic feature based on the first high-dimensional semantic feature and the second high-dimensional semantic feature includes: Performing matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain a mapped semantic feature, wherein the dimension of the mapped semantic feature is greater than the dimension of the first high-dimensional semantic feature and the dimension of the second high-dimensional semantic feature; Based on the mapped semantic features, the target high-dimensional semantic features are determined.

4. The method according to claim 3, characterized in that Performing matrix mapping on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain a mapped semantic feature includes: Perform matrix multiplication on the first high-dimensional semantic feature and the second high-dimensional semantic feature to obtain the mapped semantic feature.

5. The method according to claim 3, characterized in that: Determining the target high-dimensional semantic feature based on the mapped semantic feature includes: Acquire feature dimension parameters and quantity parameters for limiting the target high-dimensional semantic features; A resampling operation is performed on the mapped semantic feature based on the feature dimension parameter and the quantity parameter to obtain the target high-dimensional semantic feature.

6. The method according to any one of claims 1 to 5, characterized in that: Based on the description text features, determining a mask image region in the original scene image for placing the object to be fused includes: Determining the category information of the object to be fused based on the descriptive text features; Based on the category information, determining a target fusion area in the original scene image for placing the object to be fused; Perform mask processing on the target fusion area to obtain the mask image area.

7. The method according to claim 6, characterized in that Based on the category information, determining a target fusion area in the original scene image for placing the object to be fused includes: Obtaining the adaptation category corresponding to each image area in the original scene image; Based on the category information and each adapted category, a target fusion area for placing the object to be fused in the original scene image is determined.

8. The method according to claim 7, characterized in that Based on the category information and each adapted category, determining a target fusion area in the original scene image for placing the object to be fused includes: In each of the adaptation categories, determine a target adaptation category that matches the category information; The image region corresponding to the target adaptation category is determined as the target fusion region.

9. The method according to claim 8, characterized in that In each adaptation category, determine the target adaptation category that matches the category information, including: Obtaining the similarity between the category information and each adapted category; When the similarity is greater than or equal to a preset threshold, the adaptation category corresponding to the similarity is determined as the target adaptation category.

10. The method according to any one of claims 1 to 5, characterized in that: Redrawing the mask image region based on the description text feature, the target high-dimensional semantic feature and the original scene image to obtain the target scene image includes: Obtain an image generation model for implementing an image redrawing operation; The descriptive text features, the target high-dimensional semantic features, and the mask image area are used as injection parameters for redrawing the mask image area, the original scene image and the mask image area are input into the image generation model for redrawing, and the target scene image output by the image generation model is obtained.

11. The method according to claim 10, characterized in that Using the descriptive text feature, the target high-dimensional semantic feature, and the mask image region as injection parameters for redrawing the mask image region, inputting the original scene image and the mask image region into the image generation model for redrawing, and obtaining the target scene image output by the image generation model, including: Obtaining a line drawing of the mask image area; determining a coding feature of the line drawing; The descriptive text features, the target high-dimensional semantic features, and the encoding features are used as injection parameters for redrawing the mask image area, the original scene image and the mask image area are input into the image generation model, and the target scene image output by the image generation model is obtained.

12. The method according to claim 11, characterized in that The image generation model includes a first network layer and a plurality of other network layers located after the first network layer; Using the descriptive text feature, the target high-dimensional semantic feature, and the mask image region as injection parameters for redrawing the mask image region, inputting the original scene image and the mask image region into the image generation model for redrawing, and obtaining the target scene image output by the image generation model, including: Among multiple other network layers, determining a target network layer for injecting the description text feature, the target high-dimensional semantic feature, and the encoding feature; In the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the descriptive text features, the target high-dimensional semantic features, and the encoding features are used as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model.

13. The method according to claim 12, characterized in that In the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the description text feature, the target high-dimensional semantic feature, and the encoding feature are used as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model, including: Processing the target high-dimensional semantic feature using a first cross-attention module to obtain a first calculation result; Processing the description text features using a second cross-attention module to obtain a second calculation result; In the process of inputting the original scene image and the mask image area into the image generation model for redrawing operation, the first calculation result, the second calculation result, and the encoding feature are used as part of the input parameters of the target network layer to obtain the target scene image output by the image generation model.

14. An image processing device, characterized in that: include: A first acquisition module is used to acquire an image to be fused and an original scene image, wherein the image to be fused includes an object to be fused; A first determination module is used to determine the description text features and target high-dimensional semantic features of the image to be fused, wherein the description text features identify the semantic space of the image to be fused, and the target high-dimensional semantic features are used to identify the object to be fused in the image to be fused in the semantic space; The first determination module is further used to determine, based on the description text features, a mask image region in the original scene image for placing the object to be fused; The first processing module is used to redraw the mask image area based on the descriptive text features, the target high-dimensional semantic features and the original scene image to obtain a target scene image, wherein the target scene image includes the object to be fused.

15. An electronic device, characterized in that: include: A memory, a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method as described in any one of claims 1-13.

16. A computer storage medium, characterized in that: Used to store a computer program, which enables a computer to implement the method according to any one of claims 1 to 13 when executed.

17. A computer program product, characterized in that include: A computer program, when executed by a processor of an electronic device, causes the processor to execute the steps of any one of the methods of claims 1 to 13.

Images