Image design method, device, storage medium and electronic equipment

Abstract

The application provides an image design method and device, a storage medium and an electronic device, and relates to the field of computer vision. The image design method comprises the following steps: acquiring a first design image and a modification description text corresponding to the first design image; determining a reference image matched with the modification description text based on the modification description text corresponding to the first design image; and determining a second design image corresponding to the first design image based on the reference image. Through the scheme in the application, the first design image can be quickly modified according to the modification opinions of the customer, and the labor cost and time cost are saved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an image design method, apparatus, storage medium, and electronic device. Background Technology

[0002] In graphic design work, designers often complete graphic designs based on client needs. However, the designs presented to clients often receive a lot of feedback and revisions. Even after adjusting the designs based on these feedback, the final designs may still not meet the client's requirements, resulting in a waste of manpower and time. Summary of the Invention

[0003] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an image design method, apparatus, storage medium, and electronic device.

[0004] In a first aspect, one embodiment of this application provides an image design method, comprising: acquiring a first design image and a modification description text corresponding to the first design image; determining a reference image matching the modification description text based on the modification description text corresponding to the first design image; and determining a second design image corresponding to the first design image based on the reference image.

[0005] In conjunction with the first aspect, in some implementations of the first aspect, determining the second design image corresponding to the first design image based on the reference image includes: determining the sampling probability corresponding to a pixel in the reference image; determining the fused pixel value corresponding to a pixel in the first design image based on the sampling probability corresponding to a pixel in the reference image and the pixel value of a pixel in the first design image; and determining the second design image corresponding to the first design image based on the fused pixel value corresponding to a pixel in the first design image.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, determining the fused pixel value corresponding to a pixel in the first design image based on the sampling probability corresponding to a pixel in the reference image and the pixel value of a pixel in the first design image includes: determining the fusion weight corresponding to a pixel in the reference image; and determining the fused pixel value corresponding to a pixel in the first design image based on the sampling probability and fusion weight corresponding to a pixel in the reference image and the pixel value of a pixel in the first design image.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, determining a second design image corresponding to the first design image based on the fused pixel values ​​corresponding to the pixels in the first design image includes: generating a modified image corresponding to the first design image based on the fused pixel values ​​corresponding to the pixels in the first design image; determining the representation vector of the modified image and the representation vector of the modified description text; and determining the second design image based on the representation vector of the modified image and the representation vector of the modified description text.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, determining a second design image based on the representation vector of the modified image and the representation vector of the modified descriptive text includes: if the vector similarity between the representation vector of the modified image and the representation vector of the modified descriptive text is less than a preset similarity threshold, then adjusting the sampling probability corresponding to the pixel in the reference image; and determining the second design image based on the adjusted sampling probability and the pixel value of the pixel in the modified image.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, determining the second design image based on the representation vector of the modified image and the representation vector of the modified description text includes: if the vector similarity between the representation vector of the modified image and the representation vector of the modified description text is greater than or equal to a preset similarity threshold, then the modified image is determined as the second design image.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, determining a reference image matching the modified description text based on the modified description text corresponding to the first design image includes: determining the representation vector of the modified description text; obtaining at least one candidate image and the representation vector of the image description text of each of the at least one candidate image; and determining a reference image matching the modified description text from the at least one candidate image based on the vector similarity between the representation vector of the modified description text and the representation vector of the image description text of each of the at least one candidate image.

[0011] Secondly, one embodiment of this application provides an image design apparatus, including: an acquisition module, configured to acquire a first design image and a modification description text corresponding to the first design image; a first determination module, configured to determine a reference image matching the modification description text based on the modification description text corresponding to the first design image; and a second determination module, configured to determine a second design image corresponding to the first design image based on the reference image.

[0012] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the image design method described in the first aspect.

[0013] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to execute the image design method described in the first aspect.

[0014] In this embodiment, a first design image and its corresponding modification description text are first obtained. Based on the modification description text, a reference image matching the modification description text is determined. Then, based on the reference image, a second design image corresponding to the first design image is determined. This solution achieves intelligent modification of the first design image, unlike the method used by designers to repeatedly adjust the first design image based on client feedback. This solution saves time and labor costs associated with manual modification, reduces rework, and improves the efficiency of image design. Attached Figure Description

[0015] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 The diagram shown is a scenario applicable to an embodiment of this application.

[0017] Figure 2 The diagram shown is a flowchart illustrating an exemplary embodiment of the image design method provided in this application.

[0018] Figure 3 The diagram shown is a flowchart illustrating the process of determining a second design image according to an exemplary embodiment of this application.

[0019] Figure 4 The diagram shown is a schematic flowchart of determining the fused pixel value provided in an exemplary embodiment of this application.

[0020] Figure 5 The diagram shown is a flowchart illustrating the process of determining a second design image according to another exemplary embodiment of this application.

[0021] Figure 6 The diagram shown is a flowchart illustrating the process of determining a reference image according to an exemplary embodiment of this application.

[0022] Figure 7 The diagram shown is a schematic representation of the structure of an image design apparatus provided in an exemplary embodiment of this application.

[0023] Figure 8The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Application Overview

[0026] In graphic design, designers often create finished images based on client needs. However, graphic design is inherently a subjective art form, not a fixed scientific problem. Therefore, the finished images often receive various suggestions for modification from clients. After receiving feedback from clients, designers may find that their subjective understanding does not match the client's actual needs. This can lead to numerous revisions of the finished image that still fail to meet the client's requirements. The revision process is time-consuming and labor-intensive, wasting significant time and manpower resources.

[0027] Based on the above, this application proposes an image design method that aims to quickly modify existing design images based on modification suggestions through algorithms, and allow clients to determine whether the style of the modified design image meets their aesthetic preferences. After confirmation, designers can then make detailed modifications. This process will save a lot of manpower costs.

[0028] Specifically, this application first obtains a first design image and its corresponding modification description text. Based on the modification description text, a reference image corresponding to the modification description text is determined. Finally, based on the reference image, a second design image corresponding to the first design image is determined. In other words, this application proposes a solution for automatically modifying a first design image based on modification description text. Only the first design image and its corresponding modification description text need to be input to complete the modification of the first design image. Unlike designers who repeatedly adjust the first design image based on client feedback, this solution saves the time and labor costs associated with manual modification, reduces the number of rework attempts, and improves the efficiency of image design.

[0029] Exemplary scenario

[0030] The data processing method proposed in this application can be executed by an electronic device, which can be a terminal, such as a smartphone, tablet computer, desktop computer, etc.; or the electronic device can also be a server, such as an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0031] Based on the image design method proposed in this application, an embodiment of this application provides a schematic diagram of the implementation environment of the image design method. Figure 1 The diagram shown illustrates a scenario applicable to an embodiment of this application. Figure 1 As shown, the application scenarios mentioned in this application embodiment include server 110 and user terminal 120, and there is a communication connection between user terminal 120 and server 110.

[0032] User terminal 120 can be a smartphone, tablet computer, desktop computer, etc. User terminal 120 has an application installed that can drive server 110 to execute image design methods. Server 110 can be a physical machine or a virtual machine, and there can be one or more servers; this application embodiment does not limit the type or number of servers.

[0033] For example, the designer uploads a first design image and modified description text to the user terminal 120. The user terminal 120 then sends the first design image and modified description text to the server 110 via an application. The server 110 determines a reference image based on the modified description text and, based on the reference image, determines a second design image corresponding to the first design image. Specifically, the server 110 is equipped with an encoder. The server 110 encodes the modified description text, the first design image, and the reference image using the encoder. After determining the second design image, the server 110 sends the second design image to the user terminal 120. The user terminal 120 then presents the second design image to the designer through a relevant application and allows the designer to download and use it.

[0034] Exemplary methods

[0035] Figure 2 The diagram shown is a flowchart illustrating an exemplary embodiment of the image design method provided in this application. Figure 2 As shown, the image design method provided in this application includes the following steps.

[0036] Step S210: Obtain the first design image and the corresponding modified description text.

[0037] For example, the first design image represents a first design style. The first design image may be a draft image designed by a designer based on the client's needs; or the first design image may be a revised image obtained by the designer based on the client's feedback, after modifying the draft. The objects depicted in the first design image may be people, objects, animals, or landscapes, and the embodiments of this application do not limit the specific content of the first design image.

[0038] Modify the description text to reflect the client's feedback on the initial design image. For example, modify the description text to "I want to add a touch of Picasso style."

[0039] Step S220: Based on the modified description text corresponding to the first design image, determine a reference image that matches the modified description text.

[0040] For example, a reference image matching the modified description text is obtained from an image library, such as Baidu Baike or Wikipedia. The reference image can be one or multiple images. The reference image represents a second design style, and its content can be the same as or different from the content of the first design image.

[0041] Step S230: Based on the reference image, determine the second design image corresponding to the first design image.

[0042] The content of the second design image remains unchanged compared to the first design image. Specifically, based on the second design style of the reference image, the style of the first design image is modified to obtain a second design image that meets the customer's requirements.

[0043] In this embodiment, a first design image and its corresponding modification description text are first obtained. Based on the modification description text, a reference image matching the modification description text is determined. Then, based on the reference image, a second design image corresponding to the first design image is determined. This solution achieves intelligent modification of the first design image, unlike the method used by designers to repeatedly adjust the first design image based on client feedback. This solution saves time and labor costs associated with manual modification, reduces rework, and improves the efficiency of image design.

[0044] Figure 3 The diagram shown is a schematic flowchart illustrating the process of determining a second design image according to an exemplary embodiment of this application. Figure 2 Extending from the illustrated embodiment Figure 3 The illustrated embodiment will be described in detail below. Figure 3 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0045] like Figure 3 As shown in the embodiments of this application, determining the second design image corresponding to the first design image based on the reference image includes the following steps.

[0046] Step S310: Determine the sampling probability corresponding to the pixel in the reference image.

[0047] For example, there are N reference images, and the representation vectors corresponding to each of the N reference images are determined. The representation vector of each reference image can be represented in matrix form [h, w, c], where h is the height of the reference image, w is the width of the reference image, and c is the number of channels. Further, for the representation vector of each reference image, sampling is performed with a probability p, where p represents the sampling probability of a pixel in the representation vector of the reference image.

[0048] Step S320: Based on the sampling probability of the pixel in the reference image and the pixel value of the pixel in the first design image, determine the fused pixel value of the pixel in the first design image.

[0049] Specifically, an encoder is used to encode the first design image to obtain a representation vector corresponding to the first design image. The meaning of encoding is to transform an image visible to the human eye into numbers that a machine can understand and calculate. For example, embodiments of this application use an open-source pre-trained image encoder to encode the first design image, such as the image encoder in the open-source Contrastive Language-Image Pre-training (CLIP) model.

[0050] Furthermore, a preset sampling threshold is set, and sampling points in the reference image are determined based on the sampling probabilities corresponding to pixels in the reference image. Based on the sampling points in the reference image, the pixel values ​​of pixels in the first design image corresponding to the sampling point positions are determined. Based on the pixel values ​​of the sampling points and the pixel values ​​of pixels in the first design image corresponding to the sampling point positions, the fused pixel values ​​are determined.

[0051] For example, when the sampling probability is greater than a preset sampling threshold, the pixel corresponding to that sampling probability is considered a sampling point. Assuming the preset sampling threshold is 70%, and the sampling probability corresponding to the pixel at position [1, 1, c] is 80%, then the pixel at position [1, 1, c] is determined as a sampling point. Accordingly, the pixel value of the pixel at position [1, 1, c] in the first design image and the pixel value of the sampling point at position [1, 1, c] in the reference image are determined. Based on the pixel values ​​at position [1, 1, c] in the two images, the fused pixel value corresponding to the pixel at position [1, 1, c] in the first design image is determined. The value of the sampling probability p can be adjusted automatically; if more pixels in the reference image are to be sampled, the value of p can be increased.

[0052] Step S330: Determine the second design image corresponding to the first design image based on the fused pixel values ​​corresponding to the pixels in the first design image.

[0053] Specifically, for each sampling point in the reference image, the fused pixel value of the sampling point and the corresponding pixel in the first design image is calculated. Based on the newly obtained fused pixel value and the initial pixel value of the pixel in the first design image, the second design image corresponding to the first design image is obtained.

[0054] For example, the fused pixel value 'a' corresponds to position [1, 1, c] in the first design image, while no sampling point corresponds to position [2, 2, c] in the reference image. Therefore, the initial pixel value 'b' corresponds to position [2, 2, c] in the first design image. Based on the above method, all fused pixel values ​​and initial pixel values ​​corresponding to the first design image are determined. Then, based on all fused pixel values ​​and initial pixel values ​​corresponding to the first design image, the second design image corresponding to the first design image is determined.

[0055] This application describes in detail how to determine the second design image. Specifically, based on sampling points in the reference image and pixels in the first design image at corresponding positions in the reference image, the fused pixel value corresponding to each pixel is obtained. Then, the second design image is determined based on all the fused pixel values. The scheme in this application allows for the fusion of different image styles in the reference image. Furthermore, by adjusting the sampling probability p, the style diversity of the obtained second design image is adjusted. The style fusion method in this application is simpler, easier to implement, and has wider applicability.

[0056] Figure 4 The diagram shown is a schematic representation of a flowchart illustrating the process of determining fused pixel values ​​according to an exemplary embodiment of this application. Figure 3 Extending from the illustrated embodiment Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 3The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0057] like Figure 4 As shown in this embodiment, the pixels in the first design image are determined based on the sampling probability corresponding to the pixels in the reference image and the pixel values ​​of the pixels in the first design image.

[0058] The fused pixel value corresponding to the point includes the following steps.

[0059] Step S410: Determine the fusion weights corresponding to the pixels in the reference image.

[0060] Specifically, for each sampling point in the reference image, the corresponding fusion weight r is determined.

[0061] The value of r can be adjusted. For example, if you want to make the sampled pixels have a greater influence, you can increase the value of r.

[0062] Step S420: Based on the sampling probability and fusion weight of the pixel in the reference image and the pixel value of the pixel in the first design image, determine the fused pixel value of the pixel in the first design image.

[0063] Following the previous example, for the sampling point [1, 1, c] in the reference image, its corresponding fusion weight is 0.9, and the pixel value corresponding to the sampling point [1, 1, c] is 26. [1, 1, c] in the first design image

[0064] If the fusion weight of the pixel at position [1, 1, c] is 0.1 and the pixel value is 310, then the fusion pixel value of the pixel at position [1, 1, c] in the first design image is 0.9*26+0.1*310=54.4.

[0065] In this embodiment, when calculating the fused pixel value corresponding to the pixel point in the first design image, a fusion weight is introduced. This fusion weight can be adaptively adjusted according to requirements to change the style of the first design image and obtain the second design image. Similarly, the solution in this embodiment allows for more refined adjustment of the fused pixel value based on the sampling probability, better meeting customer needs.

[0066] Figure 5 The following is a flowchart of determining a second design image provided by another exemplary embodiment of this application.

[0067] Schematic diagram. Figure 3 Extending from the illustrated embodiment Figure 5 The embodiment shown below will be described in detail in 5 aspects. Figure 5 The illustrated embodiments and Figure 3The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0068] like Figure 5 As shown in the embodiment of this application, determining the second design image corresponding to the first design image based on the fused pixel values ​​corresponding to the pixels in the first design image includes the following steps.

[0069] Step S510: Based on the fused pixel values ​​corresponding to the pixels in the first design image, generate a modified image corresponding to the first design image.

[0070] Specifically, based on all the fused pixel values ​​corresponding to the pixels in the first design image and all the initial pixel values, the modified image corresponding to the first design image is obtained.

[0071] Step S520: Determine the representation vector of the modified image and the representation vector of the modified description text.

[0072] For example, an encoder is used to encode the modified descriptive text to obtain the representation vector corresponding to the modified descriptive text. Specifically, the encoder must have the ability to map the semantic space of the modified descriptive text to the space of the modified image. Alternatively, open-source models can be used to determine the representation vector of the modified descriptive text, such as the text encoder in DALL-E2 and its diffusion prior model.

[0073] Step S530: Determine the second design image based on the representation vector of the modified image and the representation vector of the modified description text.

[0074] For example, the vector similarity between the modified image representation vector and the modified descriptive text representation vector is calculated, for example, the vector similarity is cosine similarity.

[0075] In one implementation, if the vector similarity between the representation vector of the modified image and the representation vector of the modified descriptive text is less than a preset similarity threshold, the sampling probability corresponding to the pixel in the reference image is adjusted; based on the adjusted sampling probability and the pixel value of the pixel in the modified image, a second design image is determined.

[0076] For example, if the similarity between the representation vector of the modified image and the representation vector of the modified description text is less than a preset similarity threshold t, it means that the modified image does not match the customer's modification request (modified description text). In this case, the sampling probability p corresponding to the pixels in the reference image needs to be adjusted to obtain a sampling probability q. Based on the sampling probability q and the pixel values ​​of the pixels in the modified image, the modified image is determined. If the similarity is less than the preset similarity threshold, the sampling probability q of the pixels in the reference image is adjusted again, and the steps for determining the fused pixel values ​​in the above embodiments are repeated until the number of repetitions exceeds n, or the similarity between the representation vector of the modified image and the representation vector of the modified description text is greater than or equal to the preset similarity threshold t.

[0077] After determining the sampling probability q, the sampling points of the reference image are determined based on the sampling probability q and the preset sampling threshold. Preferably, after determining the sampling points of the reference image, the fusion weights of the sampling points are further determined, and the modified image is redefined based on the fusion weights of the sampling points of the reference image and the pixel values ​​of the pixels in the modified image.

[0078] In another implementation, if the vector similarity between the representation vector of the modified image and the representation vector of the modified descriptive text is greater than or equal to a preset similarity threshold, then the modified image is determined as the second design image.

[0079] Specifically, if the similarity between the representation vector of the modified image and the representation vector of the modified description text is greater than or equal to the preset similarity threshold t, it means that the modified image is consistent with the customer's modification opinion (modified description text). In this case, the modified image is determined as the second design image.

[0080] For example, once a satisfactory representation vector of the second design image is obtained, the second design image can be generated using an image decoder. For example, an open-source image decoder, such as the DALL-E2 diffusion decoder, can be used.

[0081] In this embodiment, after obtaining the modified image, the similarity between the representation vector of the modified image and the representation vector of the modified description text is further determined. Based on the similarity result, it is determined whether the modified image is a second design image that meets the customer's requirements. This embodiment provides a double guarantee for the satisfaction of the second design image. Furthermore, if the similarity between the representation vector of the modified image and the representation vector of the modified description text does not meet a preset similarity threshold, the sampling probability is readjusted, and the modified image is re-determined based on the adjusted sampling probability and the modified image obtained in the previous round. This differs from re-determining the modified image based on the first design image. This embodiment allows for faster acquisition of modified images that meet the similarity threshold.

[0082] Figure 6The diagram shown is a schematic flowchart illustrating the determination of a reference image according to an exemplary embodiment of this application. Figure 2 Extending from the illustrated embodiment Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0083] like Figure 6 As shown in the embodiment of this application, determining a reference image that matches the modified description text based on the modified description text corresponding to the first design image includes the following steps.

[0084] Step S610: Determine the representation vector of the modified description text.

[0085] For example, a pre-trained text encoder can be used to encode modified descriptive text, resulting in a representation vector corresponding to the modified descriptive text. For instance, the text encoder is a Bidirectional Encoder Representations (BERT). BERT considers the context from both sides (left and right) of a word, which helps to better understand the context of the word. Furthermore, BERT is designed to be capable of multi-task learning; that is, BERT can perform different natural language processing tasks simultaneously.

[0086] Step S620: Obtain at least one candidate image and the representation vector of the image description text for each candidate image.

[0087] For example, an image library is pre-built, containing at least one candidate image and corresponding image description text for each candidate image. The candidate images and their corresponding image description texts can be obtained from any source, such as Baidu Encyclopedia, Wikipedia, books, etc. The image description text refers to text that can describe the design concept, design style, or design content of the image.

[0088] Suppose that for the candidate image - Van Gogh's Sunflowers, the corresponding descriptive text is: "Through this series of works, Van Gogh expressed his understanding of life to the world and showed his unique spiritual world. This series of works also conveys the following message: treat family with gratitude, treat others with kindness, treat friends with sincerity, treat work with dedication, treat life with gratitude, and enjoy art with an appreciative heart, just like the sunflowers blooming before our eyes."

[0089] For the image description text corresponding to the candidate image, a text encoder is used to obtain its corresponding representation vector. For example, the text encoder is BERT.

[0090] Step S630: Based on the vector similarity between the representation vector of the modified description text and the representation vector of the image description text of at least one candidate image, a reference image matching the modified description text is determined from the at least one candidate image.

[0091] Specifically, the similarity between the representation vector of the image description text for each reference image and the representation vector of the modified description text is calculated; for example, the similarity is cosine similarity. Then, the similarity scores are sorted from largest to smallest, and the candidate images corresponding to the top N image description texts in the sorting results are selected as reference images to match the modified description text, where N is a positive integer greater than 1. For example, if the image library contains 1000 candidate images, the top 10 image description texts most similar to the modified description text are selected from the sorting results, and the candidate images corresponding to these 10 image description texts are used as reference images.

[0092] In this embodiment, the representation vector of the modified description text is first determined. Further, the representation vector corresponding to the image description text of the candidate image is determined. By comparing the similarity between the representation vector of the modified description text and the representation vector of the image description text of the candidate image, a reference image is determined from the candidate images. That is, by using the same text dimension representation method for both the modified description text and the image description text, the reference image can be determined more accurately and quickly by comparing their similarity.

[0093] Exemplary device

[0094] The above text combined Figures 2 to 6 The following describes in detail the embodiments of the image design method of this application, in conjunction with... Figure 7 This application describes in detail embodiments of the image design apparatus. It should be understood that the descriptions of the image design method embodiments correspond to the descriptions of the image design apparatus embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0095] Figure 7 The diagram shown is a structural schematic of an image design apparatus provided in an exemplary embodiment of this application. Figure 7 As shown, the image design apparatus 70 provided in this application embodiment includes:

[0096] The acquisition module 710 is used to acquire the first design image and the corresponding modification description text of the first design image;

[0097] The first determining module 720 is used to determine a reference image that matches the modified description text based on the modified description text corresponding to the first design image;

[0098] The second determining module 730 is used to determine the second design image corresponding to the first design image based on the reference image.

[0099] In one embodiment of this application, the second determining module 730 is further configured to: determine the sampling probability corresponding to a pixel in the reference image; determine the fused pixel value corresponding to a pixel in the first design image based on the sampling probability corresponding to a pixel in the reference image and the pixel value of a pixel in the first design image; and determine the second design image corresponding to the first design image based on the fused pixel value corresponding to a pixel in the first design image.

[0100] In one embodiment of this application, the second determining module 730 is further configured to: determine the fusion weight corresponding to the pixel in the reference image; and determine the fusion pixel value corresponding to the pixel in the first design image based on the sampling probability and fusion weight corresponding to the pixel in the reference image and the pixel value of the pixel in the first design image.

[0101] In one embodiment of this application, the second determining module 730 is further configured to: generate a modified image corresponding to the first design image based on the fused pixel values ​​corresponding to the pixels in the first design image; determine the representation vector of the modified image and the representation vector of the modified description text; and determine a second design image based on the representation vector of the modified image and the representation vector of the modified description text.

[0102] In one embodiment of this application, the second determining module 730 is further configured to: if the vector similarity between the representation vector of the modified image and the representation vector of the modified description text is less than a preset similarity threshold, adjust the sampling probability corresponding to the pixel in the reference image; and determine the second design image based on the adjusted sampling probability and the pixel value of the pixel in the modified image.

[0103] In one embodiment of this application, the second determining module 730 is further configured to determine the modified image as the second design image if the vector similarity between the representation vector of the modified image and the representation vector of the modified description text is greater than or equal to a preset similarity threshold.

[0104] In one embodiment of this application, the first determining module 720 is further configured to: determine the representation vector of the modified description text; obtain at least one candidate image and the representation vector of the image description text of each of the at least one candidate image; and determine a reference image that matches the modified description text from the at least one candidate image based on the vector similarity between the representation vector of the modified description text and the representation vector of the image description text of each of the at least one candidate image.

[0105] Below, for reference Figure 8 This describes an electronic device according to embodiments of the present application. Figure 8 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0106] like Figure 8As shown, the electronic device 80 includes one or more processors 801 and memory 802.

[0107] The processor 801 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 80 to perform desired functions.

[0108] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the image design methods of the various embodiments of this application described above and / or other desired functions. The computer-readable storage medium may also store various contents such as a first design image, modified description text, a second design image, a reference image, etc.

[0109] In one example, the electronic device 80 may also include an input device 803 and an output device 804, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0110] The input device 803 may include, for example, a keyboard, a mouse, etc.

[0111] The output device 804 can output various information to the outside, including a first design image, modified description text, a second design image, and a reference image. The output device 804 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0112] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 80 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 80 may include any other suitable components depending on the specific application.

[0113] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image design methods according to various embodiments of this application as described above.

[0114] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0115] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image design methods according to various embodiments of this application described above.

[0116] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0118] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0119] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An image design method, characterized in that, include: Obtain the first design image and the corresponding modified description text; Based on the modified description text corresponding to the first design image, a reference image matching the modified description text is determined; Determine the sampling probability corresponding to each pixel in the reference image; wherein, the sampling probability corresponding to each pixel in the reference image can be adjusted individually; Based on the sampling probability of the pixel in the reference image and the pixel value of the pixel in the first design image, the fused pixel value of the pixel in the first design image is determined; Based on the fused pixel values ​​corresponding to the pixels in the first design image, a modified image corresponding to the first design image is generated. Determine the representation vector of the modified image and the representation vector of the modified description text; The second design image is determined based on the representation vector of the modified image and the representation vector of the modified description text.

2. The method according to claim 1, characterized in that, The step of determining the fused pixel value corresponding to the pixel in the first design image based on the sampling probability corresponding to the pixel in the reference image and the pixel value of the pixel in the first design image includes: Determine the fusion weights corresponding to the pixels in the reference image; Based on the sampling probability and fusion weight of the pixels in the reference image and the pixel value of the pixels in the first design image, the fused pixel value of the pixels in the first design image is determined.

3. The method according to claim 1, characterized in that, Determining the second design image based on the representation vector of the modified image and the representation vector of the modified description text includes: If the vector similarity between the representation vector of the modified image and the representation vector of the modified description text is less than a preset similarity threshold, then the sampling probability corresponding to the pixel in the reference image is adjusted. The second design image is determined based on the adjusted sampling probability and the pixel values ​​of the pixels in the modified image.

4. The method according to claim 1, characterized in that, Determining the second design image based on the representation vector of the modified image and the representation vector of the modified description text includes: If the vector similarity between the representation vector of the modified image and the representation vector of the modified description text is greater than or equal to a preset similarity threshold, then the modified image is determined as the second design image.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining a reference image that matches the modified description text based on the modified description text corresponding to the first design image specifically includes: Determine the representation vector of the modified description text; Obtain at least one candidate image and the representation vector of the image description text for each of the at least one candidate image; Based on the vector similarity between the representation vector of the modified description text and the representation vector of the image description text of each of the at least one candidate image, a reference image matching the modified description text is determined from the at least one candidate image.

6. An image design device, characterized in that, include: The acquisition module is used to acquire the first design image and the modification description text corresponding to the first design image; The first determining module is used to determine a reference image that matches the modified description text based on the modified description text corresponding to the first design image; The second determining module is used to determine the sampling probability corresponding to each pixel in the reference image; wherein the sampling probability corresponding to each pixel in the reference image can be adjusted individually; based on the sampling probability corresponding to each pixel in the reference image and the pixel value of each pixel in the first design image, the module determines the fused pixel value corresponding to each pixel in the first design image; based on the fused pixel value corresponding to each pixel in the first design image, the module generates a modified image corresponding to the first design image; the module determines the representation vector of the modified image and the representation vector of the modified description text; and based on the representation vector of the modified image and the representation vector of the modified description text, the module determines a second design image.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image design method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is used to execute the image design method according to any one of claims 1 to 5.

Images