Image generation method and device

Abstract

The invention discloses an image generation method and device, and relates to the technical field of image processing and artificial intelligence. A specific embodiment of the method comprises the following steps: generating a first image pixel matrix containing a display background for an article based on description information by using an image generation model; using the control network model to generate a second image pixel matrix containing the object shape; generating a weight pixel matrix containing object edge pixels based on the object base map; multiplying object edge pixels contained in the weight pixel matrix by pixel values of corresponding positions of the second image pixel matrix to obtain a third image pixel matrix after edge weighting; and combining the first image pixel matrix and the third image pixel matrix to generate an article display image with a display background. According to the embodiment of the invention, the problem that the appearance of the article in the generated display background image is changed or abnormal in the prior art is solved, and the display information accuracy and the display effect of the article in the display background image are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing and artificial intelligence technology, and in particular to an image generation method and device. Background Art

[0002] Image generation technology is currently being used more and more in different scenarios. For example, image generation technology is used to generate display images with display backgrounds for given items, and to build different scenes for different items. By generating background display images with higher visual effects for items, compared with unmodified independent item images, the display effect of the items and the user experience when browsing item information can be improved.

[0003] When existing image generation technology generates a background display image for an object, the objects included in the background display image may have the problem of changed appearance. For example, an extra part extends from the original edge of the object, resulting in a decrease in the display accuracy and authenticity of the object in the background display image, affecting the display effect of the object background image on the object. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an image generation method and device, which can generate a first image pixel matrix including a display background for an object based on description information using an image generation model; generate a second image pixel matrix including the shape of the object using a control network model; and generate a weight pixel matrix including the edge pixels of the object based on the object base map; multiply the object edge pixels included in the weight pixel matrix with the pixel values ​​of the corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix; combine the first image pixel matrix and the third image pixel matrix to generate an object display image with a display background. The embodiment of the present invention overcomes the problem that the appearance of the object in the display background image generated in the prior art is changed or abnormal, and improves the accuracy of the display information and the display effect of the object in the display background image.

[0005] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, there is provided an image generation method, characterized in that it includes: obtaining description information of a display background of an object, and using an image generation model to generate a first image pixel matrix including the display background for the object based on the description information; obtaining an object base map of the object, and using a control network model to generate a second image pixel matrix based on the object base map; wherein the object base map indicates the shape of the object; based on the object base map, generating a weighted pixel matrix including object edge pixels of the object; multiplying the object edge pixels included in the weighted pixel matrix with the pixel values ​​at corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix; and combining the first image pixel matrix and the third image pixel matrix to generate an object display image with a display background.

[0006] Optionally, based on the object base map, a weighted pixel matrix containing object edge pixels of the object is generated, including: using an image processing model to extract a transparency layer from the object base map; generating a base map pixel matrix of the object according to the transparency layer; processing multiple shape pixels indicating the shape of the object contained in the base map pixel matrix into edge pixels indicating the edge of the object, and obtaining a weighted pixel matrix containing multiple edge pixels.

[0007] Optionally, multiple shape pixels indicating the shape of the object contained in the base map pixel matrix are processed into edge pixels indicating the edge of the object to obtain a weighted pixel matrix containing multiple edge pixels, including: using a preset expansion degree value, performing an expansion operation on the multiple shape pixels to obtain a first pixel matrix containing expanded shape pixels; using a preset corrosion degree value, performing an corrosion operation on the multiple shape pixels to obtain a second pixel matrix containing the corrosion shape pixels; subtracting the pixel values ​​at corresponding positions of the second pixel matrix from those of the first pixel matrix to obtain a weighted pixel matrix containing multiple edge pixels.

[0008] Optionally, obtaining a weighted pixel matrix including a plurality of edge pixels includes: performing a Gaussian blur operation on pixel values ​​corresponding to matrix elements of the weighted pixel matrix, and using the pixel matrix obtained after the Gaussian blur operation as the weighted pixel matrix.

[0009] Optionally, based on the object base map, a weight pixel matrix containing object edge pixels of the object is generated, including: constructing a first parameter matrix; wherein the matrix structures of the first parameter matrix, the weight pixel matrix and the second image pixel matrix are the same; setting the element value of each matrix element of the first parameter matrix to a first default value; multiplying each pixel value contained in the weight pixel matrix with the first default value of the corresponding position in the first parameter matrix to obtain a first weight matrix; and using the first weight matrix as the weight pixel matrix.

[0010] Optionally, the image generation method further includes: constructing a second parameter matrix; wherein the matrix structure of the second parameter matrix is ​​the same as that of the first parameter matrix; setting each matrix element of the second parameter matrix to a second default value; superimposing each pixel value contained in the first weight matrix with the second default value at the corresponding position of the second parameter matrix to obtain a second weight matrix; and using the second weight matrix as a weight pixel matrix.

[0011] Optionally, the image generation method further includes: in response to changing the weights of the pixels indicating the edge of the object in the third image pixel matrix, obtaining a plurality of edge matrix elements indicating the edge of the object from the weight pixel matrix; changing the first default value corresponding to the position of the edge matrix element in the first parameter matrix according to the edge matrix elements, and / or changing the second default value corresponding to the position of the edge matrix element in the second parameter matrix; and using the first weight matrix after changing the matrix elements and / or the second weight matrix after changing the matrix elements as the weight pixel matrix.

[0012] To achieve the above object, according to a second aspect of an embodiment of the present invention, an image generation device is provided, characterized in that it comprises: a generation module, a processing module and a target module; wherein:

[0013] A generation module is used to obtain description information of a display background of an item, and generate a first image pixel matrix for the item based on the description information of the display background using an image generation model; obtain an item base map of the item, and generate a second image pixel matrix based on the item shape indicated by the item base map using a control network model;

[0014] A processing module is used to generate a weighted pixel matrix containing object edge pixels of the object based on the object base map; multiply the object edge pixels contained in the weighted pixel matrix by the pixel values ​​of corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix;

[0015] The target module is used to combine the first image pixel matrix and the third image pixel matrix to generate an object display image with a display background.

[0016] Optionally, an image generating device is used to generate a weighted pixel matrix containing object edge pixels of the object based on the object base map, including: using an image processing model to extract a transparency layer from the object base map; generating a base map pixel matrix of the object based on the transparency layer; processing multiple shape pixels indicating the shape of the object contained in the base map pixel matrix into edge pixels indicating the edge of the object, and obtaining a weighted pixel matrix containing multiple edge pixels.

[0017] Optionally, the image generating device is used to process multiple shape pixels indicating the shape of the object contained in the base map pixel matrix into edge pixels indicating the edge of the object to obtain a weighted pixel matrix containing multiple edge pixels, including: using a preset expansion degree value, performing an expansion operation on the multiple shape pixels to obtain a first pixel matrix containing expanded shape pixels; using a preset corrosion degree value, performing an corrosion operation on the multiple shape pixels to obtain a second pixel matrix containing the corrosion shape pixels; subtracting the pixel values ​​at corresponding positions of the second pixel matrix from those of the first pixel matrix to obtain a weighted pixel matrix containing multiple edge pixels.

[0018] Optionally, the image generating device is used to obtain a weighted pixel matrix including multiple edge pixels, including: performing a Gaussian blur operation on pixel values ​​corresponding to matrix elements of the weighted pixel matrix, and using the pixel matrix obtained after the Gaussian blur operation as the weighted pixel matrix.

[0019] Optionally, an image generating device is used to generate a weighted pixel matrix of object edge pixels including objects based on an object base map, including: constructing a first parameter matrix; wherein the matrix structures of the first parameter matrix, the weighted pixel matrix and the second image pixel matrix are all the same; setting the element value of each matrix element of the first parameter matrix to a first default value; multiplying each pixel value included in the weighted pixel matrix by the first default value of the corresponding position in the first parameter matrix to obtain a first weight matrix; and using the first weight matrix as the weighted pixel matrix.

[0020] Optionally, the image generating device is further used to construct a second parameter matrix; wherein the matrix structure of the second parameter matrix is ​​the same as that of the first parameter matrix; each matrix element of the second parameter matrix is ​​set to a second default value; each pixel value contained in the first weight matrix is ​​superimposed with the second default value at the corresponding position of the second parameter matrix to obtain a second weight matrix; and the second weight matrix is ​​used as a weight pixel matrix.

[0021] Optionally, the image generating device is further used to obtain multiple edge matrix elements indicating object edge pixels from the weight pixel matrix in response to changing the weights of the edge pixels indicating the object in the third image pixel matrix; change the first default value corresponding to the edge matrix element position in the first parameter matrix according to the edge matrix elements, and / or change the second default value corresponding to the edge matrix element position in the second parameter matrix; and use the first weight matrix after changing the matrix elements and / or the second weight matrix after changing the matrix elements as the weight pixel matrix.

[0022] To achieve the above-mentioned purpose, according to the third aspect of an embodiment of the present invention, there is provided an electronic device for image generation, characterized in that it includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any method as described in the above-mentioned image generation method.

[0023] To achieve the above-mentioned purpose, according to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, a method as described in any one of the above-mentioned image generation methods is implemented.

[0024] To achieve the above-mentioned purpose, according to a fifth aspect of an embodiment of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, any one of the above-mentioned image generation methods is implemented.

[0025] One embodiment of the above invention has the following advantages or beneficial effects: it is possible to generate a first image pixel matrix including a display background for an object based on description information using an image generation model; generate a second image pixel matrix including the shape of the object using a control network model; and generate a weight pixel matrix including object edge pixels based on the object base map; multiply the object edge pixels included in the weight pixel matrix by the pixel values ​​of the corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix; and generate an object display image with a display background by combining the first image pixel matrix and the third image pixel matrix. The embodiment of the present invention overcomes the problem of changes or anomalies in the appearance of the object in the display background image generated in the prior art, and overcomes the problem of disproportion of the object in the background content in the object display image generated in the prior art, thereby improving the accuracy of the display information and the display effect of the object in the display background image.

[0026] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0028] Figure 1 is a flowchart of an image generating method provided by an embodiment of the present invention;

[0029] Figure 2 This is an example diagram of an existing image generation effect provided by an embodiment of the present invention;

[0030] Figure 3A is a schematic diagram of a flow chart of generating a weighted pixel matrix provided by an embodiment of the present invention;

[0031] Figure 3B is a schematic diagram of determining object edge pixels based on an object image matrix provided by an embodiment of the present invention;

[0032] Figure 4A This is an example of image generation based on an image generation method provided by an embodiment of the present invention. Figure 1 ;

[0033] Figure 4B This is an example of image generation based on an image generation method provided by an embodiment of the present invention. Figure 2 ;

[0034] Figure 5 is a structural schematic diagram of an image generating device provided by an embodiment of the present invention;

[0035] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0036] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0039] Generating a background image for an item is an application scenario of using image generation technology. The exquisite data effects contained in the item background image can increase the user's browsing rate and click-through rate for the item, thereby increasing the exposure rate of the item and improving the user's experience. Among them, image generation technology is a branch of the AIGC (Artificial Intelligence Generated Content) field.

[0040] Usually, the object image can be provided by the party that needs the object background image, for example Figure 2 As shown, Figure 201A is the image of the object to be processed (kettle transparent background image), and Figure 201B is the object (kettle) display image generated according to the description information of the application scenario information (for example: kitchen, countertop, etc.). As shown in Figure 201B, an extra layer is added to the lid part of the "kettle". It can be seen that the existing graphics generation model has the problem of changing the shape of the object due to the low accuracy of generating the contour information of the object, which leads to the problem of inconsistency between the actual object and the object image, affecting the display effect of the object background image on the object and affecting the user experience.

[0041] In view of this, if Figure 1As shown, an embodiment of the present invention provides an image generation method, which may include the following steps:

[0042] Step S101: Obtain description information of a display background of an object, and generate a first image pixel matrix including the display background for the object based on the description information using an image generation model; obtain an object base map of the object, and generate a second image pixel matrix based on the object base map using a control network model; wherein the object base map indicates the shape of the object.

[0043] Step S102: Based on the object base map, generate a weighted pixel matrix containing object edge pixels; multiply the object edge pixels contained in the weighted pixel matrix by the pixel values ​​at corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix.

[0044] Specifically, the object is the target to be displayed, and the display background is the background part added to the display image based on the object image (for example, a display image with a kitchen countertop background is generated for a kettle). In the embodiment of the present invention, the image of the object can be a picture showing the shape and content of the object provided by the demander of the object background image, wherein the object image can be processed into an object base image containing a single background color (such as Figure 2 The kettle image shown in 201A).

[0045] The image generation model provided in the embodiment of the present invention may be any image generation model implemented based on AIGC (Artificial Intelligence Generated Content).

[0046] Further, the description information of the display background of the item is obtained, and the image generation model is used to generate a first image pixel matrix containing the display background for the item based on the description information; specifically, the description information of the display background of the item is obtained; and the image generation model is used to generate a first image pixel matrix containing the display background based on the description information. The item base map of the item is obtained, and the control network model is used to generate a second image pixel matrix based on the item base map; wherein the item base map indicates the shape of the item.

[0047] Specifically, the description information for the item image and the display background is set according to the actual item and the background scene to be displayed. For example, the description information includes a kitchen counter, a kettle, etc. The image generation model implemented based on generative artificial intelligence parses the description information, and generates an item display image (such as a kitchen counter with a display background) containing the item on the kitchen counter according to the semantics of the description information. Figure 4A 401B in FIG. 401B ), wherein the first image pixel matrix is ​​a matrix output by the model formed by encoding the image, for example, the first image pixel matrix is ​​represented as Wherein, H represents height, W represents width, and H*W represents the width and height of the image. After the first image pixel matrix is ​​generated based on the description information using the image generation model, the existing method generates the item display image according to the first image pixel matrix.

[0048] It is understandable that the objects and backgrounds of the present invention are examples, and the specific objects, backgrounds, description information, generation conditions, etc. of the present invention are not limited.

[0049] Furthermore, an item base map of the item is obtained, and a second image pixel matrix is ​​generated based on the item base map using a control network model, wherein the control network model is, for example, an image control plug-in (ControlNET or T2I-Adapter, etc.), which is used to generate a second image pixel matrix according to the item base map of the item, and this step further performs adjustment and control on the first image pixel matrix generated by the image generation model, thereby improving the visual effect and authenticity of the item image with the display background. Among them, the party demanding the item display image can provide generation condition information (such as the shape of the item, the key points of the item, the segmentation map, the item location information, etc.). In an embodiment of the present invention, an item base map of the item is obtained, and the item base map indicates the shape of the item (for example, Figure 2 201A in FIG. 201A ), a second image pixel matrix is ​​generated based on the object base map using a control network model. For example, the second image pixel matrix is ​​represented as Among them, H represents height, W represents width, and H*W represents the width and height of the image corresponding to the matrix.

[0050] Furthermore, the first image pixel matrix and the second image pixel matrix are combined to generate an item display image with a display background. For example, the result of superimposing the positions corresponding to the matrix elements of the first image pixel matrix and the second image pixel matrix is ​​represented by formula (1), and decoding and other operations are performed on the superimposed matrix to obtain the final item display image.

[0051] Y=X1+X2 (1)

[0052] The display background generation result of the object display image contained in the image pixel matrix generated by the image generation model is adjusted through the object background map and the control network model, thereby improving the controllability and accuracy of the content contained in the object display image.

[0053] Further preferably, in an embodiment of the present invention, a weighted pixel matrix containing object edge pixels of the object is generated based on the object base map; the object edge pixels contained in the weighted pixel matrix are multiplied by the pixel values ​​of corresponding positions in the second image pixel matrix to obtain an edge-weighted third image pixel matrix.

[0054] Specifically, the embodiment of the present invention processes the edge pixels of the object and increases the weight of the edge pixels, thereby adjusting the contour range (edge ​​portion) of the object image in the object display image, so as to overcome the problem of changes in the object shape in the object background display image obtained by the existing image generation processing method.

[0055] In an embodiment of the present invention, a weighted pixel matrix including object edge pixels of the object is generated based on the object base map; and the weight of the object edge in the object display image is adjusted by the weighted pixel matrix.

[0056] Specifically, based on formula (1), formula (2) formed by introducing the weight pixel matrix is ​​as follows:

[0057] Y=X1+(W′*A+B)*X2 (2)

[0058] Wherein, X1 represents the first weight pixel matrix generated based on the description information using the image generation model, and X2 represents the second image pixel matrix generated based on the object base map using the control network model; W', W'*A and W'*A+B all represent weight pixel matrices; wherein the weight pixel matrix has the same matrix structure as the first image pixel matrix and the second image pixel matrix (that is, it has the same number of rows and columns, and the positions of each matrix element have a corresponding relationship). Wherein, the contour image matrix is ​​used to adjust the weight of the object contour in the object display image. A and B are variable matrices. In one embodiment of the present invention, the pixel values ​​in the A and B matrices are set to 1 by default, and the initial values ​​of each element in the W' matrix are set to 0.

[0059] Combine the following Figure 3A A schematic diagram of a process for generating a weighted pixel matrix W' is shown; and Figure 3B The steps shown are to generate a weighted pixel matrix W' based on the instructions for determining object edge pixels in the object image matrix:

[0060] like Figure 3A The steps shown are as follows:

[0061] Step S301: using an image processing model, extracting a transparency layer from the object base map.

[0062] Step S302: Generate a base map pixel matrix of the object according to the transparency layer.

[0063] Specifically, when the object image provided by the party demanding the object background image contains irrelevant background, the image processing model can be used to extract the object separately to obtain the object base map; further, based on the original matrix of the object base map, the base map pixel matrix corresponding to the transparency layer is separated.

[0064] Step S303: performing an expansion operation on a plurality of shape pixels using a preset expansion degree value to obtain a first pixel matrix including expanded shape pixels.

[0065] Step S304: performing an erosion operation on a plurality of shape pixels using a preset erosion degree value to obtain a second pixel matrix including eroded shape pixels.

[0066] Step S305: Subtract the pixel values ​​at corresponding positions of the second pixel matrix from those of the first pixel matrix to obtain a weighted pixel matrix W' containing a plurality of edge pixels.

[0067] Specifically, the description of steps S301 to S305 is as follows: Figure 3B 301 to 304 are taken as examples, where FIG301 is an example of a transparent base map (with a base map pixel matrix) of an object (kettle), FIG302 is a first pixel matrix (corresponding to the expanded image) containing expanded shape pixels by dilating a plurality of shape pixels using a preset expansion degree value, and FIG303 is a second pixel matrix (corresponding to the expanded image) containing the expanded shape pixels by corroding a plurality of shape pixels using a preset corrosion degree value. FIG304 is a weighted pixel matrix (corresponding to the image) containing a plurality of edge pixels by subtracting the pixel values ​​at the corresponding positions of the second pixel matrix from the first pixel matrix. The dilation operation and the corrosion operation can be completed using the processing functions contained in the image processing model OpenCV. Furthermore, a preset expansion degree value can be set according to actual needs. For example, the preset expansion degree value is the expanded convolution kernel value (i.e., kernel size), which is set to the short side of the image * 0.0625 (i.e., if the image is 1000*600, the convolution kernel value is 600*0.0625=37.5, and the integer part 37 is taken); similarly, the corrosion degree value is set. It can be understood that the preset expansion degree value and the preset corrosion degree value can be the same or different to adjust the style of the contour image in the contour image. For the display effect of the object, the preset expansion degree value and the preset corrosion degree value can also be adjusted multiple times to achieve the optimal display effect.

[0068] The description of the above steps S301 to S305 is: that is, based on the object base map, a weighted pixel matrix containing the object edge pixels is generated, including: using an image processing model to extract a transparency layer from the object base map; generating a base map pixel matrix of the object according to the transparency layer; processing a plurality of shape pixels indicating the shape of the object contained in the base map pixel matrix into edge pixels indicating the edge of the object, and obtaining a weighted pixel matrix containing a plurality of edge pixels.

[0069] Furthermore, a plurality of shape pixels indicating the shape of the object contained in the base map pixel matrix are processed into edge pixels indicating the edge of the object to obtain a weighted pixel matrix containing a plurality of edge pixels, including: using a preset expansion degree value, performing an expansion operation on the plurality of shape pixels to obtain a first pixel matrix containing the expanded shape pixels; using a preset corrosion degree value, performing an corrosion operation on the plurality of shape pixels to obtain a second pixel matrix containing the corrosion shape pixels; subtracting the pixel values ​​at corresponding positions of the second pixel matrix from those of the first pixel matrix to obtain a weighted pixel matrix containing a plurality of edge pixels.

[0070] Further, preferably, obtaining a weighted pixel matrix including a plurality of edge pixels comprises: performing a Gaussian blur operation on pixel values ​​corresponding to matrix elements of the weighted pixel matrix, and using the pixel matrix obtained by the Gaussian blur operation as the weighted pixel matrix.

[0071] Specifically, a Gaussian blur operation is performed on the pixel values ​​corresponding to the matrix elements of the weight pixel matrix to obtain a pixel matrix (corresponding blurred effect image) obtained after the Gaussian blur operation as shown in FIG305, and the pixel matrix obtained after the Gaussian blur operation is used as the weight pixel matrix. The Gaussian blur process can make the weight effect of the weight pixel matrix on the second image pixel matrix smoothly transition, so that the overall generation result of the item display image is more natural.

[0072] Furthermore, the object edge pixels contained in the weighted pixel matrix are multiplied by the pixel values ​​of the corresponding positions of the second image pixel matrix to obtain the edge-weighted third image pixel matrix. Specifically, the weighted pixel matrix corresponds to the edge pixel information of the object. Therefore, by superimposing the weighted pixel matrix onto the second image pixel matrix generated based on the object base map, the control of the object edge pixels is strengthened, and the pixel weights far from the object edge pixels are reduced, thereby overcoming the problem of the object shape change caused by the edge extension of the object in the object display image, and improving the rationality of the ratio of the object to the background in the overall object display image.

[0073] Further preferably, the embodiment of the present invention can further enhance the control effect on the edge of the object by adjusting the matrix element values ​​of the A and B matrices in formula (2) and further changing the pixel weights of the edge pixels included in X2 (i.e., the second image pixel matrix). As shown in formula (2), A and B are variable parameter matrices.

[0074] Y=X1+(W′*A+B)*X2=X1+W′*A*X2+B*X2

[0075] Specifically, before adjusting the matrix element values ​​of the A and B matrices in formula (2) to further enhance the suppression effect on the edges of objects, the parameter matrices corresponding to A and B are constructed.

[0076] The method of generating a weight pixel matrix based on W', A, and B includes:

[0077] The first method is to use W' directly as the weight pixel matrix.

[0078] The second method: using the result matrix obtained by W'*A as the weight pixel matrix. That is, based on the object base map, generating a weight pixel matrix containing the object edge pixels, including: constructing a first parameter matrix; wherein the matrix structures of the first parameter matrix, the weight pixel matrix and the second image pixel matrix are the same; setting the element values ​​of each matrix element of the first parameter matrix to the first default value; multiplying each pixel value contained in the weight pixel matrix with the first default value of the corresponding position in the first parameter matrix to obtain a first weight matrix; using the first weight matrix as the weight pixel matrix. Specifically, only A is constructed (when the B matrix is ​​not constructed or the default pixel values ​​of the B matrix are all 0), wherein the matrix structures of the first parameter matrix, the weight pixel matrix and the second image pixel matrix are the same; further, the element value of each matrix element of the first parameter matrix is ​​set to a first default value (for example, set to 1); each pixel value contained in the weight pixel matrix is ​​multiplied by the first default value at the corresponding position in the first parameter matrix to obtain a first weight matrix (i.e., the result obtained by W'*A), which is the weight pixel matrix; further, the weight pixel matrix is ​​multiplied by the pixel values ​​at the corresponding positions of the second image pixel matrix (i.e., W′*A*X2) to obtain an edge-weighted third image pixel matrix (i.e., the result matrix obtained by W′*A*X2).

[0079] The third method: the result matrix obtained by (W'*A+B) is used as the weight pixel matrix. Construct a second parameter matrix; wherein the matrix structure of the second parameter matrix is ​​the same as that of the first parameter matrix; each matrix element of the second parameter matrix is ​​set to the second default value; each pixel value contained in the first weight matrix is ​​superimposed with the second default value of the corresponding position of the second parameter matrix to obtain a second weight matrix; and the second weight matrix is ​​used as the weight pixel matrix. Specifically, construct A and B, that is, construct a second parameter matrix (represented by B); wherein the matrix structure of the second parameter matrix is ​​the same as that of the first parameter matrix; each matrix element of the second parameter matrix is ​​set to the second default value (for example, 1); each pixel value contained in the first weight matrix is ​​superimposed with the second default value of the corresponding position of the second parameter matrix to obtain a second weight matrix, which is the weight pixel matrix (that is, the result matrix obtained by W'*A+B); further, multiply the weight pixel matrix with the pixel value of the corresponding position of the second image pixel matrix to obtain the edge-weighted third image pixel matrix, that is, the result matrix of (W'*A+B)*X2. It can be understood that the third image pixel matrix can also be obtained by calculating X1+W′*A*X2+B*X2.

[0080] Further preferably, the display effect with the display image generated by the method according to the embodiment of the present invention can generate a changed weight pixel matrix by changing the pixel default values ​​in the A and / or B matrices (i.e., the first default value for the A matrix and the second default value for the B matrix) through iterative experiments, so as to further strengthen (or reduce) the object edge weights contained in the third image pixel matrix, that is, in response to changing the weights of the object edge pixels indicating the object edge pixels in the third image pixel matrix, obtain a plurality of edge matrix elements indicating the object edge pixels from the weight pixel matrix; according to the edge matrix elements, change the first default value corresponding to the edge matrix element position in the first parameter matrix, and / or change the second default value corresponding to the edge matrix element position in the second parameter matrix; and use the first weight matrix after changing the matrix elements and / or the second weight matrix after changing the matrix elements as the weight pixel matrix.

[0081] If A and B are set too large, the weight result for X2 may have a greater impact on the overall item display image, resulting in a decrease in the image generation effect. Therefore, it is possible to adjust the pixel values ​​in the matrices of A and B according to the actual scene through iterative experiments. The method provided by the embodiment of the present invention does not require additional training for the image generation model, and the additional calculation amount for the weight image matrix is ​​O(1); the effect and efficiency of the image generation model in generating item display images with display backgrounds for items are improved.

[0082] Step S103: combining the first image pixel matrix and the third image pixel matrix to generate an item display image with a display background.

[0083] Specifically, according to the third image pixel matrix with object edge pixel weights generated through steps S101 to S102.

[0084] Furthermore, the first image pixel matrix and the third image pixel matrix are combined, and the pixel values ​​at corresponding positions of the matrices are superimposed (e.g., matrix elements) to obtain a target matrix. Decoding and other operations are performed through the superimposed target matrix (e.g., obtained by X1+(W′*A+B)*X2) to obtain the final object display image.

[0085] The following uses Figure 4A The difference between the item display image 401A generated by the existing method and the item display image 401B with a display background generated by the image generation method of the embodiment of the present invention is compared and explained.

[0086] like Figure 4A As shown in Figure 401A of FIG. 401A , there is an additional extended “handle” on the right side of the kettle, which changes the appearance of the kettle, and Figure 4AThe size of the shelf on the kitchen countertop and the container on the shelf shown in FIG. 401A is out of proportion with the size of the kettle, which makes the display effect of the kettle on the kitchen countertop poor. Figure 4A The item display image with display background shown in FIG. 401B in FIG. 401B adjusts the edge of the item by including the weighted pixel matrix of the item edge pixels, thereby overcoming the problem of the additional extension of the outline edge of the item (extra lid or handle, etc.) causing the shape to change, and making the ratio of the shelf on the countertop and the container and kettle on the shelf within a reasonable range. The accuracy of the information of the item in the display image of the item display background is improved, and the display effect is improved.

[0087] Further, using Figure 4B The difference between the item display image 402A generated by the existing method and the item display image 402B with a display background generated by the image generation method of the embodiment of the present invention is compared and explained.

[0088] like Figure 4B The lower edge of the container containing the real object shown in FIG. 402A in FIG. 402A has an additional extension, so that the preset plate becomes a bowl, that is, the shape of the container is changed, which affects the user's judgment of the food capacity in the container. The object display image with a display background generated by the image generation method of the embodiment of the present invention is as follows Figure 4B As shown in FIG. 402B , by adjusting the edge of the object through a weighted pixel matrix including the edge pixels of the object, the problem of the additional extension (extra bottom) of the outline edge of the object is overcome, the accuracy of the information of the object image in the object display image is improved, and the display effect is improved.

[0089] like Figure 5 As shown, the embodiment of the present invention provides an image generation device 500, including: a generation module 501, a processing module 502 and a target module 503; wherein,

[0090] The generation module 501 is used to obtain description information of the display background of the item, and generate a first image pixel matrix for the item based on the description information of the display background using an image generation model; obtain an item base map of the item, and generate a second image pixel matrix based on the item shape indicated by the item base map using a control network model;

[0091] The processing module 502 is used to generate a weighted pixel matrix including the object edge pixels of the object based on the object base map; multiply the object edge pixels included in the weighted pixel matrix by the pixel values ​​at corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix;

[0092] The target module 503 is used to combine the first image pixel matrix and the third image pixel matrix to generate an item display image with a display background.

[0093] An embodiment of the present invention also provides an electronic device for image generation, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0094] An embodiment of the present invention further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in any of the above embodiments is implemented.

[0095] An embodiment of the present invention further provides a computer program product, including a computer program, and when the computer program is executed by a processor, any of the above-mentioned image generating methods is implemented.

[0096] Figure 6 An exemplary system architecture 600 is shown to which the image generating method or the image generating apparatus according to the embodiment of the present invention can be applied.

[0097] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0098] Users can use terminal devices 601, 602, 603 to interact with server 605 through network 604 to receive or send messages, etc. Various client applications can be installed on terminal devices 601, 602, 603, such as electronic mall client applications, web browser applications, search applications, etc.

[0099] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting various client applications, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0100] The server 605 may be a server that provides various services, such as a background management server that provides support for client applications used by users using the terminal devices 601, 602, and 603. The background management server may process the received request for generating an item display image with a display background for the item image, and feed back the generated and processed item display image to the terminal device.

[0101] It should be noted that the image generating method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the image generating device is generally disposed in the server 605 .

[0102] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0103] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device suitable for implementing an embodiment of the present invention. Figure 7 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0104] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0105] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0106] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0107] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be set in a processor, for example, they may be described as: a processor includes an information extraction module, an image generation module, and an image adjustment module; wherein the names of these modules do not constitute a limitation on the modules themselves in certain circumstances, for example, the information extraction module may also be described as a "module for extracting object contour information from an object image".

[0110] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: obtaining description information of the display background of the item, using the image generation model to generate a first image pixel matrix containing the display background for the item based on the description information; obtaining the item base map of the item, using the control network model to generate a second image pixel matrix based on the item base map; wherein the item base map indicates the shape of the item; based on the item base map, generating a weight pixel matrix containing the item edge pixels of the item; multiplying the item edge pixels contained in the weight pixel matrix with the pixel values ​​of the corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix; combining the first image pixel matrix and the third image pixel matrix to generate an item display image with a display background.

[0111] The embodiments of the present invention can generate a first image pixel matrix including a display background for an object based on description information using an image generation model; generate a second image pixel matrix including the shape of the object using a control network model; and generate a weight pixel matrix including object edge pixels based on the object base map; multiply the object edge pixels included in the weight pixel matrix by the pixel values ​​of the corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix; and generate an object display image with a display background by combining the first image pixel matrix and the third image pixel matrix. The embodiments of the present invention overcome the problem of changes or anomalies in the appearance of the object in the display background image generated in the prior art, and overcome the problem of disproportion of the object in the background content in the object display image generated in the prior art, thereby improving the accuracy of the display information and the display effect of the object in the display background image.

[0112] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: include: Acquire description information of a display background of an item, and generate a first image pixel matrix including the display background for the item based on the description information using an image generation model; Acquire an object base map of the object, and generate a second image pixel matrix based on the object base map using a control network model; wherein the object base map indicates the shape of the object; Based on the object base map, generating a weighted pixel matrix including object edge pixels of the object; Multiplying the object edge pixels included in the weighted pixel matrix with the pixel values ​​at corresponding positions in the second image pixel matrix to obtain an edge-weighted third image pixel matrix; The first image pixel matrix and the third image pixel matrix are combined to generate an item display image with a display background.

2. The method according to claim 1, characterized in that The step of generating a weighted pixel matrix including edge pixels of the object based on the object base map comprises: Using an image processing model, extracting a transparency layer from the object base map; Generate a base map pixel matrix of the object according to the transparency layer; A plurality of shape pixels indicating the shape of the object contained in the base image pixel matrix are processed into edge pixels indicating the edge of the object, and a weight pixel matrix containing the plurality of edge pixels is obtained.

3. The method according to claim 2, characterized in that The step of processing a plurality of shape pixels indicating the shape of an object contained in the base image pixel matrix into edge pixels indicating the edge of the object to obtain a weighted pixel matrix containing a plurality of the edge pixels comprises: Using a preset expansion degree value, a dilation operation is performed on a plurality of the shape pixels to obtain a first pixel matrix including the expanded shape pixels; Using a preset corrosion degree value, an corrosion operation is performed on a plurality of the shape pixels to obtain a second pixel matrix including the corroded shape pixels; Subtract the pixel values ​​at corresponding positions of the second pixel matrix from those of the first pixel matrix to obtain a weighted pixel matrix including a plurality of the edge pixels.

4. The method according to claim 2, characterized in that: The step of obtaining a weighted pixel matrix including a plurality of edge pixels comprises: A Gaussian blur operation is performed on pixel values ​​corresponding to matrix elements of the weighted pixel matrix, and a pixel matrix obtained by the Gaussian blur operation is used as the weighted pixel matrix.

5. The method according to claim 1, characterized in that The step of generating a weighted pixel matrix including edge pixels of the object based on the object base map comprises: Constructing a first parameter matrix; wherein the matrix structures of the first parameter matrix, the weight pixel matrix and the second image pixel matrix are all the same; Setting the element value of each matrix element of the first parameter matrix to a first default value; Multiply each pixel value included in the weight pixel matrix by a first default value at a corresponding position in the first parameter matrix to obtain a first weight matrix; The first weight matrix is ​​used as the weight pixel matrix.

6. The method according to claim 5, characterized in that Further including: Constructing a second parameter matrix; wherein the matrix structure of the second parameter matrix is ​​the same as that of the first parameter matrix; Setting each matrix element of the second parameter matrix to a second default value; Superimposing each pixel value included in the first weight matrix with a second default value at a corresponding position of the second parameter matrix to obtain a second weight matrix; The second weight matrix is ​​used as the weight pixel matrix.

7. The method according to claim 6, characterized in that Further including: In response to changing the weights of the pixels indicating the edge of the object in the third image pixel matrix, obtaining a plurality of edge matrix elements indicating the edge of the object from the weighted pixel matrix; According to the edge matrix element, changing a first default value corresponding to the edge matrix element position in the first parameter matrix, and / or changing a second default value corresponding to the edge matrix element position in the second parameter matrix; The first weight matrix after changing the matrix elements and / or the second weight matrix after changing the matrix elements are used as the weight pixel matrix.

8. An image generating device, characterized in that: include: Generation module, processing module and target module; among them, The generation module is used to obtain description information of a display background of an item, and generate a first image pixel matrix for the item based on the description information of the display background using an image generation model; obtain an item base map of the item, and generate a second image pixel matrix based on an item shape indicated by the item base map using a control network model; The processing module is used to generate a weighted pixel matrix containing object edge pixels of the object based on the object base map; multiply the object edge pixels contained in the weighted pixel matrix by the pixel values ​​of corresponding positions of the second image pixel matrix to obtain an edge-weighted third image pixel matrix; The target module is used to generate an item display image with a display background by combining the first image pixel matrix and the third image pixel matrix.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images