Image element generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image element generation method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring image generation indication information; performing image generation on the image generation indication information based on a preset image generator to obtain a target image; performing mask extraction on the target image based on a preset image element extractor to obtain initial mask data; performing mask screening on the initial mask data to obtain candidate mask data; performing mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data; and performing element extraction on the target image according to the target mask data to obtain a target element image. According to the embodiment of the invention, the image element generation speed can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating image elements, an electronic device, and a storage medium. Background Art

[0002] Image elements are the basic parts of an image. For example, a cartoon character image includes hair, clothing, facial expressions and other elements. For example, in marketing activities in financial scenarios, image elements are usually needed to create marketing posters or marketing brochures, which are a combination of images and text. Currently, image elements are usually drawn manually, which is inefficient. Therefore, how to increase the generation speed of image elements has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a method and device for generating image elements, an electronic device and a storage medium, aiming to increase the generation speed of image elements.

[0004] To achieve the above objective, a first aspect of an embodiment of the present application provides a method for generating an image element, the method comprising:

[0005] Obtaining image generation instruction information;

[0006] Performing image generation on the image generation instruction information based on a preset image generator to obtain a target image;

[0007] Performing mask extraction on the target image based on a preset image element extractor to obtain initial mask data;

[0008] Performing mask screening on the initial mask data to obtain candidate mask data;

[0009] performing mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data;

[0010] Element extraction is performed on the target image according to the target mask data to obtain a target element image.

[0011] In some embodiments, the image generation instruction information includes text instruction information and image instruction information, and the step of performing image generation on the image generation instruction information based on a preset image generator to obtain a target image includes:

[0012] Performing text encoding on the text indication information based on the image generator to obtain text encoding information;

[0013] Performing image encoding on the image indication information based on the image generator to obtain image encoding information;

[0014] The target image is obtained by performing image generation on the text encoding information and the image encoding information based on the image generator.

[0015] In some embodiments, the step of generating an image based on the text encoding information and the image encoding information by the image generator to obtain the target image includes:

[0016] Generate a noise image based on the image generator to obtain an initial noise image;

[0017] Generate time features based on the image generator to obtain time step embedding features;

[0018] Performing noise encoding on the initial noise image based on the image generator to obtain a noise encoding feature map;

[0019] Based on the image generator, image generation is performed on the time step embedding feature, the noise coding feature map, the text coding information and the image coding information to obtain the target image.

[0020] In some embodiments, the performing noise encoding on the initial noise image based on the image generator to obtain a noise encoding feature map includes:

[0021] Performing image convolution on the initial noise map based on the image generator to obtain a convolution noise map;

[0022] Based on the image generator, activation processing is performed on the convolution noise map to obtain an activation noise map;

[0023] The activation noise map is normalized based on the image generator to obtain the noise coding feature map.

[0024] In some embodiments, the step of generating an image based on the image generator on the time step embedding feature, the noise coding feature map, the text coding information, and the image coding information to obtain the target image includes:

[0025] Based on the image generator, feature splicing is performed on the text encoding information and the image encoding information to obtain indication encoding information;

[0026] Performing attention feature fusion on the noise coding feature map based on the image generator and the indication coding information to obtain a fused coding feature map;

[0027] Performing time step embedding on the fused coding feature map based on the image generator and the time step embedding feature to obtain an embedded coding feature map;

[0028] Performing downsampling processing on the embedded coding feature map based on the image generator to obtain a downsampled coding feature map;

[0029] The downsampled encoded feature map is decoded based on the image generator to obtain the target image.

[0030] In some embodiments, performing mask screening on the initial mask data to obtain candidate mask data includes:

[0031] Calculating the mask area of ​​the initial mask data to obtain mask area data;

[0032] Calculate the mask overlap area between the initial mask data to obtain overlap area data

[0033] The initial mask data is subjected to mask screening according to the mask area data and the overlapping area data to obtain the candidate mask data.

[0034] In some embodiments, performing mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data includes:

[0035] Performing connected region screening on the candidate mask data to obtain connected region mask data;

[0036] Performing minimum circumscribed matrix calculation according to the connected region mask data to obtain circumscribed matrix data;

[0037] Performing coordinate sampling on the external matrix data to obtain sampling coordinate data;

[0038] Performing mask extraction on the target image based on the image element extractor and the sampling coordinate data to obtain sampling mask data;

[0039] The sampling mask data is screened for area overlap according to the connected region mask data to obtain the target mask data.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image element generating device, the device comprising:

[0041] An information acquisition module, used for acquiring image generation indication information;

[0042] An image generation module, used for performing image generation on the image generation instruction information based on a preset image generator to obtain a target image;

[0043] A mask extraction module, used for performing mask extraction on the target image based on a preset image element extractor to obtain initial mask data;

[0044] A mask screening module, used for performing mask screening on the initial mask data to obtain candidate mask data;

[0045] A mask optimization module, configured to perform mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data;

[0046] The element extraction module is used to extract elements from the target image according to the target mask data to obtain a target element image.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The present application proposes an image element generation method and device, an electronic device and a storage medium, which obtain image generation indication information and then generate an image for the image generation indication information based on a preset image generator, thereby quickly generating multiple images to obtain a target image; further, based on a preset image element extractor, mask extraction is performed on the target image to obtain initial mask data that can extract image elements from the target image, thereby realizing rapid acquisition of image elements, and mask screening is performed on the initial mask data to obtain candidate mask data, and mask optimization is performed on the candidate mask data according to the image element extractor and the target image, thereby obtaining target mask data that can accurately segment image elements, and finally element extraction is performed on the target image according to the target mask data to obtain a target element image, thereby realizing rapid and accurate acquisition of image elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the image element generation method provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0052] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0053] Figure 4 yes Figure 3 Flow chart of step S303 in FIG.

[0054] Figure 5 yes Figure 3 Flow chart of step S304 in FIG.

[0055] Figure 6 yes Figure 1 Flow chart of step S104 in FIG.

[0056] Figure 7 yes Figure 1 Flow chart of step S105 in FIG.

[0057] Figure 8 is a structural schematic diagram of an image element generating device provided in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Image Masking: Image masking is a technique commonly used in the field of image processing and computer vision. It involves covering or highlighting specific areas in an image in order to perform specific operations on these areas. Image masking is a branch of artificial intelligence and an interdisciplinary subject between computer science and image engineering. By applying masks, objects in an image can be selected, edited, or hidden. It is commonly used in technical fields such as image editing, visual effects production, medical imaging analysis, and machine learning model training. Image masking application systems use digital computers or related equipment to simulate, extend, and expand the processing capabilities of visual information, helping to acquire and utilize visual data in order to improve image quality, extract useful information, or perform content security management.

[0064] Image Elements: Image elements are the basic components of an image and are used to define the details of each visible part of the image. These elements can be subdivided into different features of a scene or object. For example, in a person image, image elements can include hair, clothing, facial expressions, etc. By identifying and analyzing image elements, accurate image segmentation, object detection, and feature extraction can be achieved. This technology is widely used in facial recognition, image editing, virtual makeup trial, autonomous driving vision systems, medical image analysis and other fields. The processing method of image elements can simulate the information processing mechanism of the human visual system, thereby obtaining the semantic information of the image in visual tasks and achieving a deep understanding of the scene, promoting the application of artificial intelligence technology in image analysis.

[0065] Image Noise: Image noise is a random variation or error introduced during image capture, processing or transmission for various reasons, usually manifested as unwanted information or interference on the image. Image noise belongs to the field of image processing and computer vision, which is an application branch of artificial intelligence. This noise may originate from camera sensors, ambient light conditions, interference from electronic devices, or data transmission errors. Image noise processing is an important research field in computer science and image engineering, involving technologies such as noise identification, noise reduction, and image quality restoration. Processing image noise is critical to improving image quality, ensuring the accuracy of image analysis, and the effectiveness of machine learning models. For example, in applications such as medical imaging, astronomical observation, and surveillance video, reducing image noise can help extract more accurate information and support more effective decision-making and diagnosis. By simulating and improving the information processing methods of the human visual system, the research and application of image noise has promoted the development of digital image technology and application innovation in multiple fields.

[0066] Connected Region: Connected Region is a concept in the field of image processing and computer vision, which refers to the region composed of adjacent pixels with the same characteristics in the image. These characteristics usually include color, grayscale or texture. Connected Region is an application branch of artificial intelligence and an interdisciplinary subject between computer science and image analysis. Connected Region Analysis is one of the basic methods to understand and interpret image content, and is often used in technical fields such as object recognition, image segmentation, target tracking and image compression. For example, in medical image analysis, identifying and analyzing connected regions can help identify and measure tumors and other pathological structures; in satellite image processing, connected region analysis can be used to identify geographic features and environmental changes. Techniques for processing and applying connected regions include region growing, labeling algorithms and mathematical morphology, which further promote the development and application innovation of digital image technology by simulating the human visual recognition process.

[0067] Image elements are the basic parts of an image. For example, a cartoon character image includes hair, clothing, facial expressions and other elements. For example, in marketing activities in financial scenarios, image elements are usually needed to create marketing posters or marketing brochures, which are a combination of images and text. Currently, image elements are usually drawn manually, which is inefficient. Therefore, how to increase the generation speed of image elements has become an urgent problem to be solved.

[0068] Based on this, the embodiments of the present application provide a method and device for generating image elements, an electronic device and a storage medium, aiming to increase the generation speed of image elements.

[0069] The embodiments of the present application provide a method and device for generating image elements, an electronic device, and a storage medium, which are specifically described through the following embodiments. First, the method for generating image elements in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0072] The image element generation method provided in the embodiment of the present application relates to the field of image processing technology. The image element generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image element generation method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] Figure 1 is an optional flowchart of the image element generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0075] Step S101, obtaining image generation instruction information;

[0076] Step S102, performing image generation on the image generation instruction information based on a preset image generator to obtain a target image;

[0077] Step S103, performing mask extraction on the target image based on a preset image element extractor to obtain initial mask data;

[0078] Step S104, performing mask screening on the initial mask data to obtain candidate mask data;

[0079] Step S105, performing mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data;

[0080] Step S106, extracting elements from the target image according to the target mask data to obtain a target element image.

[0081] In steps S101 to S106 shown in the embodiment of the present application, by acquiring image generation indication information, and then performing image generation on the image generation indication information based on a preset image generator, a plurality of images are quickly generated to obtain a target image; further, mask extraction is performed on the target image based on a preset image element extractor to obtain initial mask data that can extract image elements from the target image, thereby realizing rapid acquisition of image elements, and mask screening is performed on the initial mask data to obtain candidate mask data, and mask optimization is performed on the candidate mask data according to the image element extractor and the target image, thereby obtaining target mask data that can accurately segment image elements, and finally, element extraction is performed on the target image according to the target mask data to obtain a target element image, thereby realizing rapid and accurate acquisition of image elements.

[0082] In step S101 of some embodiments, the image generation instruction information is text, image, or a combination of text and image, which is used to instruct the image generator to generate the image later. For example, the text is: "Please help me generate a cartoon image of a rabbit wearing a hat." For example, the image is an image of a rabbit, and the combination of text and image is (text: "Please help me put glasses on this image of a frog", and the image is an image of a frog).

[0083] See also Figure 2 In some embodiments, the image generation indication information includes text indication information and image indication information. Step S102 may include but is not limited to steps S201 to S203:

[0084] Step S201, performing text encoding on the text indication information based on the image generator to obtain text encoding information;

[0085] Step S202, performing image encoding on the image indication information based on the image generator to obtain image encoding information;

[0086] Step S203: performing image generation on the text encoding information and the image encoding information based on the image generator to obtain a target image.

[0087] In steps S201 to S203 shown in the embodiment of the present application, text encoding is performed on the text indication information based on the image generator to obtain text encoding information, and then image encoding is performed on the image indication information based on the image generator to obtain image encoding information. Finally, image generation is performed on the text encoding information and the image encoding information based on the image generator to obtain a target image, thereby instructing the image generator to generate an image from both the text and image perspectives at the same time, and accurately obtaining the target image.

[0088] In step S201 of some embodiments, the image generator includes a text encoder. In one embodiment, the text encoder is a CLIP text encoder, and text encoding information is obtained by inputting text indication information into the CLIP text encoder.

[0089] In step S202 of some embodiments, the image generator includes an image encoder. In a real-time example, the image encoder is a CLIP image encoder. Image encoding information is obtained by inputting image indication information into the CLIP image encoder.

[0090] See also Figure 3 In some embodiments, step S203 may include but is not limited to steps S301 to S304:

[0091] Step S301, generating a noise image based on an image generator to obtain an initial noise image;

[0092] Step S302, generating time features based on the image generator to obtain time step embedding features;

[0093] Step S303, performing noise encoding on the initial noise image based on the image generator to obtain a noise encoding feature map;

[0094] Step S304, based on the image generator, image generation is performed on the time step embedding features, the noise coding feature map, the text coding information and the image coding information to obtain a target image.

[0095] In steps S301 to S304 shown in the embodiment of the present application, noise image generation is performed based on the image generator to obtain an initial noise image, and then time feature generation is performed based on the image generator to obtain a time step embedding feature, and noise encoding is performed on the initial noise image based on the image generator to obtain a noise coding feature map, and finally image generation is performed based on the time step embedding feature, the noise coding feature map, the text coding information and the image coding information to obtain a target image, thereby instructing the image generator to generate an image from both the text and image perspectives at the same time, and accurately obtaining the target image.

[0096] In step S301 of some embodiments, the image generator continuously adds noise based on a preset clear image to obtain an initial noisy image, as shown in formula (1):

[0097]

[0098] Among them, t is the iteration time step, that is, the current cycle number, ranging from 1 to the preset time step threshold, x t is the noise image in the current iteration, x t-1 is the noise image relative to the previous one in the current iteration, β t is the preset noise parameter, which changes with the time step, and ∈ is the random noise obtained from the preset standard normal distribution. When the iteration step reaches the time step threshold, x t is the initial noise image.

[0099] In step S302 of some embodiments, time feature generation is performed based on the image generator to obtain a time step embedding feature. The time feature generation is shown in formula (2):

[0100]

[0101] Among them, D t is the dimension of the preset embedding vector, i is the index of the current dimension, and the range of i is E t is the time step embedding feature, t is the iteration time step, and the range is from 1 to the preset time step threshold.

[0102] See also Figure 4 In some embodiments, step S303 may include but is not limited to steps S401 to S403:

[0103] Step S401, performing image convolution on the initial noise image based on the image generator to obtain a convolution noise image;

[0104] Step S402, performing activation processing on the convolution noise map based on the image generator to obtain an activated noise map;

[0105] Step S403, normalizing the activation noise map based on the image generator to obtain a noise coding feature map.

[0106] In steps S401 to S403 shown in the embodiment of the present application, an image generator is used to perform image convolution on the initial noise map, extract local patterns and details in the noise map, capture more complex structural information, and obtain a convolution noise map. The convolution noise map is then activated by the image generator, so that the image has stronger expressive power and a more diverse and realistic activation noise map is obtained. Finally, the activation noise map is normalized by the image generator to obtain a noise coding feature map.

[0107] In step S401 of some embodiments, the image generator includes a convolution layer, and the initial noise map is input into the convolution layer for image convolution, so as to extract local patterns and details in the initial noise map, capture more complex structural information, and obtain a convolution noise map.

[0108] In step S402 of some embodiments, the image generator includes an activation function, and the convolution noise map is input into the activation function for nonlinear change, so that the convolution noise map has stronger expression ability and obtains a more diverse and realistic activation noise map.

[0109] In step S403 of some embodiments, the image generator includes a normalization layer, and the activation noise map is input into the normalization layer for normalization processing to obtain a noise coding feature map.

[0110] See also Figure 5 In some embodiments, step S304 includes but is not limited to steps S501 to S505:

[0111] Step S501, based on the image generator, feature splicing is performed on the text coding information and the image coding information to obtain the indication coding information;

[0112] Step S502, performing attention feature fusion on the noise coding feature map based on the image generator and the indication coding information to obtain a fused coding feature map;

[0113] Step S503, performing time step embedding on the fused coding feature map based on the image generator and the time step embedding feature to obtain an embedded coding feature map;

[0114] Step S504, performing downsampling processing on the embedded coding feature map based on the image generator to obtain a downsampled coding feature map;

[0115] Step S505: performing image decoding on the downsampled encoded feature map based on the image generator to obtain a target image.

[0116] In steps S501 to S505 shown in the embodiment of the present application, based on the image generator, feature splicing is performed on the text coding information and the image coding information to obtain the indication coding information, and then based on the image generator and the indication coding information, attention feature fusion is performed on the noise coding feature map to obtain the fused coding feature map, and then based on the image generator and the time step embedding feature, the fused coding feature map is time-step embedded to obtain the embedded coding feature map, and based on the image generator, the embedded coding feature map is down-sampled to obtain the down-sampled coding feature map, and finally, based on the image generator, the down-sampled coding feature map is decoded to obtain the target image, thereby quickly generating the target image based on the indication information, providing an image basis for subsequent element extraction.

[0117] In step S501 of some embodiments, feature splicing refers to connecting text encoding information and image encoding information to form a comprehensive feature representation, namely, indication encoding information. The indication encoding information contains dual information from text and image, and can provide clear instructions during the generation process to ensure that the subsequently generated image is consistent with the description of the indication information. Feature splicing is shown in formula (3):

[0118] c=Concat(c text ,c image ) (3),

[0119] Among them, c text Encode information for text, c image is the image coding information, c is the indication coding information, and Concat() is the concatenation function.

[0120] In step S502 of some embodiments, the image generator includes a self-attention layer, which is used to indicate the encoding information and the noise encoding feature. Figure 1 And input it into the self-attention layer for attention feature fusion to obtain the fused encoding feature map. The attention feature fusion is shown in formula (4):

[0121]

[0122] Among them, W Q ,W K ,W v To indicate the coding information and noise coding characteristics Figure 1 And input to the output obtained after the self-attention layer, (W K E) T For the preset (W K E) is the transpose, is the preset scaling factor, F is the noise coding feature map, and F′ is the fusion coding feature map.

[0123] By fusing the indication coding information with the noise coding feature map, the subsequently generated target image can be more in line with the requirements of the indication information and the target image can be generated accurately.

[0124] In step S503 of some embodiments, the image generator includes a pre-trained linear transformation matrix for converting the time step embedding feature to the same dimension as the fused encoding feature map. The time step embedding is shown in formula (5):

[0125] F embeded =F ′ +W e E t (5)

[0126] Among them, F embeded is the embedded coding feature map, F′ is the fusion coding feature map, W e is the pre-trained linear transformation matrix, E t Embed features for time steps.

[0127] In step S504 of some embodiments, the image generator includes a downsampling layer, and the embedded coding feature map is input into the downsampling layer to capture the global context information of the embedded coding feature map, generate a more semantically meaningful high-level representation by aggregating local features, and understand and generate image details at different scales through downsampling, thereby improving the quality and consistency of the subsequent generation of the target image, and obtaining the downsampled coding feature map.

[0128] In step S505 of some embodiments, the image generator includes a decoder based on the U-Net architecture, and the downsampled encoded feature map is input into the decoder based on the U-Net architecture for decoding, so as to obtain a target image that finally conforms to the description of the image generation indication information.

[0129] In step S103 of some embodiments, the preset image element extractor is the SAM (vit-h) model, and the mask extraction is a global extraction method SamAutomaticMaskGenerator based on the SAM (vit-h) model that performs mask extraction on the target image to obtain initial mask data.

[0130] See also Figure 6 In some embodiments, step S104 includes but is not limited to steps S601 to S603:

[0131] Step S601, calculating the mask area of ​​the initial mask data to obtain mask area data;

[0132] Step S602, calculating the mask overlap area between the initial mask data to obtain overlap area data;

[0133] Step S603: performing mask screening on the initial mask data according to the mask area data and the overlap area data to obtain candidate mask data.

[0134] In steps S601 to S603 shown in the embodiment of the present application, the mask area of ​​the initial mask data is calculated to obtain the mask area data, and then the mask overlapping area between the initial mask data is calculated to obtain the overlapping area data. Finally, the initial mask data is mask-screened according to the mask area data and the overlapping area data to obtain the candidate mask data, thereby screening the mask data so that the mask data can accurately extract image elements from the image.

[0135] In step S601 of some embodiments, the initial mask data is a two-dimensional array, and the dimension of the initial mask data is the same as that of the target image. The number of pixels representing the image elements in the initial mask data is obtained to obtain the mask area data. For example, the initial mask data is [[1,0,0], [0,0,1], [1,0,0]], where 1 represents the pixel where the image element is located, and the number of all mask values ​​1 is obtained, and the mask area data is 3.

[0136] In step S602 of some embodiments, the initial mask data includes a plurality of mask data, each mask data corresponding to an image element. For example, the mask data corresponding to the image element "hat" is [[1,1,1], [0,0,0], [0,0,0]], and the mask data corresponding to the image element "pants" is [[0,1,0], [0,1,0], [0,1,0]]. Then, the number of overlapping pixels between each mask data is obtained to obtain overlapping area data. For example, if the pixel in the middle of the first row and second column of the image element "hat" and the image element "pants" is both 1, then the overlapping area is 1.

[0137] In step S603 of some embodiments, the ratio between the mask area data and the overlap area data is compared with a preset overlap threshold, and if it is greater than the overlap threshold, the corresponding initial mask data is filtered out. For example, the overlap area between the image element "hat" and the image element "pants" is 1, the mask area data of the image element "hat" is 3, the ratio between the overlap area data and the mask area data is 33%, and the preset overlap threshold is 30%, which means that the mask data of the image element "hat" is not accurate, so the mask value of the image element "hat" is filtered out, and after screening each image element corresponding to the initial mask data, the mask data corresponding to the image elements that are not filtered out constitute the candidate mask data.

[0138] See also Figure 7 In some embodiments, step S105 may include but is not limited to steps S701 to S705:

[0139] Step S701, performing connected region screening on candidate mask data to obtain connected region mask data;

[0140] Step S702, performing minimum circumscribed matrix calculation according to the connected region mask data to obtain circumscribed matrix data;

[0141] Step S703, coordinate sampling is performed on the external matrix data to obtain sampled coordinate data;

[0142] Step S704, performing mask extraction on the target image based on the image element extractor and the sampling coordinate data to obtain sampling mask data;

[0143] Step S705 , performing area overlap screening on the sampled mask data according to the connected region mask data to obtain target mask data.

[0144] In steps S701 to S705 shown in the embodiment of the present application, connected area mask data is obtained by performing connected area screening on candidate mask data, and then the minimum circumscribed matrix is ​​calculated based on the connected area mask data to obtain circumscribed matrix data, and then coordinate sampling is performed on the circumscribed matrix data to obtain sampled coordinate data, and mask extraction is performed on the target image based on the image element extractor and the sampled coordinate data to obtain sampled mask data, and finally the sampled mask data is screened for area overlap based on the connected area mask data to obtain target mask data, thereby obtaining more accurate target mask data.

[0145] In step S701 of some embodiments, the connected region screening is to perform connected region calculation on the candidate mask data, then obtain the pixel data corresponding to the largest connected region, and set other regions to zero, so as to ensure that the mask corresponds to a coherent image element. For example, the candidate mask data includes the mask data corresponding to the image element "hat", which is [[1,1,1], [0,0,0], [0,1,0]], and then the connected region calculation can obtain two connected regions, namely [1,1,1] in the first row and [0,1,0] in the third row, and then obtain the largest connected region level [1,1,1] in the first row, and set the 1 of [0,1,0] in the third row to zero, and obtain the final connected region mask data as [[1,1,1], [0,0,0], [0,0,0]].

[0146] In step S702 of some embodiments, the minimum circumscribed matrix is ​​calculated to calculate the minimum circumscribed matrix corresponding to the connected region mask data. The calculation method may be a Flood Fill algorithm, a Union-Find algorithm or other algorithms, and this application does not make any specific limitations.

[0147] In step S703 of some embodiments, coordinates of the external matrix data are randomly sampled, that is, the anchor frame formed by the external matrix data is randomly sampled, and the coordinate data within the anchor frame is sampled to obtain sampled coordinate data.

[0148] In step S704 of some embodiments, the sampling coordinate data and the target image are input into an image element extractor for mask extraction, specifically, the sampling coordinate data and the target image are input into a prompt-based extraction method SamPredictor of a SAM (vit-h) model to extract new mask data, i.e., sampling mask data.

[0149] In step S705 of some embodiments, the overlapping area between the connected region mask data and the sampling mask data is calculated. If the ratio between the overlapping area and the area of ​​the sampling mask data is less than a preset overlapping ratio threshold, resampling is performed until the ratio between the area of ​​the sampling mask data and the overlapping area is greater than the preset overlapping ratio threshold, and the sampling mask data is used as the target mask data.

[0150] After performing mask extraction on the target image and then screening the mask data, the candidate mask data is obtained, and after connected area screening, minimum circumscribed matrix calculation and coordinate sampling, it is ensured that the obtained sampling coordinate data corresponds to a unique element in the image. Then, based on the sampling coordinate data as an indication, the image element extractor can perform more accurate mask extraction based on the sampling coordinate data, thereby accurately obtaining the target mask data corresponding to the image element.

[0151] In step S106 of some embodiments, element extraction is to retain the pixel points of the target image corresponding to the pixel point coordinates with a value of 1 in the target mask data, and set the pixel points of the target image corresponding to the pixel point coordinates with a value of 0 in the target mask element to 0, thereby achieving element extraction and obtaining a target element image.

[0152] See also Figure 8 The embodiment of the present application further provides an image element generation device, which can implement the above-mentioned image element generation method, and the device includes:

[0153] The information acquisition module 801 is used to acquire image generation instruction information;

[0154] An image generation module 802 is used to generate an image based on the image generation instruction information based on a preset image generator to obtain a target image;

[0155] The mask extraction module 803 is used to perform mask extraction on the target image based on a preset image element extractor to obtain initial mask data;

[0156] The mask screening module 804 is used to perform mask screening on the initial mask data to obtain candidate mask data;

[0157] The mask optimization module 805 is used to perform mask optimization on the candidate mask data according to the image element extractor and the target image to obtain the target mask data;

[0158] The element extraction module 806 is used to extract elements from the target image according to the target mask data to obtain a target element image.

[0159] The specific implementation of the image element generating device is substantially the same as the specific implementation of the above-mentioned image element generating method, and will not be described in detail here.

[0160] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image element generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0161] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0162] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0163] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image element generation method of the embodiment of this application;

[0164] Input / output interface 903, used to implement information input and output;

[0165] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0166] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0167] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0168] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image element generation method is implemented.

[0169] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The image element generation method, image element generation device, electronic device and storage medium provided in the embodiments of the present application obtain image generation indication information, and then generate an image for the image generation indication information based on a preset image generator, thereby quickly generating multiple images to obtain a target image; further, mask extraction is performed on the target image based on a preset image element extractor to obtain initial mask data that can extract image elements from the target image, thereby realizing rapid acquisition of image elements, and mask screening is performed on the initial mask data to obtain candidate mask data, and mask optimization is performed on the candidate mask data according to the image element extractor and the target image, thereby obtaining target mask data that can accurately segment image elements, and finally element extraction is performed on the target image according to the target mask data to obtain a target element image, thereby realizing rapid and accurate acquisition of image elements.

[0171] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for generating an image element, characterized in that: The method comprises: Obtaining image generation instruction information; Performing image generation on the image generation instruction information based on a preset image generator to obtain a target image; Performing mask extraction on the target image based on a preset image element extractor to obtain initial mask data; Performing mask screening on the initial mask data to obtain candidate mask data; performing mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data; Element extraction is performed on the target image according to the target mask data to obtain a target element image.

2. The method according to claim 1, characterized in that The image generation instruction information includes text instruction information and image instruction information, and the image generation instruction information is generated by a preset image generator to obtain a target image, including: Performing text encoding on the text indication information based on the image generator to obtain text encoding information; Performing image encoding on the image indication information based on the image generator to obtain image encoding information; The target image is obtained by performing image generation on the text encoding information and the image encoding information based on the image generator.

3. The method according to claim 2, characterized in that The step of generating an image based on the text encoding information and the image encoding information by the image generator to obtain the target image comprises: Generate a noise image based on the image generator to obtain an initial noise image; Generate time features based on the image generator to obtain time step embedding features; Performing noise encoding on the initial noise image based on the image generator to obtain a noise encoding feature map; Based on the image generator, image generation is performed on the time step embedding feature, the noise coding feature map, the text coding information and the image coding information to obtain the target image.

4. The method according to claim 3, characterized in that The step of performing noise encoding on the initial noise image based on the image generator to obtain a noise encoding feature map comprises: Performing image convolution on the initial noise map based on the image generator to obtain a convolution noise map; Based on the image generator, activation processing is performed on the convolution noise map to obtain an activation noise map; The activation noise map is normalized based on the image generator to obtain the noise coding feature map.

5. The method according to claim 3, characterized in that: The step of generating an image based on the image generator on the time step embedding feature, the noise coding feature map, the text coding information and the image coding information to obtain the target image includes: Based on the image generator, feature splicing is performed on the text encoding information and the image encoding information to obtain indication encoding information; Performing attention feature fusion on the noise coding feature map based on the image generator and the indication coding information to obtain a fused coding feature map; Performing time step embedding on the fused coding feature map based on the image generator and the time step embedding feature to obtain an embedded coding feature map; Performing downsampling processing on the embedded coding feature map based on the image generator to obtain a downsampled coding feature map; The downsampled encoded feature map is decoded based on the image generator to obtain the target image.

6. The method according to any one of claims 1 to 5, characterized in that: The performing mask screening on the initial mask data to obtain candidate mask data includes: Calculating the mask area of ​​the initial mask data to obtain mask area data; Calculate the mask overlap area between the initial mask data to obtain overlap area data The initial mask data is subjected to mask screening according to the mask area data and the overlapping area data to obtain the candidate mask data.

7. The method according to any one of claims 1 to 5, characterized in that: The step of performing mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data includes: Performing connected region screening on the candidate mask data to obtain connected region mask data; Performing minimum circumscribed matrix calculation according to the connected region mask data to obtain circumscribed matrix data; Performing coordinate sampling on the external matrix data to obtain sampling coordinate data; Performing mask extraction on the target image based on the image element extractor and the sampling coordinate data to obtain sampling mask data; The sampling mask data is screened for area overlap according to the connected region mask data to obtain the target mask data.

8. An image element generating device, characterized in that: The device comprises: An information acquisition module, used for acquiring image generation indication information; An image generation module, used for performing image generation on the image generation instruction information based on a preset image generator to obtain a target image; A mask extraction module, used for performing mask extraction on the target image based on a preset image element extractor to obtain initial mask data; A mask screening module, used for performing mask screening on the initial mask data to obtain candidate mask data; A mask optimization module, configured to perform mask optimization on the candidate mask data according to the image element extractor and the target image to obtain target mask data; The element extraction module is used to extract elements from the target image according to the target mask data to obtain a target element image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image element generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image element generation method according to any one of claims 1 to 7 is implemented.

Images