Image element generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image element generation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a target image, inputting the target image into an element scoring model, and determining an element score of each candidate element in the target image; based on the element score of each candidate element, screening out a target element from the plurality of candidate elements; determining the similarity between each target element and a historical element; on the basis of the similarity between each target element and a historical element, screening out a difference element from the multiple target elements; and removing the difference elements in the target elements to obtain generation elements. According to the embodiment of the invention, the high-quality elements close to the business can be screened out step by step by means of score screening and similarity screening, the correlation between the generated image elements and the financial promotion business can be improved, and the accuracy and practicability of the generated image elements are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for generating image elements, an electronic device, and a storage medium. Background Art

[0002] Image generation is a very active research area in the field of computer vision with many practical application scenarios. The images generated by the diffusion model are more diverse and of higher quality than those generated by the GAN. The training process of the diffusion model is also more stable, making image element generation service businesses such as financial business promotion and marketing, creative design, etc. a reality.

[0003] In the related technologies, due to the randomness of the diffusion generation process and the lack of domain adaptation capabilities, the generated results often cannot directly meet business needs, and usually require secondary modification and creation to extract useful elements for the completion of the design draft. Therefore, it is urgent to provide an image element generation to solve the above technical problems. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device for generating image elements, an electronic device and a storage medium, which can improve the relevance of the generated image elements to the business and enhance the accuracy and practicality of the generated image elements.

[0005] To achieve the above objective, a first aspect of an embodiment of the present application provides a method for generating an image element, the method comprising:

[0006] Acquire a target image, input the target image into an element scoring model, and determine an element score of each candidate element in the target image;

[0007] Based on the element score of each candidate element, screening out a target element from a plurality of candidate elements;

[0008] Determining the similarity between each of the target elements and the historical elements;

[0009] Based on the similarity between each of the target elements and the historical elements, filter out the difference elements from the multiple target elements;

[0010] The difference element in the target element is removed to obtain a generated element.

[0011] In some embodiments, inputting the target image into an element scoring model to determine an element score of each candidate element in the target image includes:

[0012] Inputting the target image into an element scoring model to perform image encoding on the target image to obtain image features;

[0013] Get the preset element features corresponding to the preset elements of each element type;

[0014] Matching each of the preset element features with the image features, and selecting candidate elements matching the image features from the preset elements;

[0015] Determine a candidate position of each candidate element in the target image;

[0016] An element score of each candidate element is determined based on the candidate element type and the corresponding candidate position of each candidate element.

[0017] In some embodiments, after removing the difference element from the target element to obtain the generated element, the method further includes:

[0018] In response to the reference element acquisition instruction, acquire the reference element and the element expected description;

[0019] extracting reference information of the reference element;

[0020] The reference information and the element expected description are input into an image element generation model to generate an expected element.

[0021] In some embodiments, before obtaining the reference element and the element expected description in response to the reference element obtaining instruction, the method further includes:

[0022] Displaying each of the generated elements in a specified interface;

[0023] The step of obtaining the reference element and the expected description of the element in response to the reference element obtaining instruction includes:

[0024] In response to a selection operation on the generation element, determining a target generation element selected by the selection operation, and determining the target generation element as a reference element;

[0025] Displaying a desired description control in the specified interface;

[0026] In response to an input instruction for the expected description control, the description content of the input instruction in the expected description control is determined, and the description content is determined as the element expected description.

[0027] In some embodiments, before displaying each of the generated elements in the specified interface, the method further includes:

[0028] Obtaining the usage frequency of each of the generated elements;

[0029] The displaying of each generated element in the specified interface includes:

[0030] Determine the display order of each of the generated elements in descending order of usage frequency;

[0031] Each of the generated elements is displayed in the designated interface in the display order of the generated elements.

[0032] In some embodiments, the step of selecting a target element from a plurality of candidate elements based on the element score of each candidate element includes:

[0033] The candidate elements are sorted in descending order of scores, and the candidate element with a first preset proportion in the sorting is determined as the target element.

[0034] In some embodiments, the step of selecting a difference element from a plurality of the target elements based on the similarity between each of the target elements and the historical elements includes:

[0035] The target elements are sorted in descending order of similarity, and the target element with the second preset proportion in the last order is determined as a difference element.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image element generating device, including:

[0037] A first acquisition unit is used to acquire a target image, input the target image into an element scoring model, and determine an element score of each candidate element in the target image;

[0038] A first screening unit, configured to screen a target element from a plurality of candidate elements based on an element score of each candidate element;

[0039] A determination unit, used to determine the similarity between each of the target elements and the historical elements;

[0040] A second screening unit, configured to screen out difference elements from the plurality of target elements based on the similarity between each target element and the historical element;

[0041] The elimination unit is used to eliminate the difference element in the target element to obtain a generated element.

[0042] In some embodiments, the first screening unit is used to:

[0043] Inputting the target image into an element scoring model to perform image encoding on the target image to obtain image features;

[0044] Get the preset element features corresponding to the preset elements of each element type;

[0045] Matching each of the preset element features with the image features, and selecting candidate elements matching the image features from the preset elements;

[0046] Determine a candidate position of each candidate element in the target image;

[0047] An element score of each candidate element is determined based on the candidate element type and the corresponding candidate position of each candidate element.

[0048] In some embodiments, the apparatus further comprises:

[0049] A second acquisition unit, configured to acquire a reference element and an expected description of the element in response to a reference element acquisition instruction;

[0050] an extraction unit, used for extracting reference information of the reference element;

[0051] An input unit is used to input the reference information and the expected description of the element into an image element generation model to generate an expected element.

[0052] In some embodiments, the apparatus further comprises:

[0053] A display unit, used to display each of the generated elements in a specified interface;

[0054] The response unit comprises:

[0055] A first determining subunit is used to determine, in response to a selection operation on the generation element, a target generation element selected by the selection operation, and determine the target generation element as a reference element;

[0056] A display subunit, used for displaying a desired description control in the specified interface;

[0057] The second determining subunit is configured to, in response to an input instruction for the expected description control, determine the description content of the input instruction in the expected description control, and determine the description content as the element expected description.

[0058] In some embodiments, the apparatus further comprises:

[0059] A third acquisition unit, used to acquire the usage frequency of each of the generated elements;

[0060] The display subunit is used for:

[0061] Determine the display order of each of the generated elements in descending order of usage frequency;

[0062] Each of the generated elements is displayed in the designated interface in the display order of the generated elements.

[0063] In some embodiments, the first screening unit is used to:

[0064] The candidate elements are sorted in descending order according to the element scores, and the candidate element with a first preset proportion in the sorting is determined as the target element.

[0065] In some embodiments, the second screening unit is used to:

[0066] The target elements are sorted in descending order of similarity, and the target element with the second preset proportion in the last order is determined as a difference element.

[0067] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0068] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0069] The image element generation method and device, electronic device and storage medium proposed in the present application obtain a target image, input the target image into an element scoring model, determine the element score of each candidate element in the target image; based on the element score of each candidate element, select the target element from multiple candidate elements; determine the similarity between each target element and a historical element; based on the similarity between each target element and a historical element, select the difference element from multiple target elements; remove the difference element from the target element to obtain a generated element. In this way, high-quality elements that are close to the business are gradually selected through scoring screening and similarity screening, which can improve the relevance of the generated image elements to the business, and improve the accuracy and practicality of the generated image elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is a flow chart of the image element generation method provided in an embodiment of the present application;

[0071] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0072] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0073] Figure 4 yes Figure 1 The flowchart after step S105 in FIG.

[0074] Figure 5 A schematic diagram of an interface for a formulation interface provided in an embodiment of the present application;

[0075] Figure 6 yes Figure 4 Flow chart of step S401 in FIG.

[0076] Figure 7 Another schematic diagram of the interface provided by the embodiment of the present application;

[0077] Figure 8 yes Figure 1 Flowchart of step S104 in .

[0078] Fig. 9 is a structural schematic diagram of an image element generating device provided in an embodiment of the present application;

[0079] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0081] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0083] First, some nouns involved in this application are analyzed:

[0084] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0085] Network model: A network model is an abstract representation and mathematical description of various network systems in the real world. It usually consists of a set of nodes and edges connecting the nodes, and is used to simulate and analyze different types of network structures and their behaviors. The components include: 1. Nodes, which represent entities in the network. They can be physical entities such as computers, sensors, etc., or abstract concepts such as people, organizations, etc. Each node may have specific attributes such as name, status, function, etc. 2. Edges, which represent the relationship or connection between nodes. Edges can be directed or undirected, with or without weights. For example, in a social network, an edge can represent the relationship between people, and the weight can represent the strength of the relationship.

[0086] The functions of network models include: 1. Understanding and analyzing network structure: By analyzing the network model, we can understand the network's topological structure, connection patterns between nodes, centrality and other characteristics. This helps to discover key nodes, community structures and potential vulnerabilities in the network. 2. Predicting network behavior: Based on the network model, various predictions can be made, such as the speed of information propagation in the network, the reliability of the network, the behavioral changes of nodes, etc. This is very important for decision-making and risk management. 3. Optimizing network performance: Network models can be used to optimize the performance indicators of the network, such as improving the throughput of communication networks, reducing the congestion of traffic networks, and enhancing the information dissemination efficiency of social networks. By adjusting the network structure, node attributes or edge weights, the optimal network configuration can be found. 4. Designing and planning networks: Network models can provide guidance when designing new network systems or expanding existing networks. According to specific needs and goals, appropriate network models can be constructed, and simulations and analyses can be performed to determine the best design solution.

[0087] Optical Character Recognition (OCR): is a technology that can convert text information in an image into editable text. It mainly involves image preprocessing, text detection, text recognition, etc. OCR models are widely used in many fields, such as document digitization in office automation, book and document digitization in libraries, license plate recognition in the transportation field, bill recognition in the financial field, etc., which greatly improves data processing efficiency and the convenience of information acquisition. With the development of deep learning technology, the performance of OCR models has been continuously improved, and the accuracy and speed have been significantly improved, which can better adapt to the text recognition needs in various complex scenarios.

[0088] Web crawler: Also known as web spider or web robot. It is a program or script that automatically crawls the World Wide Web information according to certain rules. Its working principle is as follows: the crawler program starts with one or more URLs of initial web pages. These initial URLs can be specified by the user or obtained from some seed websites. Then, the crawler program sends requests to these URLs to obtain the content of the web pages. After receiving the web page content, the crawler program analyzes the web page and extracts the links (URLs) and other useful information, such as text, pictures, videos, etc. Then, the crawler program adds the extracted links to the URL queue to be crawled. Repeat the above process, continuously take out URLs from the URL queue, send requests, obtain web page content, and extract links until certain stop conditions are met, such as reaching the preset crawling depth, crawling time limit, and crawling number of web pages.

[0089] Based on this, the embodiments of the present application provide a method and device for generating image elements, an electronic device and a storage medium, aiming to improve the relevance of the generated image elements to the business and reduce or avoid the probability that the generated image elements cannot be directly used.

[0090] The image element generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image element generation method in the embodiments of the present application is described.

[0091] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0092] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0093] The image element generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image element generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image element generation method, etc., but is not limited to the above forms.

[0094] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0095] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0096] Figure 1 is an optional flowchart of the image element generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0097] Step S101 : acquiring a target image, inputting the target image into an element scoring model, and determining an element score of each candidate element in the target image.

[0098] Elements are the components that make up an image, mainly including the following: shapes, textures, lines, patterns, and objects. The target image is an image that contains at least one element and is obtained from the Internet in advance. The acquisition method can be to crawl the design poster image from the design image library on various networks through a web crawler, or to obtain a frame of a video from a video website, and convert the design poster in different image formats such as PNG and JPEG into a specified format (such as BMP format) to obtain the target image. In addition, in order to avoid copyright issues, it is also necessary to perform copyright screening on each crawled image to screen out images without copyright factors (such as brand logos, brand names) as target images.

[0099] Specifically, since the elements generated in the related art cannot be directly used by designers due to their low quality, the embodiment of the present application inputs the target image into the element scoring model to determine the element score of each candidate element in the target image, thereby screening high-quality elements according to the element score. High quality here can refer to factors that affect element quality, such as high color depth, high contrast, and high resolution.

[0100] Step S102: based on the element score of each candidate element, select a target element from the plurality of candidate elements.

[0101] In step S101, after scoring each candidate element in each target image by the element scoring model, high-quality target elements can be screened out from the candidate elements according to the element scores.

[0102] For example, the target images are target image A, target image B and target image C, the element score of candidate element a1 contained in target image A is 90 points, the element score of candidate element a2 is 60 points, the element score of candidate element b1 contained in target image B is 85 points, the element score of candidate element c1 contained in target image C is 80 points, and the element score of candidate element c2 is 50 points. Then, high-quality target elements can be screened out from candidate element a1, candidate element a2, candidate element b1, candidate element c1 and candidate element c2 according to the element scores.

[0103] Step S103: determining the similarity between each of the target elements and the historical elements.

[0104] The historical elements are the historical elements stored locally and are used as elements included in the historical images of the designed posters or promotional posters.

[0105] Specifically, in order to increase the probability that subsequently generated elements can be adopted and used by designers, the similarity between each target element and the historical elements adopted and used by designers can be calculated, so as to retain elements similar to the historical elements from multiple target elements, so that the retained elements can be adopted and used by designers with a high probability.

[0106] Among them, the calculation method of the similarity between each target element and the historical element includes: extracting the target feature vector of each target element and extracting the historical feature vector of the historical element, calculating the cosine distance between the target feature vector and the historical feature vector to obtain a calculation result in which the cosine distance is between [-1,1]. The closer the calculation result is to 1, the closer the directions of the two vectors are, and the higher the similarity is; the closer the value is to -1, the opposite directions of the two vectors, and the lower the similarity is; a value of 0 indicates that the two vectors are orthogonal and unrelated, and the similarity between the target element and the historical element is calculated in this way.

[0107] Specifically, the calculation method of the cosine distance can refer to the following formula:

[0108]

[0109] Among them, A·B represents the dot product (inner product) of vector A and vector B, ‖A‖ and ‖B‖ represent the modulus (length) of vector A and vector B respectively, and d cos (A,B) is the cosine distance between vector A and vector B.

[0110] Step S104: based on the similarity between each of the target elements and the historical elements, filter out the difference elements from the multiple target elements.

[0111] After the similarity between each target element and the historical element is determined in step S103, different elements with lower similarity can be screened out from the plurality of target elements according to the similarity.

[0112] For example, the target elements are target element 1, target element 2 and target element 3, the similarity between target element 1 and historical element 1 is 0.8, the similarity between target element 2 and historical element 2 is 0.6, and the similarity between target element 3 and historical element 1 is 0.7. Based on the similarity, different elements with lower similarity can be filtered out from target element 1, target element 2 and target element 3.

[0113] Step S105: remove the difference element in the target element to obtain a generated element.

[0114] After the difference elements are screened out in step S104, all the difference elements in the target element that are unlikely to be adopted and used by the designer are removed to retain the generated elements that are likely to be adopted and used by the designer.

[0115] Steps S101 to S105 shown in the embodiment of the present application are to obtain a target image, input the target image into an element scoring model, determine the element score of each candidate element in the target image; based on the element score of each candidate element, select the target element from multiple candidate elements; determine the similarity between each target element and a historical element; based on the similarity between each target element and a historical element, select the difference element from multiple target elements; remove the difference element from the target element to obtain a generated element. In this way, high-quality elements that are close to the business are gradually selected through scoring screening and similarity screening, which can improve the relevance of the generated image elements to the business, and improve the accuracy and practicality of the generated image elements.

[0116] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S205:

[0117] Step S201: input the target image into an element scoring model to perform image encoding on the target image to obtain image features.

[0118] After the target image is input into the element scoring model, the convolutional layer of the element scoring model will extract the local features of the target image to convert the image from the original pixel space to a more abstract and semantic feature space. This image feature contains the details and semantic information of the target image at different levels, providing a basis for subsequent analysis and processing, thereby extracting image features such as edges and textures.

[0119] Specifically, in order not to affect the element scoring model's recognition of elements in the target image, the target image needs to be input into the text detection and recognition model before being input into the element scoring model, so that the text content in the target image and the position information corresponding to the text content can be identified through the text detection and recognition model, and the text content can be eliminated at the position corresponding to the text content, and the eliminated position can be filled with the average value of the pixels in the adjacent area corresponding to the position of the original text content to obtain a target image without text content.

[0120] Taking the Practical UltraLightweight OCR System (pp-ocr) model as an example, the text detection and recognition model mainly consists of three parts: text detection, detection frame correction and text recognition. First, input the image to be detected and detect the text line; then perform direction detection on each detected text line; finally, perform text recognition on the text line after direction correction to obtain the final result. Text detection uses the (differentiable binarization, DB) algorithm, which is based on the idea of ​​segmentation and can obtain a compact bounding box of the text, and the post-processing is simple.

[0121] Step S202: Obtain preset element features corresponding to preset elements of each element type.

[0122] The preset elements are predefined image elements with specific features, such as specific objects, shapes, etc. The preset element features are an abstract representation of these preset elements, which are obtained by learning a large number of similar elements. Acquiring the preset element features is a preparation for subsequent matching with the target image features.

[0123] Step S203, matching each of the preset element features with the image features, and selecting candidate elements matching the image features from the preset elements.

[0124] Among them, by comparing the characteristics of the target image with the characteristics of the preset elements, the preset elements with similar characteristics to the target image are found. This matching can be based on various similarity measurement methods, such as calculating cosine distance, Euclidean distance, etc. By screening out candidate elements, the scope of subsequent analysis can be narrowed down to focus on elements related to the preset elements.

[0125] Step S204: determining a candidate position of each candidate element in the target image.

[0126] After the candidate elements are determined, their specific locations in the target image need to be determined. This can be achieved through various target detection or positioning algorithms, such as the region-based convolutional neural network (R-CNN) series of algorithms, the YOLO series of algorithms, etc. Determining the candidate locations helps to more accurately evaluate the importance and role of the candidate elements in the target image.

[0127] Step S205 : determining an element score of each candidate element based on the candidate element type and the corresponding candidate position of each candidate element.

[0128] The element score comprehensively considers the type and location information of the candidate element. Different types of candidate elements may have different importance.

[0129] For example, in a landscape image, different types of elements such as the sky, mountains, and trees may have different visual weights. At the same time, the location of the candidate element will also affect its score. For example, an element located in the center of the image may attract more attention than an element located at the edge. By determining the element score, a quantitative basis can be provided for subsequent image analysis, editing, or other applications.

[0130] In this way, the target image is input into the element scoring model to determine the element score of each candidate element in the target image, so as to screen high-quality elements according to the element score and improve the accuracy and practicality of the generated image elements.

[0131] See also Figure 3 In some embodiments, step S102 may include but is not limited to step S301:

[0132] Step S301 , sorting the candidate elements in descending order of element scores, and determining the candidate element with a first preset proportion in the sorting as the target element.

[0133] Among them, after determining the element score of each candidate element, each candidate element is sorted in descending order according to the element score to obtain a candidate element sequence with element scores from large to small, and the candidate element with the first preset proportion in the candidate element sequence is determined as the target element.

[0134] For example, the element score of candidate element a1 is 90 points, the element score of candidate element a2 is 60 points, the element score of candidate element b1 contained in target image B is 85 points, the element score of candidate element c1 contained in target image C is 80 points, and the element score of candidate element c2 is 50 points. Then, after sorting the candidate elements in order of element scores from large to small, the candidate element sequence formed is: candidate element a1, candidate element b1, candidate element c1, candidate element a2, candidate element c2. The first preset proportion is 40%, then the first 40% of the 5 candidate elements are the first 2, that is, candidate element a1 and candidate element b1 are determined as target elements.

[0135] After step S301 in some embodiments, the method further includes:

[0136] comparing a target number of target elements with a preset number of preset elements;

[0137] If the target number does not reach the preset number, then obtain candidate elements whose ranking is between the first preset proportion and the third preset proportion, and the third preset proportion is higher than the first preset proportion;

[0138] A target element is generated according to the candidate elements between the first preset proportion and the third preset proportion.

[0139] Among them, when the target number of target elements determined in step S301 does not reach the preset number, in order to prevent the designer from being unable to select more usable elements later, candidate elements ranked between the first preset proportion and the third preset proportion can be obtained; and additional target elements can be generated based on the candidate elements ranked between the first preset proportion and the third preset proportion.

[0140] For example, the preset number is 5, the target number is 3, the first preset proportion is 20%, and the third preset proportion is 60%. Since the target number is less than the preset number, the candidate elements ranked in the top 20% to the top 60% are obtained, and additional target elements are generated based on the candidate elements in pairs.

[0141] Specifically, the method of generating additional target elements based on the pairwise candidate elements can be: extracting key features, such as color features, shape features, texture features, etc., from two existing elements respectively. Fusion of the features of the two elements to create a new feature combination. For example, the colors of the two elements can be mixed by averaging, interpolation, etc. For example, mixing blue and yellow in a certain proportion may produce new colors such as green. Or combining the shape characteristics of the two elements to generate a new shape. Or superimposing or mixing the texture features of the two elements to create an element with a new texture. For example, if one element has a rough texture and the other has a smooth texture, an element with a medium roughness texture can be generated. Based on the fused features, the new target element is generated using image generation technology or graphics drawing tools. For example, using deep learning models such as generative adversarial networks (GANs) and variational autoencoders (VAEs), a new image is generated based on the fused feature vector.

[0142] In this way, by sorting the candidate elements in descending order according to the element score, and determining the candidate elements with the first preset proportion in the sorting as the target elements, the most valuable and important elements in the image can be quickly screened out. This helps designers or users to give priority to and use high-quality elements among many elements, improve work efficiency and decision-making accuracy. And by introducing the third preset proportion, the screening range can be expanded when the number of targets is insufficient, and target elements can be generated from more candidate elements. This increases the flexibility of the design, and designers can choose from different screening ranges according to actual conditions to meet different design needs and creative directions. By generating additional target elements, the diversity of elements can be increased. Designers can choose from more elements with different characteristics, enrich design schemes, and improve the innovation and uniqueness of the design.

[0143] See also Figure 4 , after step S105 in some embodiments, the steps further include but are not limited to steps S401 to S403:

[0144] Step S401, in response to a reference element acquisition instruction, acquire a reference element and an expected description of the element.

[0145] Among them, in order to avoid the problem that the target elements provided to designers are not adopted and used, the embodiment of the present application also provides an element generation method, specifically a method in which the designer generates elements by providing reference elements and expected element descriptions of the expected elements to be used.

[0146] Step S402: extracting reference information of the reference element.

[0147] The generation process can be more finely controlled through the control network model (ControlNet). It guides the output of the image element generation model by introducing additional conditional inputs such as sketches, edge maps, depth maps, etc. In this process, the designer's reference element can be used as one of the conditional inputs of ControlNet.

[0148] For example, if the reference element is a sketch, ControlNet can use the outline and structure information of this sketch to guide the generation process so that the generated result is more consistent with the characteristics of the sketch.

[0149] Step S403: input the reference information and the expected description of the element into an image element generation model to generate an expected element.

[0150] Among them, the image element generation model receives reference information extracted from the reference element, which may include color features, shape features, texture features, etc. At the same time, the image element generation model also receives the desired description of the element in text form. This may be some keywords or phrases that describe the specific properties of the element you want to generate, such as "bright blue circle", "triangle with metallic texture", etc. The image element generation model needs to understand these text descriptions and convert them into feature representations that can be used to generate images.

[0151] The image element generation model fuses the features corresponding to the reference information and the desired description of the element. This may involve combining the extracted reference element features with the features understood from the text description to determine the specific feature vector of the generated image. For example, if the reference element is a red square and the desired description is "blue circle", the image element generation model may adjust the color feature from red to blue and the shape feature from square to circle. Based on the fused feature vectors, the image element generation model performs a random sampling and generation process. It generates parts of the image step by step through the structure of the generation model. This process is an iterative process, and the model continuously adjusts the generated image to make it closer to the target feature. The generated image may go through multiple iterations and optimizations until certain convergence conditions are met. Finally, the image element generation model outputs the generated desired element, that is, one or more images that meet the reference information and the desired description of the element.

[0152] In addition, when there is no generated element among the displayed generated elements that the designer wishes to use as a reference element, the designer may hand-draw an element to use as a reference element.

[0153] In this way, when the target elements provided to the designer are not adopted and used, the risk of non-adoption is reduced: by providing reference elements and element expected descriptions to generate elements, the problem of the target elements provided by the designer not being adopted and used can be greatly reduced. This method makes the generated elements more targeted and in line with design requirements, and increases the possibility of being adopted.

[0154] Before step S401 in some embodiments, the following steps are also performed, including but not limited to:

[0155] Each of the generated elements is displayed in a specified interface.

[0156] Among them, Figure 5 As shown, Figure 5 The interface diagram of the formulation interface provided by the embodiment of the present application. After the generated element is obtained, a plurality of element display controls 10 are included in the specified interface, and each element display control 10 displays a generated element, such as a circle, a triangle, a five-pointed star, etc.

[0157] See also Figure 6 Step S401 may include but is not limited to steps S501 to S503:

[0158] Step 501: In response to a selection operation on the generated element, determine a target generated element selected by the selection operation, and determine the target generated element as a reference element.

[0159] When the designer wishes to select a generated element as a reference element, Figure 5 A selection operation is performed on an element display control 10 in the selection operation, so that the electronic device determines the target generation element selected by the selection operation and determines the target generation element as the reference element.

[0160] Specifically, the selection operation may be a mouse click operation or a touch operation by the designer, which is not limited here.

[0161] Step 502: display a desired description control in the specified interface.

[0162] Among them, Figure 7 As shown, Figure 7 Another interface diagram of the formulation interface provided by the embodiment of the present application, wherein after the designer selects a generated element as a reference element, an expected description control 20 will be displayed in the specified interface.

[0163] Step 503: In response to an input instruction for the expected description control, determine the description content of the input instruction in the expected description control, and determine the description content as the element expected description.

[0164] The designer may input the expected description control 20 to input the description content of the element that he / she expects to generate, such as "bright blue circle", to determine the description content as the expected description of the element.

[0165] In this way, designers can easily interact with the system by selecting through common operations such as mouse clicks or touch. This intuitive operation method lowers the threshold for use and improves the user experience. Displaying the expected description control and responding to input instructions to determine the expected description of the element allows designers to accurately express their specific requirements for the newly generated elements. This enhances the designer's controllability over the generation process and ensures that the generated elements are more in line with their expectations.

[0166] See also Figure 8 In some embodiments, step S104 may include but is not limited to step S601:

[0167] Step S601: sort the target elements in descending order of similarity, and determine the target elements with the second preset proportion in the last order as difference elements.

[0168] Among them, after determining the similarity corresponding to each target element, each target element is sorted in descending order of similarity to obtain a target element sequence with descending similarity, and the target element with the second preset proportion in the target element sequence is determined as a difference element.

[0169] In this way, by identifying the difference elements as the elements that are unlikely to be used by designers, designers can quickly understand which elements are not suitable in the current design, thereby avoiding wasting time and energy on these elements for further consideration. This helps improve design efficiency and enables designers to focus more on elements with greater potential.

[0170] Specifically, the present application scheme is explained by taking the financial product recommendation scenario as an example. When a designer needs to design a product poster image for a certain financial product, the electronic device will crawl the design poster image from the design image library on the Internet, and input the design poster image into the element scoring model to score each candidate element in the design poster image. The element score of candidate element a1 is 90 points, the element score of candidate element a2 is 60 points, the element score of candidate element b1 contained in target image B is 85 points, the element score of candidate element c1 contained in target image C is 80 points, and the element score of candidate element c2 is 50 points. The first 40% of the candidate elements of the five candidate elements, that is, candidate element a1 and candidate element b1, are determined as target elements. The similarities between historical elements and target elements in the historically designed financial product poster image are calculated in turn, and the target elements with similarities in the bottom 30% are determined as difference elements. The difference elements are removed from the target elements to obtain generated elements, and the generated elements are displayed through a specified interface to recommend elements to designers. Finally, the designer will use at least one generated element to design a financial product poster image.

[0171] When the designer does not adopt any of the displayed generated elements, the designer can select a generated element (such as a circle) from the displayed generated elements as a reference element, or hand-draw an element as a reference element, and input the desired description of the element (such as "bright blue circle"). The electronic device inputs the circle and the "bright blue circle" into the image element generation model to generate the expected element expected by the designer. Ultimately, the designer will use the expected element to design a financial product poster image.

[0172] The embodiment of the present application obtains a target image, inputs the target image into an element scoring model, determines the element score of each candidate element in the target image; based on the element score of each candidate element, selects the target element from multiple candidate elements; determines the similarity between each target element and a historical element; based on the similarity between each target element and a historical element, selects the difference element from multiple target elements; removes the difference element from the target element to obtain a generated element. In this way, high-quality elements that are close to the business are gradually selected through scoring screening and similarity screening, which can improve the relevance of the generated image elements to the business, and improve the accuracy and practicality of the generated image elements.

[0173] See also Fig. 9 The embodiment of the present application further provides an image element generation device, which can implement the above-mentioned image element generation method, and the device includes:

[0174] A first acquisition unit is used to acquire a target image, input the target image into an element scoring model, and determine an element score of each candidate element in the target image;

[0175] A first screening unit, configured to screen a target element from a plurality of candidate elements based on an element score of each candidate element;

[0176] A determination unit, used to determine the similarity between each of the target elements and the historical elements;

[0177] A second screening unit, configured to screen out difference elements from the plurality of target elements based on the similarity between each target element and the historical element;

[0178] The elimination unit is used to eliminate the difference element in the target element to obtain a generated element.

[0179] In some embodiments, the first screening unit is used to:

[0180] Inputting the target image into an element scoring model to perform image encoding on the target image to obtain image features;

[0181] Get the preset element features corresponding to the preset elements of each element type;

[0182] Matching each of the preset element features with the image features, and selecting candidate elements matching the image features from the preset elements;

[0183] Determine a candidate position of each candidate element in the target image;

[0184] An element score of each candidate element is determined based on the candidate element type and the corresponding candidate position of each candidate element.

[0185] In some embodiments, the apparatus further comprises:

[0186] A second acquisition unit, configured to acquire a reference element and an expected description of the element in response to a reference element acquisition instruction;

[0187] an extraction unit, used for extracting reference information of the reference element;

[0188] An input unit is used to input the reference information and the expected description of the element into an image element generation model to generate an expected element.

[0189] In some embodiments, the apparatus further comprises:

[0190] A display unit, used to display each of the generated elements in a specified interface;

[0191] The response unit comprises:

[0192] A first determining subunit is used to determine, in response to a selection operation on the generation element, a target generation element selected by the selection operation, and determine the target generation element as a reference element;

[0193] A display subunit, used for displaying a desired description control in the specified interface;

[0194] The second determining subunit is configured to, in response to an input instruction for the expected description control, determine the description content of the input instruction in the expected description control, and determine the description content as the element expected description.

[0195] In some embodiments, the apparatus further comprises:

[0196] A third acquisition unit, used to acquire the usage frequency of each of the generated elements;

[0197] The display subunit is used for:

[0198] Determine the display order of each of the generated elements in descending order of usage frequency;

[0199] Each of the generated elements is displayed in the designated interface in the display order of the generated elements.

[0200] In some embodiments, the first screening unit is used to:

[0201] The candidate elements are sorted in descending order according to the element scores, and the candidate element with a first preset proportion in the sorting is determined as the target element.

[0202] In some embodiments, the second screening unit is used to:

[0203] The target elements are sorted in descending order of similarity, and the target element with the second preset proportion in the last order is determined as a difference element.

[0204] The specific implementation of the image element generating device is substantially the same as the specific implementation of the above-mentioned image element generating method, and will not be described in detail here.

[0205] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image element generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0206] See also Fig.10 , Fig.10The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0207] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0208] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image element generation method of the embodiment of this application;

[0209] Input / output interface 903, used to implement information input and output;

[0210] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0211] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0212] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0213] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image element generation method is implemented.

[0214] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] The image element generation method, image element generation device, electronic device and storage medium provided by the embodiment of the present application obtain a target image, input the target image into an element scoring model, determine the element score of each candidate element in the target image; based on the element score of each candidate element, select the target element from multiple candidate elements; determine the similarity between each target element and a historical element; based on the similarity between each target element and a historical element, select the difference element from multiple target elements; remove the difference element from the target element to obtain a generated element. In this way, high-quality elements that are close to the business are gradually selected through scoring screening and similarity screening, which can improve the relevance of the generated image elements to the business, and improve the accuracy and practicality of the generated image elements.

[0216] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0217] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0218] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0220] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0222] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0226] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating an image element, characterized in that: include: Acquire a target image, input the target image into an element scoring model, and determine an element score of each candidate element in the target image; Based on the element score of each candidate element, screening out a target element from a plurality of candidate elements; Determining the similarity between each of the target elements and the historical elements; Based on the similarity between each of the target elements and the historical elements, filter out the difference elements from the multiple target elements; The difference element in the target element is removed to obtain a generated element.

2. The image element generation method according to claim 1, characterized in that: The step of inputting the target image into an element scoring model to determine an element score of each candidate element in the target image includes: Inputting the target image into an element scoring model to perform image encoding on the target image to obtain image features; Get the preset element features corresponding to the preset elements of each element type; Matching each of the preset element features with the image features, and selecting candidate elements matching the image features from the preset elements; Determine a candidate position of each candidate element in the target image; An element score of each candidate element is determined based on the candidate element type and the corresponding candidate position of each candidate element.

3. The image element generation method according to claim 1, characterized in that: After removing the difference element from the target element to obtain the generated element, the method further includes: In response to the reference element acquisition instruction, acquire the reference element and the element expected description; extracting reference information of the reference element; The reference information and the element expected description are input into an image element generation model to generate an expected element.

4. The image element generation method according to claim 3, characterized in that: Before obtaining the reference element and the element expected description in response to the reference element obtaining instruction, the method further includes: Displaying each of the generated elements in a specified interface; The step of obtaining the reference element and the expected description of the element in response to the reference element obtaining instruction includes: In response to a selection operation on the generation element, determining a target generation element selected by the selection operation, and determining the target generation element as a reference element; Displaying a desired description control in the specified interface; In response to an input instruction for the expected description control, the description content of the input instruction in the expected description control is determined, and the description content is determined as the element expected description.

5. The image element generation method according to claim 4, characterized in that: Before displaying each of the generated elements in the specified interface, the method further includes: Obtaining the usage frequency of each of the generated elements; The displaying of each generated element in the specified interface includes: Determine the display order of each of the generated elements in descending order of usage frequency; Each of the generated elements is displayed in the designated interface in the display order of the generated elements.

6. The image element generation method according to any one of claims 1 to 5, characterized in that: The step of selecting a target element from a plurality of candidate elements based on the element score of each candidate element comprises: The candidate elements are sorted in descending order according to the element scores, and the candidate element with a first preset proportion in the sorting is determined as the target element.

7. The image element generation method according to any one of claims 1 to 5, characterized in that: The step of selecting a difference element from a plurality of target elements based on the similarity between each target element and the historical element includes: The target elements are sorted in descending order of similarity, and the target element with the second preset proportion in the last order is determined as a difference element.

8. An image element generating device, characterized in that: include: A first acquisition unit is used to acquire a target image, input the target image into an element scoring model, and determine an element score of each candidate element in the target image; A first screening unit, configured to screen a target element from a plurality of candidate elements based on an element score of each candidate element; A determination unit, used to determine the similarity between each of the target elements and the historical elements; A second screening unit, configured to screen out difference elements from the plurality of target elements based on the similarity between each target element and the historical element; The elimination unit is used to eliminate the difference element in the target element to obtain a generated element.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the image element generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image element generation method according to any one of claims 1 to 7 is implemented.

Images