Method, system, storage medium and electronic device for generating autonomous digital art

Through independent creative digital art generation methods and systems, using data uploaded by users and design instructions, combined with artificial intelligence technology, digital art works that meet artistic characteristics are generated, solving the problems of high server consumption and high user operation difficulties in the existing technology in the existing technology. , realizes the combination of user subjective operations and aesthetic creation.

CN115115745BActive Publication Date: 2024-11-12BEIJING HUALU NEW MEDIA INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210726563.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-24
Publication Date
2024-11-12
Estimated Expiration
2042-06-24

AI Technical Summary

Technical Problem

Existing intelligent image generation applications have problems in server storage and performance consumption, and the learning cost and operation difficulty of independent creation platforms are high, and they cannot mobilize the creative interest of non-professional users, lack of sharing aesthetic knowledge, resulting in digital art works created by users. Lack of artistic characteristics.

Method used

Provide self-created digital art generation methods and systems, upload pictures and music data by users, combine design instructions, use the CPArtist module to extract subsets of features for semantic analysis, generate artistic feature models, retrieve picture materials with high matching degree, and use Generative adversarial network algorithms are edited to generate art works that meet users' intentions.

Benefits of technology

During the creation process of users, the subjective operation space is fully utilized, combined with the creative ability of artificial intelligence, and generate complete art works that have both user ideas and artistic characteristics, reducing server load and improving user experience and creative interest.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115115745B_ABST
    Figure CN115115745B_ABST
Patent Text Reader

Abstract

The present invention provides a method, system, storage medium and electronic device for generating self-created digital art, including the following steps: acquiring artworks and corresponding design instructions and storing them in the system; extracting text, audio and image features of the artworks and design instructions stored in the production process, and combining them into feature subsets; generating semantic descriptions and artistic feature models after semantic analysis of the feature subsets; retrieving picture art materials through the artistic feature model; after editing the picture art materials, generating multiple solutions according to preset evaluation criteria; and selecting solutions that meet the needs for publication. The present invention cooperates with the creation and retrieval capabilities of artificial intelligence, and by learning from mature digital artworks, ultimately presents a complete artwork that has both user ideas and artistic features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of intelligent art creation, and specifically relates to a method, system, storage medium and electronic equipment for generating autonomously created digital art. Background Art

[0002] With the rapid development of the Internet, more and more intelligent image generation applications have emerged. Generally speaking, these applications use AI to intelligently generate corresponding renderings based on the words given by users, which will greatly increase the consumption of server storage space, database data and server performance.

[0003] At the same time, the platforms that can be used for independent creation on the market currently have high learning costs and high operating difficulties for users, and are not suitable for use by non-professional personnel. In addition, the fields involved are too single, and there is no systematic sharing and penetration of aesthetic knowledge, which cannot mobilize users' creative interests. The present invention fully gives users room for subjective operation, and the process of user creation is also a process of cultivating and understanding aesthetic education. Combined with the creation and retrieval capabilities of artificial intelligence, by learning from mature digital art works, a complete art work that not only has user ideas but also conforms to artistic characteristics is finally presented.

[0004] In view of this, there is an urgent need to propose methods, systems, storage media and electronic devices for generating autonomous digital art, explore the user's potential expression intentions, reconstruct new artificial intelligence images, and enable users to freely create complete digital art works that are more in line with artistic characteristics. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to provide an autonomously creative digital art generation method, system, storage medium and electronic device to explore the user's potential expression intentions, reconstruct new artificial intelligence images, and enable users to arbitrarily create more complete digital art works that are more in line with artistic characteristics.

[0006] The autonomous creative digital art generation method of the present invention includes user production and CPArtist creation;

[0007] The user production comprises the following steps:

[0008] Obtain the image data, music data and the corresponding design descriptions uploaded by the user locally and store them in the system, or

[0009] Provide online drawing and drawing factor control including brushes, erasers and color drawing factors, and store the artwork and corresponding design instructions generated by users after online drawing in the system;

[0010] The CPArtist creation includes the following steps:

[0011] Extracting text, audio and image features of the artwork and design description stored during the production process and combining them into a feature subset;

[0012] After semantic analysis of the feature subset, a semantic description is generated. After combining the semantic description with the feature itself according to the weight ratio, multiple artistic feature models are obtained.

[0013] Retrieving a preset number of reference images with the highest matching degree through the art feature model, generating an evaluation index, selecting one of the art feature models as a search model according to the evaluation index, and retrieving picture art materials with a matching degree higher than a preset value through the search model;

[0014] After editing the picture art material, multiple solutions are generated according to the preset evaluation criteria;

[0015] Select a plan that meets your needs and publish it.

[0016] Furthermore, the painting factor control specifically includes:

[0017] Generate image sample features based on the design text description;

[0018] Preprocess image sample features to reduce computational effort;

[0019] Collect image data about the characteristics of image samples and establish corresponding databases and image indexes;

[0020] After training the database through a convolutional neural network, a predetermined number of pre-selected images are retrieved from the image index;

[0021] Construct an outer boundary generation algorithm based on a generative adversarial network, use the deep convolutional adversarial network CGAN algorithm based on the Keras framework to generate images for the MNIST dataset, and obtain a feature modification loss function based on the sparsity of the outer boundary graphics;

[0022] Max(x,y)=minGmaxXD(E x-y [logD(x,y)]), Gx(x,y)=-E x [log1-D(x,G(x))]

[0023] LcGAN(x,y)=Maix(x,y)+Gx(x,y)

[0024] Where x is the condition and y is the image after mapping, G represents the generator, D represents the discriminator, G(x) represents the sample generated by the generator based on the condition x, D(x,y) represents the probability that the discriminator sample (x,y) is a true sample, E represents the K-means distribution, Maix represents the maximum and minimum optimization value, and Gx represents the expected value of the noise data;

[0025] According to the sparsity of the outer boundary graphics, the loss function formula is modified as follows:

[0026]

[0027] Where m is the condition, n is the output boundary summary map after mapping, M is the total number of output samples, λ is the ratio hyperparameter, and LcGAN represents the standard generative adversarial network loss of sample nj generated according to condition mi. The first term of the loss function is the average value, which makes the generative adversarial network treat all target graphics equally; the second term is the minimum value, which makes the generative adversarial network tend to generate the best picture and prevent the picture from being blurred;

[0028] Generate drawing-aid graphics.

[0029] Furthermore, the text, audio and image features of the artwork and design description stored in the production process are extracted and combined into a feature subset, specifically:

[0030] Pre-process the text material: if the music material has lyrics, extract the lyrics first, and perform word segmentation processing together with the corresponding design description text, extract keyword information, so that each material image contains several corresponding tags;

[0031] Split the given text T into complete sentences, that is: T = [S1, S2, ..., S n ];

[0032] For each sentence, perform word segmentation and part-of-speech tagging, and filter out stop words;

[0033] Construct a candidate keyword graph model G = (V, E), add several vocabulary units after text segmentation to the graph model to form nodes;

[0034] By determining the weight distribution of word position, part of speech and domain features in the text, using keyword features such as part of speech, word length, TFIDF value, etc., the comprehensive weight is obtained through multi-feature fusion and the weight of each node is initialized;

[0035] The weight of each node is iteratively calculated using the following formula until convergence;

[0036]

[0037] Where d is the damping factor (0≤d≤1), usually set to 0.85, which represents the probability of any node in the graph jumping to other nodes; In(V i ) indicates pointing to node V i The set of all nodes of Out(V j ) represents node V j The set of all nodes pointed to;

[0038] Sort the node weights in reverse order to get the most important n words as candidate tags; if adjacent phrases are formed, they are combined into multi-word tags;

[0039] Pre-process the music materials: convert the remaining pure music materials into visual language, divide the music into five parts, including timbre, rhythm, melody, harmony and mode, define the music elements as color, point, line and surface, high and low changes of visual elements and graphic symbols, capture the relationship between music elements and visual language elements, and convert music elements into picture elements;

[0040] Extract audio elements based on CLIP, use contrast loss for feature extraction, and add multi-layer perception as a projection layer;

[0041] The cross-projection loss function is as follows:

[0042] CLoss=L(f(Image),Audio)+L(Image,g(Audio)))

[0043] Among them, △f / g: projection function, L: contrast loss function

[0044] Preprocess the image material: remove noise signals from the image through bilateral filtering algorithm and perform feature extraction;

[0045] Extraction of hue and color matching: Calculate the pixel mean of all pixels in the input image; divide the color space into multiple color regions, and determine the color region in the color space to which each pixel belongs according to the pixel value of each pixel;

[0046] Extraction of image features: Extraction of brightness, edge, texture, etc., and extraction of shape, histogram, and main components through transformation processing module;

[0047] Divide the image into PXQ partitions, apply the delineation function to the sub-image regions, and obtain n feature vectors of these regions;

[0048] The histogram statistical features of the sub-image area are used to describe the distribution of pixel gray levels in the image, which effectively describes the inherent characteristics of the texture and obtains the corresponding feature vector;

[0049] The grayscale co-occurrence matrix of the sub-image area is calculated, and the corresponding feature vector is obtained by describing the joint distribution of two pixels with spatial position relationship;

[0050] The n feature vectors are combined into a new feature subset through linear combination transformation.

[0051] Furthermore, semantic description is generated after semantic analysis of feature subsets. After combining the semantic description with the features themselves according to weight proportions, multiple art feature models are obtained, specifically:

[0052] The new feature subset is sent to the analysis processing model as the global information of the image. Through the semantic analysis of the image, the user's potential intention is processed into text and combined with the image features to form a more comprehensive art feature model. The analysis processing model uses a two-layer fully convolutional neural network.

[0053] After receiving the information, the results of the two links of independence and linear superposition are predicted respectively to generate a semantic description;

[0054] The semantic description and the extracted feature subsets are combined according to the weight ratio to obtain multiple artistic feature models.

[0055] Further, a preset number of reference images with the highest matching degree are retrieved through the art feature model, an evaluation index is generated, and an art feature model is selected as a search model according to the evaluation index, and image art materials with a matching degree higher than a preset value are retrieved through the search model, specifically:

[0056] Retrieving a preset number of reference images with the highest matching value from a picture art library using an art feature model;

[0057] Obtaining evaluation indicators of the art feature model according to the preset number of reference images;

[0058] When the evaluation index of the art feature model meets the preset conditions, selecting the art feature model as the search model;

[0059] In the case that the evaluation of the art feature model does not meet the preset conditions, a preset number of reference pictures with the highest matching degree are retrieved from the picture art material library using a deep learning model. If the matching degree is still less than 70%, the retrieval is terminated and the model is manually processed;

[0060] After retrieving a preset number of picture art materials with a matching degree of 70%, proceed to the next step.

[0061] Furthermore, after the image art material is edited, multiple solutions are generated according to preset evaluation criteria, specifically:

[0062] Create a new canvas of the same size as the original material, upload the original material image and the matching image in the database, and locate the pixel coordinates;

[0063] Through the artistic feature model, the original material layer is intelligently optimized;

[0064] Edit 4 action presets, including size action, crop action, occlusion action, and correction action;

[0065] Edit 3 system modules, including light effects, colors and details, and recognize the thresholds of brightness, contrast, highlights and shadows, and make intelligent adjustments;

[0066] You can optimize the image by adding sharpening action and texture action to the image;

[0067] According to the preset evaluation criteria, the image results with different weights are output, and the three best solutions are comprehensively evaluated and provided to the user.

[0068] Furthermore, when selecting a plan to publish, the user performs editing and publishing, and the editing process includes adding text, borders, stickers and codes.

[0069] The present invention also provides an autonomous creative digital art generation system, comprising:

[0070] User-made modules;

[0071] The user production module includes a local upload submodule and an online production submodule;

[0072] The local upload submodule obtains the picture data, music data and the corresponding design descriptions uploaded by the user locally and stores them in the system;

[0073] The online production submodule provides online drawing and painting factor control including brushes, erasers and color painting factors, and stores the artwork and corresponding design instructions generated by the user after online drawing in the system;

[0074] CPArtist module;

[0075] The CPArtist module includes a feature extraction module, an expression intention exploration module, a public domain material acquisition module, a feasibility creation module and a user decision push module;

[0076] The feature extraction module extracts text, audio and image features of the artwork and design description stored during the production process and combines them into a feature subset;

[0077] The expression intention exploration module is used to generate a semantic description after semantically parsing the feature subset, and to obtain multiple art feature models after combining the semantic description with the features themselves according to weight proportions;

[0078] The public domain material acquisition module retrieves a preset number of reference images with the highest matching degree through the art feature model, generates an evaluation index, selects one of the art feature models as a search model according to the evaluation index, and retrieves picture art materials with a matching degree higher than a preset value through the search model;

[0079] The feasibility creation module is used to generate multiple solutions according to preset evaluation criteria after editing the picture art material;

[0080] The user decision push module is used to select a solution based on evaluation criteria and user preferences and then publish it.

[0081] The present invention also provides a computer storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the autonomous creative digital art generation method.

[0082] The present invention further provides an electronic device, comprising:

[0083] processor;

[0084] a memory for storing instructions executable by the processor;

[0085] The processor is used to execute the autonomous creative digital art generation method.

[0086] The above technical solution of the present invention has the following advantages compared with the prior art:

[0087] The present invention fully provides space for users' subjective operation. The process of user creation is also a process of aesthetic education and understanding. Combined with the creation and retrieval capabilities of artificial intelligence, by learning from mature digital art works, a complete art work that not only reflects the user's thoughts but also conforms to artistic characteristics is finally presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is a schematic diagram of the structure of a user production module in a system provided by an embodiment of the present invention;

[0089] Figure 2 The structure of the CPArtist module in the system provided by the embodiment of the present invention is shown in FIG. Figure 1 ;

[0090] Figure 3 The structure of the CPArtist module in the system provided by the embodiment of the present invention is shown in FIG. Figure 2 ;

[0091] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] This embodiment is an autonomous creative digital art generation system, such as Figure 1-3 As shown, including:

[0094] User-made modules;

[0095] The user production module includes a local upload submodule and an online production submodule;

[0096] The local upload submodule obtains the picture data, music data and the corresponding design descriptions uploaded by the user locally and stores them in the system;

[0097] The online production submodule provides online drawing and painting factor control including brushes, erasers and color painting factors, and stores the artwork and corresponding design instructions generated by the user after online drawing in the system;

[0098] CPArtist module;

[0099] The CPArtist module includes a feature extraction module, an expression intention exploration module, a public domain material acquisition module, a feasibility creation module and a user decision push module;

[0100] The feature extraction module extracts text, audio and image features of the artwork and design description stored during the production process and combines them into a feature subset;

[0101] The expression intention exploration module is used to generate a semantic description after semantically parsing the feature subset, and to obtain multiple art feature models after combining the semantic description with the features themselves according to weight proportions;

[0102] The public domain material acquisition module retrieves a preset number of reference images with the highest matching degree through the art feature model, generates an evaluation index, selects one of the art feature models as a search model according to the evaluation index, and retrieves picture art materials with a matching degree higher than a preset value through the search model;

[0103] The feasibility creation module is used to generate multiple solutions according to preset evaluation criteria after editing the picture art material;

[0104] The user decision push module is used to select a solution based on evaluation criteria and user preferences and then publish it.

[0105] The autonomous creative digital art generation method of this embodiment includes online creation by users and creation by CPArtist, such as Figure 1-3 shown.

[0106] The user online production includes:

[0107] Step (1): The user can choose the production method independently: Method 1 is to upload pictures or music and attach design instructions; Method 2 is to make it online and create and draw using brushes, erasers, and color tools.

[0108] Step (2): If the user chooses to draw online, the system will control the tool parameters through the "drawing factor" to assist the user's behavior; the system itself provides the most basic drawing tools (brush, eraser, color) to reduce the difficulty of user operation. At the same time, through the "drawing factor" to control the properties of the brush, eraser and color, enhance the user's feel, produce more visual changes, and assist the user's artistic expression.

[0109] Drawing factor control description:

[0110] 2.1 Input the design description text according to the first step, analyze the text based on the CLIP (Contrastive Language-Image Pre-Training) model, and generate image sample features;

[0111] 2.2 According to the image sample features generated in step 2.1, the Google MobileNetV3 model and lightweight network design model and knowledge distillation are used to reduce the amount of calculation, and the model architecture of the output layer is adjusted to a K-means model;

[0112] 2.3 Use the Scrapy library in Python to collect image data from the iNaturalist website and store it in the database. Establish an image index system for the knowledge data graph based on the Baidu knowledge data graph data interface;

[0113] 2.4 Use the collected image dataset to perform transfer learning on the convolutional neural network, and use the Mosaic data enhancement method to avoid overfitting in the training process;

[0114] 2.5 Based on the sample features of the image generated in step 2.1, the image indexing system established in step 2.3 has a database with 500 images of different categories, and 5 images are selected for each category, for a total of 2,500 pre-selected images;

[0115] 2.6 Based on step 2.5, an outer boundary generation algorithm based on a generative adversarial network is constructed. Based on the keras framework, the deep convolutional adversarial network CGAN algorithm is used to generate images for the MNIST dataset, and the feature modification loss function is obtained according to the sparsity of the outer boundary graphics.

[0116] Max(x,y)=minGmaxXD(E x-y [logD(x,y)],Gx(x,y)=-E x [log1-D(x,G(x))]

[0117] LcGAN(x,y)=Maix(x,y)+Gx(x,y)

[0118] Where x is the condition and y is the image after mapping, G represents the generator, D represents the discriminator, G(x) represents the sample generated by the generator based on the condition x, D(x,y) represents the probability that the discriminator sample (x,y) is a real sample, E represents the K-means distribution, Maix represents the maximum and minimum optimization value, and Gx represents the expected value of the noise data.

[0119] According to the sparsity of the outer boundary graphics, the loss function formula is modified as follows:

[0120]

[0121] Among them, m is the condition, n is the output boundary summary map after mapping, M is the total number of output samples, λ is the ratio hyperparameter, and LcGAN represents the standard generative adversarial network loss of sample nj generated according to condition mi. The first term of the loss function is the average value, which makes the generative adversarial network treat all target graphics equally; the second term is the minimum value, which makes the generative adversarial network tend to generate the best picture and prevent the picture from being blurred.

[0122] 2.7 Use CPArtist.js and vue.js to build the painting module and generate painting auxiliary graphics.

[0123] Step (3): Before submitting the work, the user needs to fill in the "Design Description". The "Design Description" includes the name of the work and the introduction of the work, the purpose of creation, artistic ideas, etc., and then submit it.

[0124] The first step is to extract features:

[0125] Step (1): Input original materials: pictures, music and corresponding design instructions for each material.

[0126] Step (2): Preprocess the text material: If the music material has lyrics, extract the lyrics first, and perform word segmentation processing together with the corresponding "design description" text, extract keyword information, so that each material image contains several corresponding tags.

[0127] Step (3): Segment the given text T into complete sentences, i.e.: T = [S1, S2, ..., S n ].

[0128] Step (4): For each sentence, perform word segmentation and part-of-speech tagging, filter out stop words, and only retain words with specified parts of speech, such as nouns, verbs, adjectives, etc. These words form candidate keywords.

[0129] Step (5): Construct a candidate keyword graph model G = (V, E), add several vocabulary units after text segmentation to the graph model to form nodes. When two words appear together in a window of length K (generally in the range of 2 to 10), it means that there is an edge between the two nodes.

[0130] Step (6): Use the co-occurrence relationship to construct an edge between any two points. An edge exists between two nodes only if their corresponding words co-occur in a window of length K. Add it to the graph model to form an undirected and unweighted edge graph.

[0131] Step (7): By determining the weight distribution of word position, part of speech and domain features in the text, using keyword features such as part of speech, word length, TFIDF value, etc., a comprehensive weight is obtained through multi-feature fusion and the weight of each node is initialized.

[0132] Step (8): The weight of each node is calculated iteratively by the following formula until convergence. In the formula, d is the damping factor (0≤d≤1), usually 0.85, which represents the probability value of any node in the graph jumping to other nodes; In(V i ) indicates pointing to node V i The set of all nodes of Out(V j ) represents node V j The collection of all nodes pointed to.

[0133]

[0134] Step (9): Sort the node weights in reverse order to obtain the most important n words as candidate tags.

[0135] Step (10): If adjacent phrases are formed, they are combined into multi-word tags.

[0136] Step (11): Preprocess the music material: Perform visual language conversion on the remaining pure music material, divide the music into five parts, including timbre, rhythm, melody, harmony and mode, and define the music elements as colors, points, lines and surfaces, high and low changes of visual elements and graphic symbols through a set of visual languages ​​independently developed. The relationship between music elements and visual language elements is captured, and the music elements can be converted into picture elements.

[0137] Step (12): extract audio elements based on CLIP, use contrast loss for feature extraction, and add multi-layer perception as a projection layer;

[0138] The cross-projection loss function is as follows:

[0139] CLoss=L(f(Image),Audio)+L(Image,g(Audio)))

[0140] △f / g: projection function, L: contrast loss function

[0141] Step (13): Preprocess the image material: remove noise signals from the image using a bilateral filtering algorithm and perform feature extraction.

[0142] Step (14): Extraction of hue and color matching: Calculate the pixel mean of all pixels in the input image; divide the color space into multiple color regions, and determine the color region in the color space to which each pixel belongs according to the pixel value of each pixel. The color space is an abstract mathematical model that describes the method of using a set of values ​​to represent color, such as HSV (hue, saturation and brightness) or RGB mode, which both represent the division of color regions in the color space. The range of values ​​can represent different colors. For example, H in HSV represents hue, S represents saturation, and V represents brightness. The value range of H is "0-360", the value range of S is "0, 1", and the value range of V is "0.1". Different combinations of H, S, and V can represent different colors.

[0143] Step (15): Extraction of image features: Through the natural feature module, brightness, edge, texture, etc. are extracted; through the transformation processing module, shape, histogram and main components are extracted.

[0144] Description of image feature extraction:

[0145] 15.1 The image is divided into PXQ partitions, and the sub-image region feature extraction effectively avoids the loss of local difference information.

[0146] 15.2 Perform a delineation function on the sub-image region, i.e., the number of pixels on the region boundary, the total number of pixels, the average value of all pixels, the sorted median value of all pixels, etc., to obtain n feature vectors of these regions.

[0147] 15.3 Perform histogram statistical features on the sub-image area, and by describing the distribution of pixel gray levels in the image, effectively describe the inherent characteristics of the texture and obtain the corresponding feature vector.

[0148] 15.4 Perform a grayscale co-occurrence matrix on the sub-image area and obtain the corresponding feature vector by describing the joint distribution of two pixels with a spatial position relationship.

[0149] 15.5 Combine n feature vectors into a new feature subset through linear combination transformation.

[0150] The second step is to explore the user's potential expression intention:

[0151] Step (1): The new feature subset is sent to the analysis processing model as the global information of the image. Through the semantic analysis of the image, the user's potential intention is processed into text and combined with the image features to form a more comprehensive art feature model.

[0152] Step (2): The parsing processing model uses a two-layer fully convolutional neural network. After receiving the information, it performs independent and linear superposition result predictions to generate a more accurate semantic description.

[0153] Step (3): Combine the semantic description and the extracted feature subset according to the weight ratio to obtain multiple artistic feature models.

[0154] The third step is to obtain art materials in the public domain.

[0155] Step (1): Use the art feature model to retrieve a preset number of reference images with the highest matching value from the image art library.

[0156] Step (2): obtaining evaluation indicators of the art feature model based on the preset number of reference images.

[0157] Step (3): When the evaluation index of the art feature model meets the preset conditions, the art feature model is selected as the search model.

[0158] Step (4): When the evaluation of the art feature model does not meet the preset conditions, a deep learning model is used to retrieve a preset number of reference images with the highest matching degree from the image art material library. If the matching degree is still lower than 70%, the retrieval is terminated and the model is manually processed.

[0159] Step (5): Finally, a preset number of picture art materials with a matching degree of 70% are retrieved and entered into the creation phase.

[0160] The fourth step is to create a feasible plan.

[0161] Step (1): Create a new canvas of the same size as the original material, upload the original material image and the matching image in the database, and locate the pixel coordinates.

[0162] Step (2): Intelligently optimize the model of the original material layer through the artistic feature model.

[0163] Step (3): The intelligent optimization model edits four action presets for the shape adjustment module, including size action, cropping action, occlusion action, and correction action.

[0164] Step (4): For the color adjustment mode, edit three system modules, including light effects, color and details; identify the thresholds of brightness, contrast, highlights and shadows, and make intelligent adjustments.

[0165] Step (5): Finally, the detail module will add sharpening action and texture action to the image to complete the image optimization.

[0166] Step (6): Based on the preset evaluation criteria (artistic quality, uniqueness, creativity, design sense, expressiveness, connotation), different weights are given and different output image results are obtained. The three best solutions are comprehensively evaluated and provided to the user.

[0167] The fifth step is to push user decisions.

[0168] Step (1): The user selects one solution based on the evaluation criteria and his / her own preferences.

[0169] Step (2): Users can choose to add text, borders, stickers and codes, and publish them.

[0170] Below, reference Figure 4 To describe an electronic device according to an embodiment of the present application. Figure 4 A block diagram of an electronic device according to an embodiment of the present application is shown.

[0171] like Figure 4 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0172] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0173] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the autonomous creative digital art generation method and / or other desired functions of the various embodiments of the present application described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0174] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0175] When the electronic device is a stand-alone device, the input device 13 may be a communication network connector, which is used to receive the collected input signals from the first device and the second device.

[0176] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0177] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0178] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0179] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for predicting infection categories according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0180] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0181] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, cause the processor to execute the steps of the method for predicting infection categories according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0182] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0183] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0184] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0185] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0186] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0187] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An autonomous creative digital art generation method, characterized in that: Including user-made and CPArtist creation; The user production comprises the following steps: Obtain the image data, music data and the corresponding design descriptions uploaded by the user locally and store them in the system, or Provide online drawing and drawing factor control including brushes, erasers and color drawing factors, and store the artwork and corresponding design instructions generated by users after online drawing in the system; The CPArtist creation includes the following steps: Extracting text, audio and image features of the artwork and design description stored during the production process and combining them into a feature subset; After semantic analysis of the feature subset, a semantic description is generated. After combining the semantic description with the feature itself according to the weight ratio, multiple artistic feature models are obtained. Retrieving a preset number of reference images with the highest matching degree through the art feature model, generating an evaluation index, selecting one of the art feature models as a search model according to the evaluation index, and retrieving picture art materials with a matching degree higher than a preset value through the search model; After editing the picture art material, multiple solutions are generated according to the preset evaluation criteria; Select a plan that meets your needs and publish it.

2. The autonomous creative digital art generation method according to claim 1, characterized in that: The painting factor control specifically includes: Generate image sample features based on the design text description; Preprocess image sample features to reduce computational effort; Collect image data about the characteristics of image samples and establish corresponding databases and image indexes; After training the database through a convolutional neural network, a predetermined number of pre-selected images are retrieved from the image index; Construct an outer boundary generation algorithm based on a generative adversarial network, use the deep convolutional adversarial network CGAN algorithm based on the Keras framework to generate images for the MNIST dataset, and obtain a feature modification loss function based on the sparsity of the outer boundary graphics; Maix(x,y)=minGmaXD(E x-y [logD(x,y)]),Gx(x,y)=-E x [log1-D(x,G(x))] LcGAN(x,y)=Maix(x,y)+Gx(x,y) Where x is the condition and y is the image after mapping, G represents the generator, D represents the discriminator, G(x) represents the sample generated by the generator based on the condition x, D(x,y) represents the probability that the discriminator sample (x,y) is a true sample, E represents the K-means distribution, Maix represents the maximum and minimum optimization value, and Gx represents the expected value of the noise data; According to the sparsity of the outer boundary graphics, the loss function formula is modified as follows: Where m is the condition, n is the output boundary summary map after mapping, M is the total number of output samples, λ is the ratio hyperparameter, and LcGAN represents the standard generative adversarial network loss for sample nj generated according to condition mi; The first term of the loss function is the average value, which makes the generative adversarial network treat all target graphics equally; the second term is the minimum value, which makes the generative adversarial network tend to generate the best picture and prevent the picture from being blurred; Generate drawing-aid graphics.

3. The autonomous creative digital art generation method according to claim 2, characterized in that: The text, audio and image features of the artwork and design description stored in the production process are extracted and combined into feature subsets, specifically: Pre-process the text material: if the music material has lyrics, extract the lyrics first, and perform word segmentation processing together with the corresponding design description text, extract keyword information, so that each material image contains several corresponding tags; Split the given text T into complete sentences, that is: T = [S1, S2, ..., S n ]; For each sentence, perform word segmentation and part-of-speech tagging, and filter out stop words; Construct a candidate keyword graph model G = (V, E), add several vocabulary units after text segmentation to the graph model to form nodes; By determining the weight distribution of word position, part of speech and field features in the text, using the keyword features of part of speech, word length and TFIDF value, the comprehensive weight is obtained through multi-feature fusion and the weight of each node is initialized; The weight of each node is iteratively calculated using the following formula until convergence; Where d is the damping factor, 0≤d≤1, and its value is 0.85, which indicates the probability value of any node in the graph jumping to other nodes; In(V i ) indicates pointing to node V i The set of all nodes of Out(V j ) represents the node V j The set of all nodes pointed to; Sort the node weights in reverse order to get the most important n words as candidate tags; If adjacent phrases are formed, they are combined into multi-word tags; Pre-process the music materials: convert the remaining pure music materials into visual language, divide the music into five parts, including timbre, rhythm, melody, harmony and mode, define the music elements as color, point, line and surface, high and low changes of visual elements and graphic symbols, capture the relationship between music elements and visual language elements, and convert music elements into picture elements; Extract audio elements based on CLIP, use contrast loss for feature extraction, and add multi-layer perception as a projection layer; The cross-projection loss function is as follows: CLoss=L(f(Image),Audio)+L(Image,g(Audio))) Among them, △f / g: projection function, L: contrast loss function Preprocess the image material: remove noise signals from the image through bilateral filtering algorithm and perform feature extraction; Extraction of hue and color matching: Calculate the pixel mean of all pixels in the input image; divide the color space into multiple color regions, and determine the color region in the color space to which each pixel belongs according to the pixel value of each pixel; Extraction of image features: Extraction of brightness, edge, texture, and through the transformation processing module, extraction of shape, histogram, and main components; Divide the image into PXQ partitions, apply the delineation function to the sub-image regions, and obtain n feature vectors of these regions; The histogram statistical features of the sub-image area are used to describe the distribution of pixel gray levels in the image, which effectively describes the inherent characteristics of the texture and obtains the corresponding feature vector; The grayscale co-occurrence matrix of the sub-image area is calculated, and the corresponding feature vector is obtained by describing the joint distribution of two pixels with spatial position relationship; The n feature vectors are combined into a new feature subset through linear combination transformation.

4. The autonomous creative digital art generation method according to claim 3, characterized in that: After semantic analysis of the feature subset, a semantic description is generated. After combining the semantic description with the feature itself according to the weight ratio, multiple artistic feature models are obtained, specifically: The new feature subset is sent to the analysis processing model as the global information of the image. Through the semantic analysis of the image, the user's potential intention is processed into text and combined with the image features to form a more comprehensive art feature model. The analysis processing model uses a two-layer fully convolutional neural network. After receiving the information, the results of the two links of independence and linear superposition are predicted respectively to generate a semantic description; The semantic description and the extracted feature subsets are combined according to the weight ratio to obtain multiple artistic feature models.

5. The autonomous creative digital art generation method according to claim 4, characterized in that: Retrieving a preset number of reference images with the highest matching degree through the art feature model, generating an evaluation index, selecting one of the art feature models as a search model according to the evaluation index, and retrieving picture art materials with a matching degree higher than a preset value through the search model, specifically: Retrieving a preset number of reference images with the highest matching value from a picture art library using an art feature model; Obtaining evaluation indicators of the art feature model according to the preset number of reference images; When the evaluation index of the art feature model meets the preset conditions, selecting the art feature model as the search model; In the case that the evaluation of the art feature model does not meet the preset conditions, a preset number of reference pictures with the highest matching degree are retrieved from the picture art material library using a deep learning model. If the matching degree is still less than 70%, the retrieval is terminated and the model is manually processed; After retrieving a preset number of picture art materials with a matching degree of 70%, proceed to the next step.

6. The autonomous creative digital art generation method according to claim 5, characterized in that: After the image art material is edited, multiple solutions are generated according to the preset evaluation criteria, specifically: Create a new canvas of the same size as the original material, upload the original material image and the matching image in the database, and locate the pixel coordinates; Through the artistic feature model, the original material layer is intelligently optimized; Edit at least four action presets, including size action, crop action, occlusion action, and correction action; Edit at least three system modules, including light effects, colors and details, and recognize the thresholds of brightness, contrast, highlights and shadows, and make intelligent adjustments; You can optimize the image by adding sharpening action and texture action to the image; Output image results with different weights according to the preset evaluation criteria, comprehensively evaluate at least three solutions, and provide them to the user.

7. The autonomous creative digital art generation method according to claim 6, characterized in that: When selecting a plan to publish, the user performs editing and publishing. The editing process includes adding text, borders, stickers and codes.

8. An autonomous digital art generation system, characterized in that: include: User-made modules; The user production module includes a local upload submodule and an online production submodule; The local upload submodule obtains the picture data, music data and the corresponding design descriptions uploaded by the user locally and stores them in the system; The online production submodule provides online drawing and painting factor control including brushes, erasers and color painting factors, and stores the artwork and corresponding design instructions generated by the user after online drawing in the system; CPArtist module; The CPArtist module includes a feature extraction module, an expression intention exploration module, a public domain material acquisition module, a feasibility creation module and a user decision push module; The feature extraction module extracts text, audio and image features of the artwork and design description stored during the production process and combines them into a feature subset; The expression intention exploration module is used to generate a semantic description after semantically parsing the feature subset, and to obtain multiple art feature models after combining the semantic description with the features themselves according to weight proportions; The public domain material acquisition module retrieves a preset number of reference images with the highest matching degree through the art feature model, generates an evaluation index, selects one of the art feature models as a search model according to the evaluation index, and retrieves picture art materials with a matching degree higher than a preset value through the search model; The feasibility creation module is used to generate multiple solutions according to preset evaluation criteria after editing the picture art material; The user decision push module is used to select a solution based on evaluation criteria and user preferences and then publish it.

9. A computer storage medium storing a computer program for executing the autonomous creative digital art generation method according to any one of claims 1 to 7.

10. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to execute the autonomous creative digital art generation method described in any one of claims 1-7 above.

Citation Information

Patent Citations

  • Self-adaptive intelligent banner advertisement picture generation method and self-adaptive intelligent banner advertisement picture generation system

    CN110889883A

  • Intelligent color matching method and system

    CN113345053A

  • Systems and methods for automating video editing

    US20210272599A1

  • Music recommendation method and apparatus, and computing device and medium

    WO2020034849A1