A large-scale Chinese painting generation interaction method and device based on multi-emotion recognition

Abstract

The application discloses a large-scale Chinese painting generation interaction method based on multi-emotion recognition, which comprises the following steps: obtaining input data of a user and extracting tone prompt words in the input data; inputting the tone prompt words into a pre-trained emotion analysis model to generate corresponding emotion labels; inputting the emotion labels into a diffusion model to generate a concept picture, and performing semantic retrieval on a pre-built Chinese painting element knowledge base to screen out a historical painter and corresponding life events that are most similar to the emotion labels; using natural language generation technology to perform text fusion on the emotion labels, image information in the concept picture and the screened life events to output a final personalized interpretation text; and finally, visualizing and outputting the interaction result. The application further provides a large-scale Chinese painting generation interaction device. The method provided by the application can intelligently understand user emotions and automatically generate Chinese painting works with a high degree of artistic conception matching.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence image generation, and in particular relates to a method and apparatus for generating large-scale traditional Chinese paintings based on multi-emotion recognition. Background Technology

[0002] In the current era of deep integration between digital technology and traditional culture, traditional Chinese painting, as a core carrier of Chinese traditional art, is facing an urgent need for digital transformation in its dissemination, creation, and experience. On the one hand, traditional Chinese painting creation highly relies on the artist's artistic skills and cultural accumulation, making it difficult for ordinary users to participate. This results in a high barrier to entry for the general public, failing to meet the public's demand for personalized Chinese painting creation and emotional expression. On the other hand, existing digital technologies related to Chinese painting mostly focus on "tool attributes." For example, digital painting software provides brush and paint simulation functions, or image generation technology can only generate static Chinese painting images based on keywords, neglecting the core connection between users' "emotional needs" and the "conveying of artistic conception" in Chinese painting, making it difficult to achieve a deep binding of "emotion-artistic conception-work".

[0003] Meanwhile, with the development of artificial intelligence technologies such as affective computing and knowledge graphs, user interaction scenarios have upgraded from "command-based" to "emotional." In the field of culture and art, users not only want customized artworks but also expect to understand the cultural connotations behind the works, such as information about the painter's life and creative background that matches the emotional mood of the work, in order to form a complete closed loop of experience: "artwork appreciation - cultural understanding - emotional resonance." However, current technology has three major pain points: First, the disconnect between emotional information and the generation of traditional Chinese paintings; existing generation technologies cannot accurately capture the emotional inclination in the user's tone and translate it into the corresponding artistic conception of the painting. Second, the disconnect between cultural knowledge and the artwork; the generated paintings lack supporting background information such as the painter's life and artistic school, reducing the cultural depth of the work. Third, the uniformity of the interactive experience; users can only obtain the final artwork and cannot obtain a personalized interpretation that integrates their own emotions, the characteristics of the artwork, and the cultural background, making it difficult to meet deep emotional and cultural needs.

[0004] Patent document CN121120366A discloses a method for style transfer of Chinese painting based on multi-scale color embedding, including: acquiring a content image dataset and a style image dataset; constructing a Chinese painting style transfer network based on multi-scale color embedding, including: a content-aware position encoder, a hierarchical encoder, and a multimodal decoder; inputting the content image into the content-aware position encoder, and extracting the global semantic feature sequence of the content image through linear mapping and Transformer encoder; inputting the style image into the hierarchical encoder, and combining the global semantic feature sequence to generate an image embedding that integrates content and style through a progressive decoder; inputting the image embedding that integrates content and style into the multimodal decoder to dynamically optimize the color embedding representation.

[0005] Patent document CN117788704A discloses an automatic generation method for composite-dimensional Chinese painting image products, which includes composite-dimensional image generation and modeling, two-dimensional base plate output, and three-dimensional overlay printing steps. Based on the three-dimensional image model established by brushstroke texture samples, the Chinese painting image is generated by image extraction, rendering, and overlay. The three-dimensional part model of the Chinese painting that can be 3D printed by creating or activating three-dimensional textures is generated. The two-dimensional base plate of the Chinese painting image generated in step S1 is output on the base plate by the planar printing component (2). After the three-dimensional part model of the Chinese painting is processed by layer slicing and the printing scheme is planned, a 3D printing program is generated. The three-dimensional overlay printing component (3) performs overlay printing on the base plate with the two-dimensional base plate of the Chinese painting image according to the 3D printing program to generate the three-dimensional part of the Chinese painting, thus obtaining a composite-dimensional Chinese painting image product. Summary of the Invention

[0006] The purpose of this invention is to provide a method and device for generating large-scale traditional Chinese paintings based on multi-emotion recognition. This method can intelligently understand the user's emotions and automatically generate traditional Chinese paintings with highly matching artistic conception.

[0007] To achieve the first objective of this invention, the following technical solution is provided: a large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition, comprising: Obtain user input data and extract tone-related prompts from the input data; Input tone cues into a pre-trained sentiment analysis model to generate corresponding sentiment labels; Conceptual pictures are generated based on the sentiment tag input diffusion model, and semantic retrieval is performed in a pre-built knowledge base of traditional Chinese painting elements to filter out historical painters and their corresponding life events that are most similar to the sentiment tag. Natural language generation technology is used to fuse sentiment tags, image information from concept art, and selected life stories into text to output a final personalized interpretation text. Finally, the generated conceptual images and personalized interpretive texts are visualized as interactive results.

[0008] Specifically, the input data includes voice data or text data.

[0009] Specifically, the input data can be text information directly entered by the user, or text data obtained by converting the user's voice information through speech recognition technology.

[0010] Specifically, the tone-indicating words include one or more of the following: joy, sadness, loneliness, excitement, or calmness.

[0011] Specifically, the knowledge base of traditional Chinese painting elements is constructed using a knowledge graph structure, with emotion tags as the core nodes and painters, life events, and creative styles as associated nodes.

[0012] Specifically, a guiding algorithm is introduced into the diffusion model to generate concept drawings that satisfy the brushstroke characteristics.

[0013] Specifically, the guiding algorithm includes cross-attention or a classifier.

[0014] Specifically, the sentiment analysis model is built on the BERT-LSTM framework and includes a two-dimensional analysis channel for sentiment type and sentiment intensity.

[0015] To achieve the second objective of this invention, the following technical solution is provided: a large-scale Chinese painting generation and interaction device, used to execute the steps of the above-mentioned large-scale Chinese painting generation and interaction method based on multi-emotion recognition.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: Tone data expansion: Add input features such as speech rate variation rate and punctuation density to improve the accuracy of emotion capture; The training data for the traditional Chinese painting generation model is limited to "traditional Chinese painting + sentiment annotation" to ensure compatibility; Cultural information expansion: When searching, the system adds interpretations of the painter's representative works and their emotional connections. Attached Figure Description

[0017] Figure 1 This is a flowchart of the large-scale Chinese painting generation and interaction method based on multi-emotion recognition provided in this embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and illustrated here are usually arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but only represents the selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0019] As Figure 1 shown, a large-scale Chinese painting generation interaction method based on multi-emotion recognition provided by this embodiment can intelligently understand the user's emotions, automatically generate Chinese painting works with highly matching artistic conceptions, and provide personalized interpretations rich in cultural connotations, thus realizing a natural, empathetic, low-threshold, and educational human-computer interaction experience. To achieve the above goals, the sentiment analysis model of the present invention uses a publicly available sentiment dialogue dataset and the self-built "Chinese Painting Artistic Conception Sentiment Association Corpus" for supervised fine-tuning. The diffusion model uses the text-to-image model (Stable Diffusion V1.5) as the base, and through nearly 100,000 high-definition scanned traditional Chinese paintings collected, combined with the detailed brushwork prompts generated by the multi-modal pre-training model (BLIP-2), the model is fine-tuned using the LoRA (Low-Rank Adaptation) technique to ensure that the generated images do not have a modern photo sense but have the ink characteristics of traditional rubbing, dabbing, and dotting.

[0020] The specific steps are as follows: Step 1: Receive the tone data input by the user; Specific implementation: If the user selects voice input, the system calls the device microphone to collect audio data and immediately performs noise reduction through the integrated automatic speech recognition (ASR) engine (such as the open-source engine ESPnet or commercial API), and converts the audio stream into text data in real time. This process focuses on the accurate conversion of semantic content.

[0021] If the user directly inputs text, the automatic speech recognition (ASR) conversion step is skipped. Receive the text content input by the user (such as "With a faint longing, I want a picture of the moon and a returning boat"), and perform word segmentation and stop word filtering (such as emotionless words like "de" and "xiang").

[0022] Output: A plain text string (e.g., "I feel so lonely today").

[0023] Data storage: The preprocessed tone data is temporarily stored in a "temporary data pool" and associated with a unique user identifier (such as device ID) to ensure traceability of subsequent steps.

[0024] More specifically, whether it's text input or speech data converted through Automatic Speech Recognition (ASR), it will ultimately be converted into a standard text format (String type). For example, the input "I am happy today" will be converted into the text "I am happy today" through speech recognition technology, and this text is the standardized data type.

[0025] Step 2: Perform sentiment analysis on the tone data to obtain sentiment labels; The sentiment analysis module receives the text output from the previous step and uses a hybrid analysis scheme of "pre-trained natural language processing model (NLP) + Chinese painting sentiment dictionary" to solve the adaptation problem of "matching general sentiment analysis with the artistic conception of Chinese painting". Model layer: Based on the sentiment classification model (BERT-LSTM model), the tone data is analyzed from two dimensions: "sentiment type + sentiment intensity" (e.g., "tranquility - moderate", "joy - strong", "longing - weak"). Dictionary layer: Construct a dedicated "Chinese painting emotion mapping dictionary" to convert general emotion tags into Chinese painting-adapted tags (such as "longing" -> "moonlit night, returning boat, withered lotus" and other Chinese painting imagery-related tags); Output: Generate structured sentiment labels and their confidence scores (Example: {sentiment type: tranquility, 0.95, intensity: moderate, mood-related elements: distant mountains, lake, solitary pavilion}).

[0026] In this embodiment, a custom Chinese painting emotion mapping dictionary is established. This dictionary is mainly used to map user emotion analysis results (including emotion type and emotion intensity) onto Chinese painting creation elements to generate Chinese painting images that conform to the user's emotional mood. The dictionary contains a large number of elements related to traditional Chinese painting creation (such as composition, brushwork, color, objects, etc.) and corresponding emotional expression methods.

[0027] This emotion mapping dictionary includes mappings between emotion types and elements in traditional Chinese painting, and mappings between emotion intensity and visual expression.

[0028] The dictionary maps different types of emotions to elements in traditional Chinese painting: each type of emotion (such as joy, loneliness, sadness, etc.) is associated with specific elements in traditional Chinese painting (such as "landscape", "figures", "flowers and birds") and their expressive techniques (such as "distant mountains" and "texturing strokes"). Through this mapping, the dictionary achieves a precise correspondence between emotions and visual elements.

[0029] The mapping between emotional intensity and visual expression: The combination of emotional intensity (e.g., strong, medium, weak) and the visual expression characteristics of traditional Chinese painting (e.g., "ink density," "white space ratio," "color saturation") forms a mapping relationship between emotional intensity and artistic expression. For example, strong emotional intensity may correspond to bold ink and color, large areas of white space, and vibrant colors; medium emotional intensity may correspond to gentle color tones and balanced composition; and weak emotional intensity may correspond to light ink, delicate brushstrokes, and soft color tones.

[0030] To better illustrate the mapping method described above, the following example dictionary mapping entries are provided: Type of emotion: Joy; Traditional Chinese painting elements: "flowers, vibrant colors, and bold brushstrokes"; Brushwork: "A fast and fluid dotting technique"; Color: "High saturation, vibrant hues, such as bright red and emerald green"; Imagery: "A vibrant, sun-drenched natural scene."

[0031] Type of emotion: Loneliness; Traditional Chinese painting elements: "a solitary pavilion, distant mountains, and a desolate water surface"; Brushwork: "Sparse and light texturing techniques, delicate and fragmented brushstrokes"; Color: "Cool gray tones and diluted ink colors convey a sense of desolation"; Artistic conception: "The tranquil lake and the vast mountains and rivers create a sense of loneliness."

[0032] Emotional intensity: Strong; Visual presentation: "intense colors, heavy brushstrokes, and large areas of white space"; Composition: "An asymmetrical or highly dynamic layout that expresses intense emotions."

[0033] Emotional intensity: Moderate; Visual presentation: "Gentle color palette, balanced composition"; Composition: "Appropriate use of white space and delicate brushstrokes create both static beauty and dynamism." Emotional intensity: weak.

[0034] Visual presentation: "Light ink, delicate brushstrokes, soft tones"; Composition: "A simple layout and delicate natural scenery reflect a restrained and gentle emotion."

[0035] Step 3: Generate a traditional Chinese painting image that matches the emotional mood based on the emotional tag; Generation logic: Achieve precise mapping of "emotional tags -> traditional Chinese painting elements -> image generation" to avoid the "element stacking" problem of existing technologies; Step 1: Emotion-Element Mapping. The "Emotion-Traditional Chinese Painting Element Knowledge Base" is invoked to transform abstract emotion tags into control parameters that the model can understand. Core elements are matched based on the emotion tags (e.g., "Tranquility - Moderate" -> Composition: 60% white space, Color: bluish-gray + off-white, Subject: Distant mountains (light ink), lake (light blue), solitary pavilion (ochre), Brushstrokes: fine, fragmented texturing; Imagery: lone boat, cold forest, cold moon). The second step is to use a "Chinese painting style fine-tuning diffusion model (such as a text-to-image model (StableDiffusion V1.5 fine-tuning version)", input the matched element parameters, and generate an image that conforms to the brushwork of traditional Chinese painting (such as cunfa and dotting), rather than a photo-like image; Quality Control: A built-in "Traditional Chinese Painting Style Verification Module" strongly guides the generation process during the denoising of the diffusion model through mechanisms such as cross-attention or classifier-free guidance, ensuring that the final output image visually reflects the target emotion. If the generated image does not conform to brushstroke characteristics (e.g., pixelated edges appear), parameters are automatically readjusted for regeneration.

[0036] Step 4: Based on sentiment tags, retrieve relevant biographical information about the painters from the knowledge graph; Knowledge Graph Design: Construct a "Traditional Chinese Painting Culture Knowledge Graph", with core nodes including "emotional tags", "painters", "life events", and "creative styles", and examples of relationships between nodes (using graph databases such as Neo4j).

[0037] “Tranquility” Emotional Tag -> Associated Painter -> Biography (“She lived in seclusion in Huangshan and was skilled at painting light ink landscapes”) -> Creative Style (“Minimalist composition, lots of blank space, reflecting a tranquil artistic conception”); Search logic: The algorithm uses "sentiment tag priority matching" to calculate semantic similarity, instead of the existing "element matching" technology. Input an emotional tag (e.g., "joy") -> Search the knowledge graph for painters whose creative style is strongly correlated with the mood of joy (e.g., Xu Wei, whose ink grape paintings are brightly colored and embody a cheerful and open-minded joy) -> Extract key biographical information (e.g., "expressing a positive state of mind through painting in his later years"). When searching for "loneliness," the system will search for painter nodes whose life experiences include events such as "unfulfilled talent," "seclusion," and "a desolate old age" (e.g., Xu Wei, Bada Shanren), and return the most relevant fragments of their life stories.

[0038] Step 5: Generate personalized interpretation text that integrates information from multiple sources; Integration Logic: Using "emotional tags" as the core link, connecting "characteristics of traditional Chinese painting images" with "the painter's life" to avoid information fragmentation; Example of text structure: "Your tone reflects a 'moderate tranquility' emotion, therefore this painting, 'Distant Mountains in Light Ink,' was generated---the painting uses 60% white space (corresponding to a sense of tranquility and openness), and the distant mountains are painted with light ink washes (to weaken the visual impact); this is consistent with the style of a certain painter from Era A, who often used minimalist landscapes to express his inner peace during his seclusion in Huangshan. His life experience of 'not serving in office and living in seclusion' also resonates with your current state of mind." Generation technology: Based on the Prompt Tuning text generation model, inputting "emotional tags + image element parameters + painter's biographical fragments" will automatically generate coherent and anthropomorphic interpretation text.

[0039] Technical Description: The interpretation and generation module receives all the outputs from the first three steps (emotional tags, generated image descriptions, and painter's biography) and integrates them into a coherent and natural text interpretation.

[0040] More specifically, generating personalized interpretive text requires combining multiple input sources (such as sentiment tags, traditional Chinese painting elements, and biographical information about the painter) to enable the model to generate highly personalized text that meets emotional needs. To ensure the generated text possesses artistry, cultural depth, and emotional resonance, the model requires certain fine-tuning and training.

[0041] Large language models (such as the BERT framework) that are pre-trained typically need to be fine-tuned to adapt to specific domains such as Chinese painting interpretation and sentiment analysis. The fine-tuning process uses a specialized dataset containing sentiment labels, artistic styles, and painter background information to help the model understand how to generate text related to specific sentiments and artistic styles.

[0042] To enable the model to understand the association between sentiment tags (such as "loneliness" and "joy") and elements of traditional Chinese painting (such as "distant mountains" and "lonely pavilion"), specific task optimizations are needed during the model's training process. By injecting this information into the training set, the model will learn how to integrate various types of data sources (such as sentiment analysis results, artistic elements, and painter biographies) into the interpreted text.

[0043] During the fine-tuning process, you will use text datasets with emotional tags (such as "loneliness", "joy", etc.) and elements of traditional Chinese painting (such as "lonely pavilion" and "distant mountains"), as well as biographical information of the painters associated with them; The goal of training is to enable the model to combine information such as sentiment tags, traditional Chinese painting elements, and historical background to generate personalized text with cultural connotations and emotional resonance. The training process typically includes annotating the text content, building a dataset, and then fine-tuning it through supervised learning so that the model can automatically associate different types of input information when generating text. The fine-tuning process uses techniques similar to prompt engineering to guide the model on how to combine multiple inputs to generate specific interpretive text. This approach helps the model generate text that adapts to emotional and cultural requirements by designing appropriate input prompts (such as "generate interpretive text based on sentiment tags and artistic styles").

[0044] Step Six: Output the traditional Chinese painting image and personalized interpretation text to the user; Output format: The system presents the generated traditional Chinese painting image and interpretation text to the user in an aesthetically pleasing layout, supporting simultaneous display on multiple terminals.

[0045] Complete an interactive experience that includes visual art achievements and in-depth cultural interpretation.

[0046] Visually: The traditional Chinese painting image is displayed in high-definition format (supports zooming to view details, such as brushstrokes), and the interpretive text accompanies the image in the form of a "floating window" (click to expand the full text); Voice input (optional): If the user's initial input is voice, the text can be read aloud and interpreted simultaneously (suitable for visually impaired users or multiple scenario needs).

[0047] This embodiment also provides a large-scale Chinese painting generation interactive device, which is used to perform the steps of the large-scale Chinese painting generation interactive method based on multi-emotion recognition provided in the above embodiment.

[0048] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0049] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0050] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition, characterized in that, include: Obtain user input data and extract tone-related prompts from the input data; Input tone cues into a pre-trained sentiment analysis model to generate corresponding sentiment labels; Conceptual pictures are generated based on the sentiment tag input diffusion model, and semantic retrieval is performed in a pre-built knowledge base of traditional Chinese painting elements to filter out historical painters and their corresponding life events that are most similar to the sentiment tag. Natural language generation technology is used to fuse emotional tags, image information from concept art, and selected life stories into text to output a final personalized interpretation text. Finally, the generated conceptual images and personalized interpretive texts are visualized as interactive results.

2. The large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition according to claim 1, characterized in that, The input data includes voice data or text data.

3. The large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition according to claim 1, characterized in that, The mood markers include one or more of the following: joy, sadness, loneliness, excitement, or calmness.

4. The large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition according to claim 1, characterized in that, The knowledge base of traditional Chinese painting elements is constructed using a knowledge graph structure. The knowledge graph structure uses emotional tags as core nodes and sets painters, life events, and creative styles as related nodes.

5. The large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition according to claim 1, characterized in that, A guiding algorithm is introduced into the diffusion model to generate concept drawings that satisfy the brushstroke characteristics.

6. The large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition according to claim 5, characterized in that, The guidance algorithm includes cross-attention or a classifier.

7. The large-scale interactive method for generating traditional Chinese paintings based on multi-emotion recognition according to claim 1, characterized in that, The sentiment analysis model is built on the BERT-LSTM framework and includes a two-dimensional analysis channel for sentiment type and sentiment intensity.

8. A large-scale interactive device for generating traditional Chinese paintings, characterized in that, The steps are for performing the large-scale Chinese painting generation and interaction method based on multi-emotion recognition as described in any one of claims 1 to 7.

Images