Interactive method and system for generating AI paintings based on real-time user input

Abstract

The present invention discloses an interactive method and system for generating AI paintings based on real-time user input, and relates to the technical field of AI painting. The present invention includes: real-time collection of user input information; pre-processing of the collected raw data; converting the user's input coordinates into line graphics; real-time collection of line information and transmission to AI for processing; processing of the generated results. The present invention greatly improves the user's interactive experience by automatically generating AI paintings based on user gestures, lines and other information. By integrating multiple pre-trained models, computer vision technology and efficient data processing methods, the system realizes efficient, extremely simple and user-friendly AI painting interaction.

Description

Technical Field

[0001] The present invention belongs to the field of AI painting technology, and in particular to an interactive method and system for generating AI paintings based on real-time user input. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of deep learning in the fields of computer vision and image generation is becoming more and more extensive. As an important direction, AI painting has shown great potential in the fields of artistic creation and design assistance.

[0003] However, traditional AI painting systems have high latency. The images or lines input by the user need to be uploaded and wait for several seconds for AI to complete the drawing. It cannot provide real-time feedback, and the operation is complicated. The user needs to set multiple parameters, such as prompt words, style, etc., which increases the complexity of use and makes the user experience poor. It is difficult for ordinary users to quickly participate in AI painting interactions, and the system cannot capture the user's creative inspiration in time.

[0004] To this end, we provide an interactive method and system for generating AI paintings based on real-time user input to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide an interactive method and system for generating AI paintings based on real-time user input, which solves the problems of high latency, complex operation and poor user experience in existing AI paintings by automatically generating AI paintings according to user gestures, lines and other information, and by integrating multiple pre-trained models, computer vision technology and efficient data processing methods.

[0006] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0007] The present invention is an interactive method for generating AI paintings based on real-time user input, comprising:

[0008] S1. Collect user input information in real time;

[0009] S2, preprocessing the collected raw data;

[0010] S3, converting the user's input coordinates into a line graphic;

[0011] S4, collect line information in real time and transmit it to AI for processing;

[0012] S5. Process the generated results.

[0013] The present invention is further configured to collect user input information in real time, including screen drawing or three-dimensional gesture input. Screen drawing includes user input of information through touch devices such as touch screen digitizers and mobile phones. Three-dimensional gesture input includes identifying the user's hand position through common somatosensory devices such as Kinect or Leap Motion and cameras, and converting it into three-dimensional coordinate data in real time.

[0014] The present invention is further configured such that the raw data is preprocessed by mapping the data to a uniform coordinate space using a normalization method and removing noise using a filtering algorithm (such as a Kalman filter);

[0015] The raw data preprocessing also includes converting the data into high-dimensional feature vectors.

[0016] The present invention is further configured such that converting the user's input coordinates into a linear graphic comprises:

[0017] T1. Use Bezier curve or spline curve to fit the user's input coordinates;

[0018] T2. Use interpolation method and Gaussian filtering to smooth the fitted curve.

[0019] The present invention is further configured to collect line information in real time and transmit it to AI for processing, including:

[0020] S6. Transmitting data through a network;

[0021] S7, compressing, encoding, mapping and converting the transmitted data;

[0022] S8. Generate preliminary results through AI models.

[0023] The present invention is further configured that the processing of the generated results includes performing detail enhancement on the preliminary results and performing high-definition restoration and amplification on the preliminary results;

[0024] The details of the preliminary results are enhanced by using the perceptual loss function to train the detail enhancement model to improve the texture and detail performance of the image;

[0025] High-definition restoration and upscaling of preliminary results include using generative adversarial network models such as SRGAN to upscale images to higher resolutions while maintaining clarity.

[0026] At the same time, the present invention also provides the following technical solution: an interactive system for generating AI paintings based on real-time user input, comprising:

[0027] Input acquisition module, used to collect user input information in real time, including screen lines, 3D coordinates (through somatosensory devices such as Kinect or Leap Motion, cameras), and image information;

[0028] The data processing module is used to extract and convert the collected input information into an input format that can be recognized by AI;

[0029] AI painting generation module, which is used to generate corresponding painting results in real time based on pre-trained AI models (including convolutional neural networks, generative adversarial networks, etc.);

[0030] The output display module is used to feed back the generated painting results to the user with extremely low latency, so that the user can observe the generated painting process in real time.

[0031] The present invention is further configured that the AI ​​painting generation module includes: a data transmission optimization module, a data compression and encoding module, a pre-trained AI model, an integrated visual recognition module, and a parameter, prompt word and model database;

[0032] The data transmission optimization module uses asynchronous threads to transmit data and supports transmission protocols such as OSC, WebSocket, and TCP.

[0033] The data compression and encoding module is used to compress, encode, and map the transmitted data;

[0034] Pre-trained AI models, including the Checkpoint model for generating the basis of a painting, the LoRA model for controlling the style, and the ControlNet model for precisely controlling the content of a painting;

[0035] The Checkpoint model is used to generate basic painting content;

[0036] The LoRA model is used to control the style output based on the low-rank matrix fine-tuning technique;

[0037] The ControlNet model is used to ensure the accuracy of user input;

[0038] The integrated visual recognition module includes a computer vision analysis system and an intention inference and prompt word generation system;

[0039] The computer vision analysis system is used to extract features and classify images input by users;

[0040] The intention inference and prompt word generation system infers the user's drawing intention by analyzing the high-level semantic features of the image, and automatically generates corresponding prompt words using a natural language generation model (such as Qwen2, etc.);

[0041] The parameter, prompt word and model database includes a parameter database, a prompt word database and a model database;

[0042] The parameter database is used to store the model's hyperparameters and operating parameters, such as learning rate, weight initialization method, etc.

[0043] The prompt word database contains rich descriptive text of painting styles and themes for the system to refer to when generating images;

[0044] The model database is used to save pre-trained models of different types and styles, and supports dynamic loading and switching.

[0045] The present invention is further configured that the output display module includes a real-time display module and a picture processing module;

[0046] The real-time display module includes random seed fixing and anti-shake mechanism;

[0047] The random seed fixed formula is expressed as: Output = AI_Model (Input, S);

[0048] The anti-shake mechanism uses technologies such as frame rate control and image caching;

[0049] The image processing module includes a detail enhancement model and a high-definition restoration and magnification model;

[0050] The detail enhancement model uses the perceptual loss function to train the detail enhancement model;

[0051] The high-definition restoration and enlargement model uses generative adversarial network models such as SRGAN to enlarge images to higher resolutions while maintaining clarity.

[0052] The present invention is further configured that the real-time saving module includes an operation record saving module and an undo algorithm;

[0053] The operation record saving module uses a stack or linked list data structure to save the user's operation sequence and an array to save the user's drawing input. Each operation includes input data and generated results.

[0054] The undo algorithm is that when the user performs an undo operation, the most recent operation is popped up from the operation record, and even real-time drawing can be restored to the previous state.

[0055] The present invention has the following beneficial effects:

[0056] 1. Real-time: The system can respond to user input within milliseconds to achieve true real-time interaction.

[0057] 2. Multimodal input: supports two-dimensional screen drawing and three-dimensional air gesture input, enhancing the flexibility and applicability of the system.

[0058] 3. Intelligence: By integrating computer vision and deep learning technologies, the system can automatically infer the user's painting intention without the user having to enter complex prompts and parameters.

[0059] 4. High-quality output: Using a variety of pre-trained models and high-definition restoration technology, the generated painting results are highly artistic and detailed.

[0060] 5. User-friendly: A simple and intuitive interaction method is designed to lower the user's usage threshold and improve the user experience.

[0061] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0063] Figure 1 Schematic diagram of an interactive method for generating AI paintings based on real-time user input.

[0064] Figure 2 Schematic diagram of an interactive system that generates AI paintings based on real-time user input.

[0065] Figure 3 Schematic diagram of the AI ​​painting generation module.

[0066] Figure 4 A schematic diagram of the output display module. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Example 1

[0069] See also Figure 1 The present invention is an interactive method for generating AI paintings based on real-time user input, comprising:

[0070] S1. Collect user input information in real time;

[0071] S2, preprocessing the collected raw data;

[0072] S3, converting the user's input coordinates into a line graphic;

[0073] S4, collect line information in real time and transmit it to AI for processing;

[0074] S5. Process the generated results.

[0075] Specifically, real-time collection of user input information includes screen drawing or 3D gesture input. Screen drawing includes user input through touch devices such as touch screen tablets and mobile phones. 3D gesture input includes identifying the user's hand position through common somatosensory devices such as Kinect or LeapMotion and cameras, and converting it into 3D coordinate data in real time.

[0076] The raw data are preprocessed by mapping the data into a uniform coordinate space using a normalization method and removing noise using a filtering algorithm (e.g., Kalman filtering);

[0077] Preprocessing the raw data also includes converting the data into a high-dimensional feature vector. For example, for a two-dimensional coordinate $(x,y)$, a feature vector $\mathbf{v}=[x,y,x^2,y^2,xy]$ can be constructed;

[0078] Converting the user's input coordinates into a line graphic includes:

[0079] T1. Use Bezier curve or spline curve to fit the user's input coordinates:

[0080] T2, use interpolation method and Gaussian filtering to smooth the fitted curve;

[0081] Real-time collection of line information and transmission to AI for processing includes:

[0082] S6. Transmitting data through a network;

[0083] S7, compressing, encoding, mapping and converting the transmitted data;

[0084] S8. Generate preliminary results through AI models;

[0085] Processing the generated results includes enhancing the details of the preliminary results and performing high-definition restoration and magnification on the preliminary results.

[0086] Furthermore, the preliminary results are enhanced in detail, including using the perceptual loss function to train the detail enhancement model to improve the texture and detail performance of the image;

[0087] High-definition restoration and upscaling of preliminary results include using generative adversarial network models such as SRGAN to upscale images to higher resolutions while maintaining clarity.

[0088] The operation process of this embodiment is as follows:

[0089] 1. Usage scenarios based on linear graphics:

[0090] When users draw graffiti lines on the screen, AI will automatically collect the user's line features in real time, and based on the features, guess the user's drawing intention, generate the drawing results in real time, and the generated image contour and shape features are highly consistent with the line features drawn by the user.

[0091] The user draws graffiti lines on paper, and AI uses the camera acquisition device to identify and process the user's real-time painting process, and generates the same result as the previous one.

[0092] 2. Usage scenarios based on coordinates:

[0093] When the user waves his arms in the air, the xyz three-dimensional coordinate information of the user's hand in the air is collected through cameras, leapmotion, Kinect and other somatosensory devices or visual recognition devices. The system will convert the coordinate information into graphic information in real time, allowing AI to draw the image content in real time. The user can see the real-time painting content on the screen in sync with his own movements.

[0094] 3. Usage scenarios based on real-time graphics:

[0095] Based on the real-time transmission of the camera's images, AI generates sequence frame images in real time based on the collected image features, with extremely low latency, and no sense of delay when viewed with the naked eye.

[0096] Based on images transmitted by the computer's built-in program, AI can generate new image content in real time based on the characteristics of these images.

[0097] Example 2

[0098] See also Figure 2-4 , an embodiment of the present invention provides an interactive system for generating AI paintings based on real-time user input, including:

[0099] The input acquisition module is used to collect user input information in real time, including screen lines, 3D coordinates (through somatosensory devices such as Kinect or Leap Motion, cameras), and image information. Through two-dimensional screen drawing and three-dimensional air gesture input, the flexibility and applicability of the system are enhanced;

[0100] The data processing module is used to extract and convert the collected input information into an input format that can be recognized by AI;

[0101] AI painting generation module, which is used to generate corresponding painting results in real time based on pre-trained AI models (including convolutional neural networks, generative adversarial networks, etc.);

[0102] The output display module is used to feed back the generated painting results to the user with extremely low latency, so that the user can observe the generated painting process in real time.

[0103] Specifically, the AI ​​painting generation module includes: data transmission optimization module, data compression and encoding module, pre-trained AI model, integrated visual recognition module, and parameter, prompt word and model database;

[0104] The data transmission optimization module uses asynchronous threads to transmit data and supports transmission protocols such as OSC, WebSocket, and TCP. Asynchronous I / O allows the system to continue to execute other tasks while waiting for data transmission to avoid blocking;

[0105] The data compression and encoding module is used to compress, encode, and map the transmitted data, thereby reducing the transmission volume, improving transmission efficiency, and reducing response time;

[0106] Pre-trained AI models, including the Checkpoint model for generating the basis of a painting, the LoRA model for controlling the style, and the ControlNet model for precisely controlling the content of a painting;

[0107] The Checkpoint model is used to generate basic painting content;

[0108] The LoRA model is used to control the style output based on the low-rank matrix fine-tuning technique;

[0109] The ControlNet model is used to ensure the accuracy of user input;

[0110] The integrated visual recognition module includes a computer vision analysis system and an intention inference and prompt word generation system. By integrating computer vision and deep learning technology, the system can automatically infer the user's painting intention without the user having to enter complex prompt words and parameters;

[0111] The computer vision analysis system is used to extract features and classify images input by users;

[0112] The intention inference and prompt word generation system infers the user's drawing intention by analyzing the high-level semantic features of the image, and automatically generates corresponding prompt words using a natural language generation model (such as Qwen2, etc.);

[0113] The parameter, prompt word and model database includes a parameter database, a prompt word database and a model database;

[0114] The parameter database is used to store the model's hyperparameters and operating parameters, such as learning rate, weight initialization method, etc.

[0115] The prompt word database contains rich descriptive text of painting styles and themes for the system to refer to when generating images;

[0116] The model database is used to store pre-trained models of different types and styles, and supports dynamic loading and switching;

[0117] The output display module includes a real-time display module and a picture processing module;

[0118] The real-time display module includes random seed fixing and anti-shake mechanism;

[0119] The random seed fixed formula is expressed as: Output = AI_Model (Input, S);

[0120] The anti-shake mechanism uses technologies such as frame rate control and image caching;

[0121] The image processing module includes a detail enhancement model and a high-definition restoration and magnification model;

[0122] The detail enhancement model uses the perceptual loss function to train the detail enhancement model, which can improve the texture and detail performance of the image. It uses a variety of pre-trained models and high-definition restoration technology to generate paintings with high artistry and detail expression.

[0123] The high-definition restoration and enlargement model uses generative adversarial network models such as SRGAN to enlarge images to higher resolutions while maintaining clarity.

[0124] The operation process of this embodiment is as follows: when the user needs to generate an AI painting, the user's input information is first collected through the input acquisition module, and then the collected input information is feature extracted and converted through the data processing module to generate an input format recognizable by AI. Then, the AI ​​painting generation module is based on a pre-trained AI model (including convolutional neural networks, generative adversarial networks, etc.) to generate the corresponding painting results in real time. Finally, the generated results are processed through the output display module, and then the generated painting results are fed back to the user with extremely low latency, so that the user can observe the generated painting process in real time.

[0125] Example 3

[0126] See also Figure 2 , based on the specific embodiment 1 and the specific embodiment 2, it also includes a real-time saving module, characterized in that the real-time saving module includes an operation record saving module and a revocation algorithm;

[0127] The operation record saving module uses a stack or linked list data structure to save the user's operation sequence and an array to save the user's drawing input. Each operation includes input data and generated results.

[0128] The undo algorithm is that when the user performs an undo operation, the most recent operation is popped up from the operation record, and even real-time drawing can be restored to the previous state.

[0129] The operation process of this embodiment is as follows: when the user needs to stop drawing, the real-time saving module can save the user's current operation and restore it to the previous state, thereby improving the flexibility of the system.

[0130] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0131] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An interactive method for generating AI paintings based on real-time user input, characterized in that: include: S1. Collect user input information in real time; S2, preprocessing the collected raw data; S3, converting the user's input coordinates into a line graphic; S4, collect line information in real time and transmit it to AI for processing; S5. Process the generated results.

2. The interactive method for generating AI paintings based on real-time user input according to claim 1, characterized in that: The real-time collection of user input information includes screen drawing or three-dimensional gesture input. The screen drawing includes the user inputting information through a touch screen, a digital tablet, a mobile phone or other touch-sensitive device. The three-dimensional gesture input includes identifying the user's hand position through common somatosensory devices such as Kinect or Leap Motion and a camera, and converting it into three-dimensional coordinate data in real time.

3. The interactive method for generating AI paintings based on real-time user input according to claim 1, characterized in that: The raw data is preprocessed by using a normalization method to map the data to a unified coordinate space and using a filtering algorithm (such as Kalman filtering) to remove noise; The raw data preprocessing also includes converting the data into a high-dimensional feature vector.

4. The interactive method for generating AI paintings based on real-time user input according to claim 1, characterized in that: The step of converting the user's input coordinates into a line graphic comprises: T1. Use Bezier curve or spline curve to fit the user's input coordinates: T2. Use interpolation method and Gaussian filtering to smooth the fitted curve.

5. The interactive method for generating AI paintings based on real-time user input according to claim 1, characterized in that: The real-time collection of line information and transmission to AI for processing includes: S6, transmitting data through the network; S7, compressing, encoding, mapping and converting the transmitted data; S8. Generate preliminary results through AI models.

6. The interactive method for generating AI paintings based on real-time user input according to claim 1, characterized in that: The processing of the generated results includes enhancing the details of the preliminary results and performing high-definition restoration and magnification on the preliminary results; The detail enhancement of the preliminary results includes training a detail enhancement model using a perceptual loss function to improve the texture and detail performance of the image; The high-definition restoration and enlargement of the preliminary results includes the use of generative adversarial network models such as SRGAN to enlarge the image to a higher resolution while maintaining clarity.

7. An interactive system for generating AI paintings based on real-time user input, characterized in that: The system of the interactive method for generating AI painting based on real-time user input according to any one of claims 1 to 6 comprises: Input acquisition module, used to collect user input information in real time, including screen lines, 3D coordinates (through somatosensory devices such as Kinect or Leap Motion, cameras), and image information; The data processing module is used to extract and convert the collected input information into an input format that can be recognized by AI; AI painting generation module, which is used to generate corresponding painting results in real time based on pre-trained AI models (including convolutional neural networks, generative adversarial networks, etc.); The output display module is used to feed back the generated painting results to the user with extremely low latency, so that the user can observe the generated painting process in real time.

8. The interactive system for generating AI paintings based on real-time user input according to claim 7, characterized in that: The AI ​​painting generation module includes: a data transmission optimization module, a data compression and encoding module, a pre-trained AI model, an integrated visual recognition module, and a parameter, prompt word and model database; The data transmission optimization module uses asynchronous threading to transmit data and supports transmission protocols such as OSC, WebSocket, and TCP; The data compression and encoding module is used to compress, encode, and map the transmitted data; The pre-trained AI model includes a checkpoint model for generating a painting basis, a LoRA model for controlling the style, and a ControlNet model for accurately controlling the painting content; The Checkpoint model is used to generate basic painting content; The LoRA model is used to control the style output based on the low-rank matrix fine-tuning technique; The ControlNet model is used to ensure the accuracy of user input; The integrated visual recognition module includes a computer vision analysis system and an intention inference and prompt word generation system; The computer vision analysis system is used to extract features and classify images input by users; The intention inference and prompt word generation system infers the user's drawing intention by analyzing the high-level semantic features of the image, and automatically generates corresponding prompt words using a natural language generation model (such as Qwen2, etc.); The parameter, prompt word and model database includes a parameter database, a prompt word database and a model database; The parameter database is used to store the hyperparameters and operating parameters of the model, such as learning rate, weight initialization method, etc.; The prompt word database contains rich descriptive texts of painting styles and themes for the system to refer to when generating images; The model database is used to store pre-trained models of different types and styles, and supports dynamic loading and switching.

9. The interactive system for generating AI paintings based on real-time user input according to claim 7, characterized in that: The output display module includes a real-time display module and a picture processing module; The real-time display module includes a random seed fixation and anti-shake mechanism; The random seed fixed formula is expressed as: Output = AI_Model (Input, S); The anti-shake mechanism adopts frame rate control and image caching technologies; The image processing module includes a detail enhancement model and a high-definition restoration and magnification model; The detail enhancement model uses a perceptual loss function to train the detail enhancement model; The high-definition restoration and magnification model uses a generative adversarial network model such as SRGAN to enlarge the image to a higher resolution while maintaining clarity.

10. The interactive system for generating AI paintings based on real-time user input according to claim 7 further comprises a real-time saving module, characterized in that: The real-time saving module includes an operation record saving module and a revocation algorithm; The operation record storage module uses a stack or linked list data structure to store the user's operation sequence and an array to store the user's drawing input. Each operation includes input data and generated results. The undo algorithm is that when the user performs an undo operation, the most recent operation is popped out from the operation record, and even real-time drawing can be restored to the previous state.

Images