Man-machine interaction method, system and equipment based on asynchronous art creation and medium

Abstract

The invention provides a man-machine interaction method, system and device based on asynchronous art creation and a medium, and particularly relates to the technical field of man-machine interaction. The scheme comprises the steps that original art works and multi-modal session information input by a visitor according to a current interaction task are collected in real time; performing dynamic analysis processing on the original artistic works and the multi-modal session information by utilizing a plurality of artificial intelligence models, and generating man-machine co-created artistic works and session abstracts; and feeding back the man-machine co-creation artistic work and the session abstract to the server, responding to a received next interaction task generated by the server based on the session abstract and the man-machine co-creation artistic work under the assistance of artificial intelligence, and feeding back the next interaction task to the visitor. According to the scheme, the difficulty of artistic creation of visitors can be greatly reduced, and the visitors can be guided to carry out deep self-provincial inversion, so that the effectiveness of interaction is improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method, system, device and medium based on asynchronous artistic creation. Background Art

[0002] Art therapy is a multimodal treatment method that combines creative art activities with verbal expression to help people release emotions and improve mental health. In this therapy, service providers and clients collaborate through interactive conversations and interactive tasks during art therapy sessions.

[0003] In the prior art, service providers recommend interactive tasks that combine artistic creation with verbal expression. For example, service providers encourage visitors to complete home creations every day, such as observing, drawing, taking pictures, and writing about the sky; artistic creation is combined with verbal exercises in interactive tasks to help visitors improve their daily coping abilities. This method not only allows visitors to explore their inner thoughts in depth and find ways to release their emotions through words, but also brings a unique healing experience with the creativity and expressiveness of artistic creation. However, the lack of guidance from service providers may increase the psychological threshold of visitors' artistic creation, and the differences in individual understanding and expression of emotions make it more challenging to express past emotions and experiences in words. At the same time, it is difficult for service providers to track the treatment situation, and it is difficult for service providers to understand the specific situation of the visitor's creative process. Therefore, ensuring the therapeutic effect of artistic creation is a problem that needs to be solved.

[0004] It can be seen that art therapy in the existing technology has problems such as difficulty in effective implementation, poor results and difficulty in tracking due to the high psychological threshold of artistic creation. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a human-computer interaction method, system, device and medium based on asynchronous art creation, aiming to solve the problems in the prior art of art therapy that are difficult to effectively execute, have poor effects and are difficult to track due to the high psychological threshold of art creation.

[0006] In order to achieve the above-mentioned object, the first aspect of the present invention provides a human-computer interaction method based on asynchronous art creation, comprising: Real-time collection of original artwork and multimodal conversation information input by visitors according to the current interactive task; Using a combination of artificial intelligence models to dynamically analyze and process the original artwork and multimodal conversation information, respectively, to generate human-machine co-created artwork and conversation summaries; Feeding back the human-machine co-created artwork and the conversation summary to the server, in response to receiving a next interactive task generated by the server based on the conversation summary and the human-machine co-created artwork with the assistance of artificial intelligence, and feeding back the next interactive task to the visitor.

[0007] Optionally, the real-time collection of original artwork and multimodal conversation information input by the visitor according to the current interactive task includes: Constructing a series of interactive questions based on the dialogue principles and example questions corresponding to the current interactive task; The interactive questions are fed back to the visitor in sequence, and the original artworks and multimodal conversation information corresponding to each of the interactive questions fed back by the visitor are collected in sequence.

[0008] Optionally, the using of the artificial intelligence model combination to dynamically analyze and process the original artwork and the multimodal conversation information respectively to generate the human-machine co-created artwork and conversation summary includes: Analyzing information other than artistic creation information in the multimodal conversation information using the first module in the artificial intelligence model combination to generate several types of conversation instructions; Feeding back the conversation instruction to the visitor, and generating several types of conversation information in response to receiving the visitor's feedback information on the conversation instruction; All the session information is integrated to generate a session summary.

[0009] Optionally, the first module in the artificial intelligence model combination is used to analyze information other than artistic creation information in the multimodal conversation information to generate several types of conversation instructions, including: Performing standardization processing and / or data conversion on the information other than the artistic creation information in the multimodal conversation information to obtain data in a target format; The first module in the artificial intelligence model combination is used to analyze the data in the target format to generate several types of conversation instructions.

[0010] Optionally, the using of the artificial intelligence model combination to dynamically analyze and process the original artwork and the multimodal conversation information respectively to generate the human-machine co-created artwork and conversation summary includes: Analyzing the art creation information in the multimodal conversation information using the second module in the artificial intelligence model combination to generate art creation instructions and an art creation summary; The art creation instruction is fed back to the visitor, and the third module in the artificial intelligence model combination is used to dynamically analyze and process the art creation information fed back by the visitor according to the art creation instruction and the art creation summary to generate a human-machine co-created art work.

[0011] Optionally, the use of the third module in the artificial intelligence model combination to dynamically analyze and process the art creation information and the art creation summary fed back by the visitor according to the art creation instruction to generate a human-machine co-created art work includes: In response to receiving the art creation information fed back by the visitor according to the art creation instruction, obtaining art creation interface information; Using the third module in the artificial intelligence model combination to perform color segmentation on the art creation interface information to obtain information of a plurality of color modules; Based on the information of all the color modules and the artistic creation summary, a human-machine co-created artwork is generated.

[0012] Optionally, the process of generating the next interactive task includes: Collect historical interaction results within a preset time period; Automatically generate a current interaction result based on the conversation summary and the human-machine co-created artwork; In response to receiving at least one of the information from the server based on the current interaction result, the human-machine co-created artwork and the historical interaction result, a next interaction task is generated with the assistance of artificial intelligence.

[0013] Optionally, collecting historical interaction results within a preset time period includes: Collecting personal information of the visitor; Collecting the number of times the visitor performs the interactive task within the preset time period; Collecting a conversation summary of each interactive task performed by the visitor within the preset time period; Based on the personal information, the times and / or the session summary, a historical interaction result is generated.

[0014] Optionally, the automatically generating a current interaction result based on the conversation summary and the human-machine co-created artwork includes: Collecting original artworks, human-computer co-created artworks and conversation summaries corresponding to several target time nodes in the current interactive task; Based on the original artwork, the human-machine co-created artwork and / or the conversation summary corresponding to all the target time nodes, the current interaction result is automatically generated.

[0015] Optionally, the next interactive task is generated with the assistance of artificial intelligence, including: In response to receiving the conversation reflection content fed back by the visitor according to the current interaction result, generating optimized conversation principles and optimized example questions; Based on the optimized dialogue principles and the optimized example questions, the next interactive task is generated with the assistance of artificial intelligence.

[0016] Optionally, the multimodal session information includes: At least one of audio data, video data, text data, artistic creation information, physiological information and environmental information.

[0017] A second aspect of the present invention provides a human-computer interaction system based on asynchronous art creation, the system comprising: The information collection module is used to collect original artworks and multimodal conversation information input by visitors according to the current interactive task in real time; An art creation module, for dynamically analyzing and processing the original artwork and multimodal conversation information using a combination of artificial intelligence models, to generate human-machine co-created artworks and conversation summaries; The interaction module is used to feed back the human-machine co-created artwork and the conversation summary to the server, and in response to receiving the next interaction task generated by the server based on the conversation summary and the human-machine co-created artwork with the assistance of artificial intelligence, feed back the next interaction task to the visitor.

[0018] A third aspect of the present invention provides a human-computer interaction device, comprising a memory for storing executable instructions; and a processor for calling and running the executable instructions in the memory to execute any one of the steps of the above-mentioned human-computer interaction method based on asynchronous artistic creation.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium, wherein program instructions are stored in the computer-readable storage medium. When the program instructions are executed by a processor, any step of the above-mentioned human-computer interaction method based on asynchronous artistic creation is implemented.

[0020] Compared with the prior art, the beneficial effects of this solution are as follows: The present invention combines artistic creation and multimodal conversation information through human-computer interaction, helping visitors to externalize their feelings and experiences in the process of artistic creation. With the assistance of artificial intelligence, the service provider provides structured guidance based on the conversation summary and human-computer co-creation of artworks, so as to integrate the service provider's professional beliefs, personal style and emotional support into the interactive tasks, and can provide customized and personalized interaction needs for visitors. The present invention realizes asynchronous collaboration between the service provider and the visitor through the human-computer interaction system, allowing the service provider and the visitor to maintain interaction between two meetings, and helping the visitor to establish a long-term trust and intimacy; at the same time, under the guidance of the artificial intelligence model, it significantly reduces the difficulty of visitors to perform artistic creation tasks, helps to improve the self-confidence of visitors, and improves the efficiency of asynchronous interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 A flow chart of the human-computer interaction method based on asynchronous artistic creation of the present invention; Figure 2 A schematic diagram of the process flow of the human-computer interaction system based on asynchronous art creation of the present invention; Figure 3 A schematic diagram of a visitor-oriented application program interface of the present invention; Figure 4 This is a schematic diagram of the second application program interface facing visitors of the present invention; Figure 5 The third schematic diagram of the application program interface for the server of the present invention; Figure 6 A fourth schematic diagram of the application program interface for service providers of the present invention; Figure 7 An interactive flow chart of stage 1 of an example of performing an interactive task of artistic creation according to the present invention; Figure 8 The interactive flow chart of stage 2 of the example of performing the interactive task of artistic creation of the present invention; Fig. 9 The interactive flow chart of stage three of the example of performing the interactive task of artistic creation of the present invention; Fig.10 It is a structural schematic diagram of the human-computer interaction device of the present invention. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0024] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0025] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0026] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0028] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0030] In the existing art therapy, service providers often recommend interactive tasks that combine art creation with verbal expression. For example, service providers encourage visitors to complete home creations every day, such as observing, painting, taking pictures, and writing about the sky; in interactive tasks, art creation is combined with verbal exercises to help visitors improve their daily coping abilities. This method not only allows visitors to explore their inner thoughts in depth and find ways to release emotions through words, but also brings a unique healing experience with the creativity and expressiveness of art creation. However, the lack of guidance from service providers may increase the psychological threshold for art creation, and the differences in individual understanding and expression of emotions make it more challenging to express past emotions and experiences in words. For example, a visitor is assigned to create a painting that expresses emotions as an interactive task, but there is no clear guidance. The lack of structural support may cause visitors to have negative beliefs that they "do not have the ability to express emotions" or "the work will not be understood", and thus feel frustrated. Such an experience will not only make visitors lose motivation for creation, but may also affect their trust in the entire therapy.

[0031] At the same time, service providers face many challenges in tracking and customizing interactive art creation tasks. Interactive tasks support asynchronous collaboration between service providers and clients, allowing clients to maintain interaction between two meetings and help build long-term trust and intimacy. This asynchronous communication requires service providers to track the records and history of interactive tasks, and these valuable data can provide clients with the mentality during the treatment interval and provide a basis for adjusting the treatment plan. However, managing and tracking these records will increase the workload of service providers and is not easy to achieve. In addition, service providers play an important role in strengthening the treatment process while recording the progress of treatment. However, creating structured family creations that take into account the emotional needs of clients is still a major problem in treatment. For example: ① The service provider learned that the client had difficulty expressing emotions and suggested that he record his emotions through a diary, but found that the client felt stressed about writing. So the service provider adjusted the family creation to "drawing emotion cards", which made it easier for clients to participate and express their emotions intuitively. However, customizing family creations for different clients is still challenging, especially when emotional reactions are difficult to predict, service providers may not be able to accurately evoke the target emotions, affecting the treatment effect. ② The client was enthusiastic when he did graffiti creation for the first time, but because the service provider failed to give enough encouragement between the two sessions, the client lost interest in creation and eventually stopped. This example illustrates the importance of consistent encouragement to maintain the execution of family creation. Lack of support may reduce the enthusiasm of the client and affect the compliance of creation. ③ After each session, the client will be assigned to emotional exploration family creation, but because the service provider does not have a systematic record, it takes a lot of time to find the client's creation record each time, affecting the continuity of the session. This increases the workload of the service provider and makes it difficult to integrate and analyze the family creation data, and fails to detect the changes in the client's emotional management in a timely manner.

[0032] From the above analysis, it can be seen that in the existing technology, when performing art interaction tasks, the problems faced by service providers and clients mainly include: Problem 1, facing the challenge of performing art creation interaction tasks: Without the guidance of service providers, therapeutic activities based on art creation may bring creative obstacles; clients may lack confidence in completing interactive tasks or having emotional reactions due to lack of guidance, resulting in reduced compliance; clients may find it difficult to interpret works in a therapeutic way without support, reducing the motivation for deep reflection; without clear direction, it is difficult for clients to create meaningful drawings to promote reflection. Problem 2, facing the challenge of customizing therapeutic activities: service providers usually customize interactive tasks based on practical experience and therapeutic techniques (such as cognitive behavioral therapy, mindfulness), and it is difficult to flexibly adjust structured instructions in the current oral or written form, causing clients to forget or give up guidance; it is difficult to provide personalized encouragement to clients outside of communication time. Problem three, the challenge of tracking interaction history: since it is difficult to count a large amount of historical interaction task data, it is difficult to track the interaction task history, which may affect the accuracy of the service provider's formulation of interaction tasks; using the existing face-to-face interaction method to perform interaction tasks is likely to miss valuable data on the visitor's emotional or mental state when completing the interaction task.

[0033] Based on this, the present invention applies human-computer interaction technology to art interaction tasks and proposes an exemplary human-computer interaction system, which is mainly composed of a front-end including a visitor-oriented application and a server-oriented application, and a back-end including an artificial intelligence model combination agent module, an AI art creation module and a memory module.

[0034] For the front-end: The front-ends of both the visitor and server applications are web-based user interfaces built using the Quasar framework, an open source framework implemented on Vue.js, a progressive JavaScript framework for building user interfaces. The conversational interactions of the client-facing application use an AI company's text-to-speech model to process client messages, generate voice messages that play automatically by default, and use an existing voice dictation service to implement speech-to-text functionality. The server-facing application interacts with the back-end server via Hypertext Transfer Protocol requests, and data is stored and transmitted in JSON format. Servers can select visitors from a list of names, view session logs, assign custom home creations, and modify the agent's guide.

[0035] For the backend: Flask (‌Flask‌ is a lightweight web application framework written in Python) is used to generate artworks, process the workflow of the combination of artificial intelligence models, and manage data. The AI ​​art creation module includes a control network segmentation unit and an art work generation AI unit. The control network segmentation unit is used to segment the color, sound, etc. of the artwork. With the art work generation AI unit, the system uses queue-based asynchronous processing and multi-threading to process multiple concurrent art work generation requests. The backend of the AI ​​model agent module workflow consists of five agents that use the same or different AI models, mainly including: The first module is a conversation agent, which is used in the discussion stage, combining the principles of the server and customized questions to interact with visitors. The second module is an art creation input agent, which is used in the art creation stage to convert the visitor's drawings and descriptions into multimodal inputs for the AI ​​art creation module. The third module is an art creation generation agent, which is used in the art creation stage to use the visitor's original artwork and the multimodal input of the AI ​​art creation module for generative creation of AI artworks. The fourth module is the conversation recording agent, which is used to process the multimodal data of the visitor's human-computer conversation and generate summary and record information that is easy to retrieve and query for the service provider to use. The fifth module is the creation recording agent, which is used to integrate the multimodal data of the entire artistic creation interaction process and provide the service provider with relevant information and analysis of the visitor. In addition, the backend handles data storage and retrieval, manages user-generated content such as drawings, images, drawing logs, and conversation history records. Images are stored in portable network graphics format, and logs and settings are saved in JSON format. All services use RESTful API (RESTful API is a web service interface based on the REST (Representational State Transfer) architectural style) to communicate. The backend service sets up multiple APIs to ensure the smoothness of front-end and back-end interaction. The data storage module of the backend is used to store all data related to the interaction task (including dialogue principles, dialogue content, original artworks, human-computer interaction artworks, and AI-compiled interaction result logs) in a database to facilitate tracking of historical interaction records and provide reference for the formulation of subsequent interaction tasks.

[0036] At the same time, the human-computer interaction system at least includes the following functional modules: an information collection module, which is used to collect in real time the original artwork and multimodal conversation information input by the visitor according to the current interaction task; an art creation module, which is used to dynamically analyze and process the original artwork and multimodal conversation information using a number of artificial intelligence models to generate human-computer co-created artworks and conversation summaries; an interaction module, which is used to feed back the human-computer co-created artworks and the conversation summaries to the service provider, and in response to receiving the next interaction task generated by the service provider based on the conversation summary and the human-computer co-created artwork with the assistance of artificial intelligence, feed back the next interaction task to the visitor.

[0037] Furthermore, the human-computer interaction system of this embodiment can be installed on a variety of devices, allowing visitors to select different devices to access art interaction tasks in different natural environments, thereby reducing the difficulty of participating in the interaction.

[0038] Based on the above-mentioned human-computer interaction system, the present invention proposes a human-computer interaction method based on asynchronous artistic creation, which is deployed on electronic devices such as computers and servers, and is applied to scenarios where asynchronous interaction between service providers and visitors needs to be achieved through human-computer interaction, and is aimed at situations where artistic creation needs to be completed in an interactive manner. The above-mentioned scenarios can be various scenarios that require the participation of both parties in the interaction, such as smart medical care, educational counseling, elderly care, and artistic creation teaching. The present invention does not impose specific restrictions on the application scenarios. Among them, the relationship between service providers and visitors includes medical staff and patients, art mentors and students, and other objects that realize interactive tasks through artistic creation and language dialogue. Specifically, such as Figure 1 and Figure 2 As shown, the steps of the method in this embodiment include: Step S100: collecting original artworks and multimodal conversation information input by the visitor according to the current interactive task in real time; Specifically, during an interactive task, the human-computer interaction system first feeds back the current interactive task to the visitor, and the visitor successively inputs original artworks (such as two-dimensional paintings, two-dimensional animations, three-dimensional models, games, music, videos, etc.) and multimodal conversation information (such as text, voice, video calls, etc.) according to the content of the current interactive task received, and the human-computer interaction system collects original artworks and multimodal conversation information in real time. Among them, the current interactive task refers to the interactive task customized based on the results of the previous interaction. Original artwork refers to the artwork file generated by using the system-customized or existing art creation software or platform to receive and store the art creation information input by the visitor through brushes, gestures, keys, etc. For multimodal conversation information, the speech recognition engine can be used to convert speech into text, the text analysis engine can be used to extract key information, and the video analysis engine can be used to capture the actions and expressions in the video call. In particular, during the collection of the original artwork information and multimodal conversation information successively input by the visitor, the system needs to ensure the use of a secure transmission protocol to protect the security of the data, and ensure the integrity and timeliness of data transmission, as well as the reliability and stability of the monitoring system. Furthermore, in the entire collection and processing process, encryption technology is used to ensure the security and compliance of the transmitted and stored data to protect the privacy of visitors. It should be noted that this embodiment does not specifically limit the type of interactive tasks, which can be presented in the form of interactive tasks, questionnaires, etc.

[0039] Furthermore, the multimodal conversation information includes at least one of audio data, video data, text data, artistic creation information, physiological information and environmental information.

[0040] Specifically, since the multimodal conversation information includes one or more of audio data, video data, text data, artistic creation information, physiological information and environmental information. Among them, the artistic creation information includes the data information of the brushstroke pressure, the thickness, color and shape of the painting lines in the process of artistic creation by the brush, the movement amplitude and fluency in the process of artistic creation by the gesture, the pressing force, frequency and selection tendency in the process of artistic creation by the operation key, and the head tracking, eye tracking, expression analysis, gesture information detection, posture detection and the relative displacement information of the visitor's body in the process of painting in various painting forms; the physiological information includes the visitor's heart rate, brain waves, body temperature and other information; the environmental information includes the location and navigation information, light environment, sound environment and other information.

[0041] This embodiment expands the input of creative instructions into a new form, incorporating multimodal information into the analysis of the system and the consideration of the service provider. In addition to converting voice information into text, the user's emotions are analyzed through voice intonation to adjust the dialogue principle of the conversation agent. In addition to manual typing of text information, handwritten text and handwritten text pictures are directly input into the dialogue agent. By identifying the pressure of the pen strokes, the approximation of the handwriting to the standard font to judge the neatness and other factors, the visitor's habits and personality are adapted, and the factors are input to the service provider for the service provider to refer to and judge the visitor's personality, and customize personalized interactive tasks or dialogue principles. The information on the pressure of the handwritten text is collected through a pressure sensor. Similarly, in the process of creating an artwork, the touch screen operation can also reflect the above information; and on devices that are not operated by a touch screen, such as the use of external input devices such as a mouse and a keyboard, such as key press frequency, key press duration and time interval can also depict the general image of the visitor in the creative process, and the more impatient, decisive, calm or entangled personality characteristics. Action information can provide the system with a way to judge the visitor's emotional state in addition to the visitor's self-description. Head tracking, eye tracking, expression analysis, gesture information detection, posture detection, and the visitor's relative body displacement information can all provide sufficient information for the system and the service provider. If the visitor inputs relatively positive content during the text input process, and the expression analysis recognizes that the visitor's expression at this time is sad, the system will understand the situation of "saying one thing and thinking another", adjust the dialogue agent's dialogue principle to be warm, and prompt the service provider to adjust the customized interaction task of the visitor's current multimodal information contradiction, guide the visitor to face his true emotions and thoughts, and provide the visitor with better artistic creation interaction effects.

[0042] Physiological information can not only provide similar effects as action information input, but also provide timely warning for specific visitors in the process of artistic creation interaction. For example, for visitors who are afraid to express themselves and have low self-confidence, when they are emotionally tense, the system recognizes that the visitor's heart rate is accelerated and the body temperature is rising. The visitor is asked whether he needs to take a break, take a deep breath, and listen to soothing music. The visitor's emotional situation is input into the reminder window of the service provider in time, and the service provider determines whether manual intervention is needed in the artistic creation interaction process. Environmental information can be used as an auxiliary means of identifying and judging the above information, such as location and navigation information, light environment, sound environment and other information. A visitor is creating art in a noisy environment, and the receptors recognize that his heart rate is accelerated and his body temperature is rising. If only physiological information is used for analysis, the system may issue an emotional warning prompt at this time. Combined with the environmental sound situation and the previous shy personality characteristics of the visitor, the system analyzes that the visitor is currently only in a strange and noisy environment, and will not issue a warning prompt, but will record the multimodal information and recognition results as a reference for subsequent service providers to review.

[0043] It is easy to understand that multimodal interactive information includes but is not limited to the above aspects. The focus is that the fusion of multiple types of interactive information can provide more accurate, effective and comprehensive support information for the system to form summaries, judge the current status of visitors, and provide intelligent prompts for service providers to customize interactive tasks.

[0044] Step S200: using an artificial intelligence model combination to dynamically analyze and process the original artwork and multimodal conversation information respectively, to generate a human-machine co-created artwork and a conversation summary; Specifically, according to the task requirements, appropriate artificial intelligence model methods are selected. These methods have natural language processing capabilities and a preliminary understanding of the visitors' artistic creation process and interim results; for the analysis of artworks, different modal inputs such as images and audio are combined to extract key elements in artistic creation. According to the preset interaction rules or algorithms, guide each artificial intelligence model to dynamically analyze and process the collected original artworks and multimodal conversation information according to the division of tasks, including first digitizing the original artworks, extracting analyzable features (such as color, shape, texture, etc.), and preprocessing the multimodal conversation information, such as speech recognition, text segmentation, noise removal, etc.; then identify the theme, style, emotion, etc. contained in the original artworks according to the features of the original artworks, and identify the theme, emotion, key viewpoints, behavior pattern recognition, etc. contained in the multimodal conversation information according to the speech, text or expression obtained by preprocessing; finally, use the artificial intelligence model to combine the recognition results of the original artworks and multimodal conversation information, comprehensively analyze the contextual relationship in the interaction process, understand the intentions and expectations of the visitors, generate human-machine co-creation artworks that can match the content of the original artworks and the intentions of the visitors, and extract the main viewpoints, emotional tendencies and key information in the multimodal conversation information, and generate concise and clear conversation summaries and records. Furthermore, it can also provide brief explanations or background information for human-machine co-creation artworks and conversation records to help service providers better understand.

[0045] Step S300: Feedback the human-machine co-created artwork and the session record to the server, in response to receiving the next interactive task formulated by the server based on the session record and the human-machine co-created artwork with the assistance of artificial intelligence, and feed back the next interactive task to the visitor.

[0046] Specifically, the human-computer interaction system sends the human-computer co-created artwork and conversation records to the service provider, and attaches a brief explanation or request to remind the service provider of the confidentiality of the materials and inform the visitor of the basic situation and needs. After receiving the current interaction information fed back by the visitor, the service provider determines the visitor's current mental, emotional, behavioral and / or cognitive conditions with the assistance of artificial intelligence based on the conversation records and human-computer co-created artwork in the current interaction information, as well as the visitor's personal information and communication conditions recorded in advance, and formulates personalized interaction principles based on the conditions of various aspects, and feeds back the interaction principles and interaction methods to the human-computer interaction system, generates the next interaction task with the assistance of artificial intelligence, and feeds back the next interaction task to the visitor, so that the visitor can perform the next interaction task in accordance with the process of executing the current interaction task under the guidance of the human-computer interaction system, thereby meeting the asynchronous interaction needs between the visitor and the service provider.

[0047] In this embodiment, asynchronous collaboration between the server and the visitor is achieved through the human-computer interaction system, so that the server and the visitor can maintain interaction between two meetings and help the visitor establish long-term trust and intimacy; at the same time, under the guidance of the artificial intelligence model, the difficulty of the visitor in performing artistic creation tasks is significantly reduced, which helps to improve the visitor's creation and expression ability and improve the effect of asynchronous interaction.

[0048] In a preferred embodiment, the real-time collection of original artworks and multimodal conversation information input by the visitor according to the current interactive task in step S100 includes: Step S110: constructing a series of interactive questions based on the dialogue principles and example questions corresponding to the current interactive task; Specifically, after receiving the current interaction information from the visitor, the service provider determines the visitor's current mental, emotional, behavioral and other conditions based on the conversation summary and human-computer co-creation of art works, as well as the visitor's personal information and communication situation, and formulates personalized dialogue principles and sample questions based on the conditions of various aspects, and feeds the dialogue principles and sample questions back to the human-computer interaction system. The human-computer interaction system customizes the interaction process based on the dialogue principles and sample questions, and generates a series of interaction questions based on the interaction process, so as to guide the visitor to successfully complete the art creation process. Among them, the dialogue principle refers to the key interaction content taken to solve the visitor's appeal for participating in the creative activity; the sample question refers to the guiding question corresponding to the interaction process set for completing the key interaction content in the dialogue principle. For example, if the current interactive task is to let the visitor draw a two-dimensional painting with a brush, the dialogue principles can be set to inform the visitor of the content of the artistic creation, guide the visitor to name his or her artwork, guide the visitor to describe his or her feelings about the artistic creation, etc. Then the sample questions can be set as: 1) Dear user, please draw the spring in your mind (for example, the visitor draws clouds and trees on the electronic painting screen with a brush); 2) What shape and color do you want the clouds and trees to be? 3) What do you want on the trees? 4) Please name this painting; 5) Please describe your feelings about creating this painting.

[0049] Step S120: Feedback the interactive questions to the visitor in sequence, and collect the original artwork and multimodal conversation information corresponding to each of the interactive questions fed back by the visitor in sequence.

[0050] Specifically, the order of setting the interactive questions of the human-computer interaction system gradually guides the visitors to the forms, types and contents of artistic creation that can be adopted, and according to the original artworks and multimodal conversation information corresponding to each of the interactive questions fed back by the visitors during the artistic creation process, gradually guides the visitors to introduce the content and psychological state of their artistic creation, so as to pave the way for the subsequent human-computer interaction system to deeply understand the mental, emotional and behavioral states and tendencies of the visitors. Among them, the forms of artistic creation include but are not limited to manual drawing, gestures and key operations, etc., and the types of artistic creation include but are not limited to two-dimensional paintings, two-dimensional animations, three-dimensional models, interactive games, music, videos, etc. The content of artistic creation includes but is not limited to characters, scenery, animals, still lifes, scenes and other types of content.

[0051] In a preferred embodiment, the step S200 uses the combination of artificial intelligence models to dynamically analyze and process the original artwork and multimodal conversation information to generate human-machine co-created artwork and conversation summaries and records, including: Step S210: Analyze the multimodal conversation information using the first module in the artificial intelligence model combination to generate several types of conversation instructions; Specifically, since the multimodal conversation information may contain one or more of multiple types of data such as audio data, video data, text data, artistic creation information, physiological information, and environmental information, and different data are processed in different ways, this embodiment performs standardization processing and / or data conversion on the information other than the artistic creation information in the multimodal conversation information to obtain data in a target format, so as to eliminate the dimensional influence between different types of data, so that different data indicators are comparable, which is conducive to improving the accuracy of comprehensive evaluation. For example, for standardization processing, the audio signal is normalized by scaling the amplitude of the audio signal to a fixed range (such as [-1, 1]) to eliminate the amplitude difference under different recording conditions and improve the effectiveness of feature extraction; for color information in video data, the color information is adjusted to a uniform range to maintain consistency under different lighting conditions and eliminate color differences under different devices or shooting conditions; for text data, the modulus of the text vector is normalized so that vectors of different lengths can be compared on the same scale; for light environment information in environmental information, features such as light intensity and color temperature are calculated and normalized to eliminate differences under different lighting conditions. For data conversion, voice is converted into text or text is converted into voice to realize the voice and text conversion function of the art creation dialogue agent window.

[0052] Then, the first module in the artificial intelligence model combination is used to analyze and reason the data in the target format to generate several types of conversation instructions. For example, if the human-computer interaction system extracts sad emotional information from the visitor's expression in the video data uploaded by the visitor, the human-computer interaction system will generate a conversation instruction for asking the visitor what or who made him sad; for another example, if the human-computer interaction system extracts happy emotional information from the text data uploaded by the visitor, the human-computer interaction system will generate a conversation instruction for asking the visitor what he most wants to do or who he most wants to see.

[0053] Step S220: Feedback the conversation instruction to the visitor, and generate several types of conversation information in response to receiving the visitor's feedback information on the conversation instruction; Specifically, the conversation instructions are fed back to the visitor, and the visitor expresses himself verbally or non-verbally under the guidance of the conversation instructions. In each interactive task, the artificial intelligence model constructs multiple conversation instructions through question-and-answer methods based on the dialogue principles and sample questions corresponding to the current interactive task, so as to gradually guide the visitor to actively complete the interactive task, and help the visitor regulate emotions, express himself, and solve the visitor's confusion.

[0054] Step S230: All the session information is integrated to generate a session summary and record.

[0055] Specifically, in each interactive task, the multiple questions and answers communicated between the artificial intelligence model and the visitor are integrated in chronological order or topic type to generate a complete conversation content, and the key information in the complete conversation content is extracted to generate a conversation summary, so that the service provider can quickly and accurately grasp the visitor's mental, emotional, behavioral and / or cognitive status information in this interactive task.

[0056] In a preferred embodiment, the step S200 uses the combination of artificial intelligence models to dynamically analyze and process the original artwork and multimodal conversation information to generate human-machine co-created artwork and conversation summaries and records, including: Step S240: Analyze the art creation information in the multimodal conversation information using the second module in the artificial intelligence model combination to generate art creation instructions and art creation records; Specifically, the second module in the artificial intelligence model combination is used to analyze the artistic creation information in the multimodal conversation information. For example, by analyzing data information such as brush stroke pressure, line thickness, color and shape, the visitor's creative content, style, and intention can be understood; by identifying information such as the amplitude and fluency of gesture movements, the visitor's creative process and possible emotions can be inferred; by analyzing information such as pressure intensity, frequency and selection tendencies, the visitor's creative decisions and preferences can be understood; by parsing text descriptions related to artistic creation, the theme and style of the artwork created by the visitor can be understood.

[0057] Then, the second module is used to generate a series of art creation instructions based on the analysis results of the art creation information in the multimodal conversation information to guide the visitor or assist the human-computer interaction system to continue or improve the art creation process. For example, it is recommended that the visitor adjust the creation style to better meet the specific theme or needs; based on the analysis of gestures and expressions, it is recommended that the visitor better express specific emotions or atmospheres in the work; the visitor's feedback is converted into specific creation instructions to guide the visitor to modify or improve, etc. At the same time, the second module is used to extract the visitor's art creation summary throughout the art creation process based on the analysis results of the art creation information in the multimodal conversation information, so as to outline the characteristics of the theme, style, composition, color, etc. of the art works created by the visitor, and analyze and summarize the emotions and intentions expressed by the visitor in the creation process.

[0058] Step S250: Feedback the art creation instruction to the visitor, and use the third module in the artificial intelligence model combination to dynamically analyze and process the art creation information and the art creation summary fed back by the visitor based on the art creation instruction to generate a human-machine co-created art work.

[0059] Specifically, the art creation instruction is fed back to the visitor, and in response to receiving the art creation information fed back by the visitor according to the art creation instruction, the art creation interface information is obtained. If the art creation is digital, the image data can be directly extracted as the art creation interface information; if the art creation is physical, it can be converted into a digital image by taking a photo or scanning, and then the image data is extracted. Then, the third module in the artificial intelligence model combination is used to identify the color and semantic information in the art creation interface information, and the art creation interface information is segmented by color and semantics to obtain the shape, position, and size of several color and semantic segmentations. Finally, based on the descriptive information about the theme, style, emotion and intention of the work provided by all art creation summaries and records, and the detailed information about the color and semantic segmentation of the work provided by all color and semantic segmentation information, the third module is used for generative creation to create a human-machine co-created art work that matches the above input information.

[0060] In a preferred embodiment, the process of generating the next interactive task in step S300 includes: Step S310: Collect historical interaction results within a preset time period; Specifically, the personal information of the visitor is collected, such as height, weight, name, email address, contact information, etc.; the number of times the visitor performs interactive tasks within a preset time period is collected, that is, by setting up a log system or database, the number of times the visitor performs various interactive tasks within a preset time period is recorded; the conversation summaries and records of each interactive task performed by the visitor within the preset time period are collected, and the fourth module is used to analyze the visitor's execution of the interactive task, and generate AI-compiled summary information, including task type, completion, error rate, interactive content summary and other information; and personal information, number of times, original artwork, human-computer co-creation artwork and / or conversation summary are integrated to generate historical interaction results, thereby forming a complete visitor interaction data set or historical interaction result report, so as to realize the evaluation of indicators such as visitor activity, task completion efficiency, and AI-assisted effect. Furthermore, the historical interaction results are intuitively displayed in the form of charts, dashboards, etc., which facilitates in-depth understanding and improves the evaluation accuracy.

[0061] Step S320: automatically generating a current interaction result based on the conversation summary and the human-machine co-created artwork; Specifically, several key time nodes in the current interactive task are identified, and these nodes are used as reference points for collecting original artworks; original artworks, human-computer co-creation artworks and conversation summaries corresponding to each target time node in the current interactive task are collected; original artworks, human-computer co-creation artworks and conversation summaries are integrated into a unified data set to ensure that the data of each time node is complete and interrelated; image recognition, text analysis and other technologies are used to deeply analyze artworks and conversation summaries, and extract key features such as style, theme, and emotion; the changing trends of artworks and conversation content at different time nodes are analyzed to identify visitors' creative preferences, the effects of AI assistance, and potential problems in the creative process; based on the integrated data and analysis results, the original artworks, human-computer co-creation artworks and conversation summaries at each time node are comprehensively evaluated, and according to the evaluation results, a report or summary of the current interactive results is automatically generated, including the advantages and disadvantages of the artworks, the effectiveness of human-computer co-creation, and the highlights of visitors' feedback. Furthermore, the interactive results are intuitively presented to visitors and service providers in the form of charts, images, videos, etc., so as to better understand the entire interactive process and the final results. It should be stated that the current interaction results only refer to the original artworks, human-computer co-created artworks and conversation summaries generated based on the visitor's corresponding to the current interaction task, while the historical interaction results are the comprehensive evaluation results of multiple task results completed by the visitor in the past corresponding to the current interaction task.

[0062] Step S330: In response to receiving at least one of the information from the server based on the current interaction result, the human-machine co-created artwork and the historical interaction result, generating the next interaction task with the assistance of artificial intelligence.

[0063] Specifically, the human-computer interaction system responds to receiving at least one information from the service provider based on the current interaction result, the human-computer co-created artwork, and the historical interaction result (even combined with the service provider's experience), and deeply understands the changes in the visitor's emotions, behaviors, cognition, etc. contained in the current interaction result, the emotional depth expressed in the artwork, the improvement of interpersonal interaction, the elements, color application, composition layout, etc. in the human-computer co-created artwork, so as to understand the visitor's emotional, cognitive and other physical and mental states. At the same time, by reviewing the historical interaction results, the change trend of the visitor in the target time period is analyzed, including progress, challenges and recurring themes, and the current interaction results are compared with the historical data to find connections and differences to guide the service provider to generate the next interaction task. In order to achieve an effective feedback mechanism, ensure that the visitor can obtain the guidance and support of the service provider in a timely manner, and enable the service provider to closely observe the visitor's performance, record the key events and the visitor's reactions during the task execution process. It is beneficial for the service provider to assist the human-computer interaction system to flexibly adjust the task content, difficulty or interaction method according to the task execution situation and the visitor's real-time feedback to ensure the effectiveness and safety of the interaction process.

[0064] In a preferred embodiment, in step S330, the next interactive task is generated with the assistance of artificial intelligence, including: Step S331: in response to receiving the conversation reflection content fed back by the visitor according to the current interaction result, generating optimized conversation principles and optimized example questions; Step S332: Based on the optimized dialogue principle and the optimized example question, generate the next interactive task with the assistance of artificial intelligence.

[0065] Specifically, this embodiment generates optimized dialogue principles and optimized sample questions by asking or asking visitors to actively feedback the conversation reflection content of the current interaction results, and using artificial intelligence models to identify the emotions and needs expressed by visitors in the reflection, such as seeking more understanding, hoping to obtain more specific guidance, and hoping for a smoother conversation, so as to make the optimized dialogue principles clearer and more concise, easy for visitors to understand and follow, and encourage visitors to actively participate in the conversation to enhance the interactivity and effectiveness of the conversation; so that the optimized sample questions can better guide visitors to think deeply and express themselves. Then, with the assistance of artificial intelligence, the optimized dialogue principles and optimized sample questions are integrated into the next interaction task to update and flexibly adjust the next interaction task in real time, thereby improving the quality and effect of the conversation and better meeting the needs and expectations of visitors.

[0066] Furthermore, the AI ​​model can be fine-tuned or retrained based on historical interaction results over a period of time to improve its accuracy and efficiency. Alternatively, more training data, especially data related to artworks and conversation information, can be introduced to enhance the generalization ability of the model, and the optimized model can be applied to new artworks and conversation information, and the entire process can be continuously iterated and optimized to improve the quality and efficiency of human-machine co-creation of artworks and conversation summaries.

[0067] It should be stated that in this embodiment, whenever data related to personal information of future visitors or service providers is applied to specific products or technologies, it is used with the user's permission or consent, and the collection, use and processing of the relevant data comply with relevant standards.

[0068] An exemplary embodiment of a human-computer interaction system and method is provided below, which is as follows: In order to support the visitor's art creation interactive task and the service provider-visitor cooperation around it, the human-computer interaction system designed in this embodiment includes: an application for visitors, which combines the art creation co-created by humans and artificial intelligence with conversational interaction to promote interactive tasks in daily environments. An application for service providers, which provides AI-compiled interactive task history and customization of visitor interactive task agents to obtain tailored guidance, see Figure 3-Figure 6 .

[0069] In order to solve the three key challenges existing in the prior art, the core design functions of the human-computer interaction system provided in this embodiment are as follows: First, combine human-AI co-creation of art with conversational interaction (visitor-facing applications). To solve problem one, the visitor-facing application uses a human-AI co-creation canvas (such as Figure 3 ) to lower the threshold for visitors to create art, and use the two-stage dialogue workflow to provide structured guidance for visitors in the "art creation stage" and "discussion stage". Figure 3As shown in , the application for visitors has three panels: AI brush window 305, user creation canvas 303 and AI generated image window 304. Among them, AI brush window 305 includes various creation tools that use AI's capabilities to generate objects with matching shapes by creating semantic brush elements such as color blocks and geometric bodies; user creation canvas 303, a visitor drawing for color segmentation; AI generated image window 304, for previewing human-AI co-created artworks. The art creation stage dialog box includes a painting description editing area 301 and an art creation dialogue agent window 302, which are respectively used to extract painting generation keywords and prompt visitors to verbally describe the work in detail. The discussion stage dialog box includes the following. Figure 4 The discussion phase dialogue agent window 401 is used to promote the visitor to conduct in-depth self-exploration and reflection according to the task instructions. The dialogue principles and sample questions of the initial task guidance are derived from existing art interaction programs and are refined through feedback from professional art service providers.

[0070] Second, it supports custom client authoring agents (server-oriented applications).

[0071] To solve problem 2, the server-oriented application has three panels for customization, such as Figure 6 As shown, it includes: AI dialogue principle customization area 603, interactive task customization 601 and personal message customization area 602. Among them, AI dialogue principle customization area 603: allows the service provider to customize the dialogue principle, modify the sample questions and adjust the dialogue process according to practical experience. Interactive task customization area 601: is used to support the service provider to assign interactive tasks. Personal message customization area 602: tailor personal information for the visitor to provide encouragement and emotional support in the art therapy interactive task.

[0072] Third, enable AI-compiled interaction task history (server-oriented applications).

[0073] To solve problem 3, the application for service providers also has the following panels: an interactive task overview area, an interactive task record area for each section, and an AI compilation summary area. Figure 5 As shown, the interactive task summary area includes a personal information area 501, a usage log area 502, and a creative work summary area 503. Figure 5 The personal information area 501 in the interface is hidden to avoid user privacy. In actual applications, this part of the interface will display the visitor's personal information. The interactive task record area of ​​each section includes the original creative work area 504, the AI-generated creative work area 506, the creative process description area 505, and the AI ​​dialogue record area 507. AI compilation summary: used to summarize the visitor's image description, feelings and experience, and provide insights for the service provider.

[0074] Based on the system provided by the exemplary embodiment, the method of the present invention is used to implement four different types of interactive tasks, including but not limited to the following: Example 1: An interactive task of creating a two-dimensional artwork using voice and text.

[0075] The creation tools included in the AI ​​brush window 305 include: brush elements with semantics: ocean, cloud, earth, lawn, mountain, tree, hill, lake, person, wall, road, etc. Drawing in the form of color blocks; brush thickness adjustment bar; eraser tool; color palette; painting style selector: watercolor, oil painting, illustration, etc. Detail adjustment is achieved during the interaction process through the art creation dialogue agent window 302.

[0076] This embodiment is based on the visitor's initial two-dimensional hand-painted work, using the art creation dialogue agent window to input detailed descriptions, and iterating multiple times to generate the target two-dimensional image art work. The specific steps are as follows: In the previous communication, the service provider learned that the visitor currently has a certain degree of anxiety in her marital relationship, and customized a task for the visitor to draw a two-dimensional image artwork. The image must include two plants, one representing herself and the other representing her partner. Through the art creation dialogue agent window 302, the visitor can understand the task of this interactive task.

[0077] In the initial stage, the visitor selects the corresponding "semantic brush element". In this embodiment, the visitor selects the "tree" brush element, and then paints the shape of a tree on the user creation canvas 303; uses the color palette to select the brush color, and uses the brush thickness adjustment lever to adjust the brush thickness; the visitor uses the eraser tool to erase the wrong and cancelled parts.

[0078] After the system recognizes that there are traces of painting on the user creation canvas 303, the dialogue agent prompts the visitor to describe the specific details of the elements through voice input, and this prompt is displayed in the art creation dialogue agent window 302. At this time, the detail adjustment stage is entered. The visitor describes the shape of the painting as two apple trees with small flowers on them through voice input; the dialogue agent generates a summary of "apple trees, blooming, two trees" based on the input and displays it in the painting description editing area; the visitor selects the "mud", "clouds", and "lawn" brush elements in turn, and adds more details to the user creation canvas 303; the visitor uses the eraser tool to erase the wrong and canceled parts; after completing the details, the visitor selects the watercolor style option on the painting style selector.

[0079] Click Generate to enter the artwork preview stage. In the AI ​​generated image window 304, the original artwork generated at this stage is displayed; visitors use voice input and creation tools to generate human-machine co-created artworks based on original artworks using artificial intelligence models, and adjust the human-machine co-created artworks many times until the visitor sees the expected artwork in the AI ​​generated image window 304. Click OK to export the final human-machine co-created artworks, generate the interaction results compiled by AI, and end this interaction task.

[0080] Example 2: An interactive task of creating an animated artwork using voice and text.

[0081] The creation tools included in the AI ​​brush window 305 include: brush elements with semantics: ocean, cloud, earth, lawn, mountain, tree, hill, lake, person, wall, road, etc. Drawing in the form of color blocks; brush thickness adjustment bar; eraser tool; color palette; painting style selector: watercolor, oil painting, illustration, etc. Detail adjustment is achieved during the interaction process through the art creation dialogue agent window 302.

[0082] Object picker: click (click on color block), range selection, all selection, inverse selection, deselection, screen selection, etc., to facilitate adding actions or special effects to a single part later; action brush: basic movement: move, rotate, scale, jump; deformation effect: stretch and compress, twist, wave effect; loop and interaction: repeat action, path movement; special effect brush: light and shadow: glow, shadow; particle: smoke, spark; weather and environment: raindrops, snowflakes; dynamic: blur, shake; color: filter, flicker; space: virtual depth, starry sky; editing and playback tools: timeline editor: track manager; keyframe editor; audio editor: input dialogue and sound effects, editable. Through the artistic creation dialogue proxy window 302, detailed adjustments can be made during the interaction process.

[0083] This embodiment is based on the visitor's initial creation of a two-dimensional animation work, using the art creation dialogue agent window 302 to input detailed descriptions, and iterating multiple times to generate the target animation art work. The specific steps are as follows: During the early communication, the service provider learned that the visitor was currently experiencing seasonal affective disorder, and customized a task for the visitor to create an animated artwork. The animation must include four elements representing spring, summer, autumn and winter, and be able to show the environmental changes in different seasons. Through the art creation dialogue agent window, the visitor can understand the task of this interactive task.

[0084] In the initial stage, the visitor selected the corresponding "semantic brush element". In this embodiment, the visitor selected the "tree", "flower", "fallen leaves" and "snowflake" brush elements, and then painted the shapes of trees, flowers, fallen leaves and snowflakes on the user creation canvas; used the color palette to select the brush color, and used the "brush thickness adjustment lever" to adjust the brush thickness; the visitor used the eraser tool to erase the wrong and canceled parts. The basic motion action brushes such as "move" and "rotate" were selected to apply to these objects, and the deformation motion action brushes such as "stretch" and "compress" were selected to apply to the fallen leaf object; the looping and interactive action brushes such as "repeat action" were selected to apply to all objects.

[0085] After the system recognizes that there are traces of painting on the user's creation canvas 303, the dialogue agent prompts the visitor to describe the specific details of the object through voice input, and this prompt is displayed in the art creation dialogue agent window 302. At this time, the detail adjustment stage has been entered. The visitor describes the shape of the painting through voice input as a tree swaying slowly in spring, a flower swaying gently in summer, a pile of fallen leaves flying in autumn, and a snowflake drifting in winter; the dialogue agent generates a summary of "spring tree, summer flower, autumn fallen leaves, winter snowflakes" based on the input and displays it in the painting description editing area; the visitor selects the "glow" special effect brush of the light and shadow category to apply to the tree element, the "smoke" special effect brush of the particle category to apply to the flower element, the "raindrop" special effect brush of the weather and environment category to apply to the fallen leaf object, and the "shake" special effect brush of the dynamic category to apply to the snowflake element, adding special effect details; after completing the details, the visitor selects the realistic style option on the painting style selector.

[0086] Entering the animation editing stage, the visitor opened the timeline editor. The "Track Manager" was used to arrange the actions and special effects of each element to the appropriate time nodes. The visitor used the "Keyframe Editor" to fine-tune the duration and transition effect of each action, making the process of changing seasons smoother.

[0087] In order to make the work more expressive, visitors added background music in the "audio editor" and added natural sounds of seasonal changes, such as birds singing in spring and cicadas chirping in summer, and other "sound effects" to enhance the integration of vision and hearing with the pictures.

[0088] Click Generate to enter the artwork preview stage. Visitors use the playback controller to preview the animation effect of the original artwork. In the AI ​​generated image window 304, the original artwork animation generated at this stage appears; based on the original artwork, visitors adjust the AI ​​generated animation multiple times through voice input, creation tools and editing tools. After the visitor uses the playback controller in the AI ​​generated image window 304 to see the expected work, click OK to export the final human-computer co-creation artwork, generate the AI-compiled interaction results, and end this interaction task.

[0089] Example 3: An interactive task of creating a three-dimensional model artwork using voice and text.

[0090] The creation tools included in the AI ​​brush window 305 are: brush elements with semantics: preset models: ocean, cloud, earth, lawn, mountain, tree, hill, lake, person, wall, road, etc.; three-dimensional primitives: cube, cylinder, cone, sphere, etc.; definition brush: ocean, cloud, earth, lawn, mountain, tree, hill, lake, person, wall, road, etc., the definition brush defines the object properties for the established three-dimensional primitives and their combinations; three-dimensional model design tools include: model parameter adjustment bar: parameter display: center coordinates, three-axis dimensions; adjustment lever and value input box: three-axis scaling lever, center adjustment lever, rotation lever; model combination options: combination: alignment, group; Boolean operation: union, difference, intersection, exclusion; deformation tools: stretching, twisting, squeezing; material and texture color palette. Detail adjustment is achieved during the interaction process through the art creation dialogue agent window 302.

[0091] This embodiment is based on the initial creation of the 3D model work by the visitor, and uses the art creation dialogue agent window 302 to input detailed descriptions and iterate multiple times to generate the target 3D model art work. The specific steps are as follows: In the previous communication, the service provider learned that the visitor was currently experiencing emotional stress, and customized a task for the visitor to create a three-dimensional model artwork. The animation must include mountains, lakes and lawns to form a natural landscape model. Through the art creation dialogue agent window 302, the visitor can understand the task of this interactive task.

[0092] In the initial stage, the visitor selected a brush element with semantics in the creation tool. In this embodiment, the visitor selected the "mountain" and "lawn" brush elements in the preset model, and then placed the shapes of the mountain and lawn in the user creation canvas; selected the "ellipsoid" brush element in the three-dimensional primitive and named the object "lake"; used the "material and texture palette" to select the model color; and formed the initial scene.

[0093] Entering the 3D model adjustment stage, visitors use the "three-axis scaling lever" to adjust the size of the lawn; use the "center adjustment lever" to adjust the position of the lake; use the "combine" function of Boolean operation to combine multiple peaks into one model; After the system recognizes that there is a model with material or texture in the user's creation canvas 303, the dialogue agent prompts the visitor to describe the specific details of the object through voice input, and this prompt is displayed in the art creation dialogue agent window 302. At this time, the detail adjustment stage has begun. The visitor describes the shape of the model as a very high and continuous snow-capped mountain, a clean lake, and a piece of green grass through voice input; the dialogue agent generates a summary of "continuous snow-capped mountains, lakes, and green lawns" based on the input and displays it in the painting description editing area; the visitor uses the "stretch" and "distort" functions in the deformation tool to refine the shape of the mountain; and uses the "squeeze tool" to increase the water depth on the lake model. After completing the details, the visitor selects the realistic style option on the painting style selector.

[0094] Click Generate to enter the artwork preview stage. Visitors use the 3D view observation bar to preview the scene effect of the original artwork. In the AI ​​generated image window 304, the 3D model generated at this stage appears; based on the original artwork, visitors adjust the AI ​​generated 3D model scene multiple times through voice input, creation tools and 3D model design tools. After the visitor sees the expected work using the 3D view observation bar in the AI ​​generated image window 304, click OK to export the final human-machine co-creation artwork, generate the interaction result compiled by AI, and end this interaction task.

[0095] Example 4: An interactive task of creating a three-dimensional interactive scene artwork using voice and text.

[0096] The creation tools included in the AI ​​brush window 305 are: brush elements with semantics: preset models: ocean, clouds, earth, lawn, mountain, trees, hills, lakes, people, walls, roads, etc.; three-dimensional primitives: cubes, cylinders, cones, spheres, etc.; definition brushes: ocean, clouds, earth, lawn, mountain, trees, hills, lakes, people, walls, roads, etc., the definition brush defines the object properties for the established three-dimensional primitives and their combinations; the three-dimensional model design tools are: model parameter adjustment bar: parameter display: center coordinates, three-axis dimensions; adjustment levers and value input boxes: three-axis scaling lever, center adjustment lever, rotation lever; model combination options: combination: alignment, group; Boolean operations: union, difference, intersection, exclusion; deformation tools: stretching, twisting, extrusion; material and texture palettes. Action Brush: Action application brush: basic movement: move, rotate, scale, jump; deformation effect: stretch and compress, twist, wave effect; loop and interaction: repeat action, path movement; bone binding selection bar and character action addition. Special effect brush: collision and impact effect: flash effect, vibration and shock wave, debris splash, sparks and smoke; physical reaction effect: drop effect, bounce effect, deformation animation; visual feedback: color change, highlight outline, transparency; particle effect: fire and explosion, smoke and dust, magic effect; sound and visual combination: sound effect synchronization, screen flashing; dynamic change effect: size change, rotation and displacement, split and combination; energy and light effect: energy flow, halo and flash; disappearance and generation effect: dissolution effect; portal effect; UI prompt: floating text, arrow or mark; trigger event effect: beam connection, chain reaction. Audio sound effects: Background music application bar: preset music / local music / AI music creation; sound effect brush: define interactive sound effects for the established three-dimensional primitives and their combinations; audio editor: input dialogue and sound effects, editable. Through the art creation dialogue agent window 302, detailed adjustments can be made during the interaction process.

[0097] This embodiment is based on the initial creation of a three-dimensional interactive scene work by the visitor, and uses the art creation dialogue agent window 302 to input detailed descriptions and iterate multiple times to generate the target three-dimensional interactive scene art work. The specific steps are as follows: In the early communication, the service provider learned that the visitor is currently going through a period of confusion and wants to seek inner peace and a sense of belonging, so the service provider customized a task for the visitor to create a three-dimensional interactive scene art work. The scene must include mountains, forests and houses to form a dynamic interactive scene. Through the art creation dialogue agent window 302, the visitor can understand the task of this interactive task.

[0098] In the initial stage, the visitor selects a semantic brush element in the creation tool. In this embodiment, the visitor selects the "mountain", "tree", "medieval dwellings" and "mud road" brush elements in the preset model, and then places the shapes of the mountain, tree, house and road in the user creation canvas 303; uses the "material and texture palette" to select the model color; and forms the initial scene.

[0099] Entering the 3D model adjustment stage, visitors use the "three-axis scaling lever" to adjust the size of the mountain; use the "center adjustment lever" to adjust the position of trees, houses, and roads; use the "combine" function of Boolean operations to combine multiple trees into one model to form a forest; use "rotation" to adjust the direction of the road.

[0100] Entering the stage of adding dynamic effects, visitors use the "Move" tool in basic motion to add movements to the trees in the forest, simulating the effect of trees being blown by the breeze; and use the "Smoke and Dust" in particle effects to add smoke to the user creation canvas 303.

[0101] After the system recognizes that there is a model with material or texture and dynamic effects in the user's creation canvas 303, the dialogue agent prompts the visitor to describe the specific details of the object through voice input, and this prompt is displayed in the art creation dialogue agent window 302. At this time, it enters the detail adjustment stage. The visitor describes the scene as a mountain, a lush forest, an ancient village, a muddy path between the village and the mountain, humid air, and mysterious light on the top of the mountain through voice input, and automatically adds other details suitable for the scene; the dialogue agent generates a summary of "mountains, forests, muddy paths, fog, mysterious light, and other details" based on the input and displays it in the painting description editing area; after completing the details, the visitor selects the oil painting style option on the painting style selector. Click Generate, and the visitor can use the 3D view observation bar to preview the dynamic scene.

[0102] After the system recognizes that a preview scene has been generated, the dialogue agent prompts the visitor to create the interaction subject and interaction effect, and then enters the interaction design stage.

[0103] First, create an interactive subject, that is, the visitor plays the role. The visitor chooses the "farmer" role in the preset model; adds a preset bone binding to the role in the "bone binding selection bar"; uses "character action addition", including "walk", "run", "jump", "raise hand", and "sit down"; sets the character interaction action to "raise hand".

[0104] Next, set the scene interaction effect. The visitor selected the "rotate" tool in the action brush and applied it to the doors and windows of the house, connecting the first interaction key; selected the "smoke and dust" in the particle effect and applied it to the muddy path, so that when the interactive subject, the player's character, passes by, the road surface can produce a dusty effect; selected the "fire and explosion" in the particle effect and applied it to the trees and the firewood and other combustible items added by the AI ​​in the scene during the detail adjustment stage, and connected the second interaction key for releasing magic; connected the tree's action to the character's "running" and "jumping" actions and the character model's coordinate position, and enhanced the action effect when triggered, so that when the character runs or jumps in the forest, the trees shake more.

[0105] In order to make the work more expressive, the visitor added background music in the "Audio Editor"; the visitor added sound effects to the character's action keys, such as the sound effects of walking, running, jumping and releasing magic; sound effects were added to the interactive actions of the scene, such as the sound effects of trees swaying slightly and violently, the sound effects of objects burning, and the sound effects of opening and closing doors and windows; and the integration of vision and hearing was enhanced by coordinating with the picture.

[0106] Click Generate to enter the artwork preview stage. Visitors use the action keys in the perspective switching and scene interaction bar to preview the three-dimensional scene display effect of the original artwork. In the AI ​​generated image window 304, the three-dimensional interactive scene generated at this stage appears; visitors adjust the AI ​​generated three-dimensional interactive scene multiple times through voice input, creation tools, three-dimensional model design tools, interactive design tools and audio and sound effect editing tools. After the visitor uses the "action keys" and "perspective switching options" in the scene interaction bar in the AI ​​generated image window 304 to see the expected work, click OK to export the final human-computer co-creation artwork, generate the interaction results compiled by AI, and end this interaction task.

[0107] Finally, a three-party interaction process for an online art interaction task is provided for the method of the present invention, which is as follows: Reason description: The client has been trying to deal with emotional problems with his partner, but feels misunderstood by his partner.

[0108] Execution steps: like Figure 7 Stage 1 shown: Preliminary communication (communication of basic situation and customization of personalized questions and support information).

[0109] 1. The visitor explains the basic personal situation of the problem he encountered to the service provider; 2. The server incorporates dialogue principles and example questions into the system to guide the dialogue agent to ask the right questions; 3. The service provider customizes questions according to the visitor's individual needs: customized interactive tasks related to the couple's relationship; 4. Provider emotional care for the individual: Supportive messages were added, “Your sensitivity and ability to put yourself in another’s shoes is truly a gift” to provide encouragement during the client’s interactive tasks; like Figure 8 Stage 2 shown: Artistic co-creation (communication of specific situations and corresponding artistic creation assignments).

[0110] 5. The client will inform the service provider of the helpless emotions he or she encounters after another quarrel; 6. The server assigns an interactive task: draw two plants, one for herself and the other for her partner; 7. The visitor opens the client application → enters the "art creation stage" on the tablet → selects the "tree" brush element from the toolbox → draws a tree; 8. Artistic creation agent prompts to describe through voice input; 9. The visitor described the tree as an apple tree full of blossoms; 10. The agent generates a summary based on the input. 11. The visitor added more semantic brush elements, such as "mud," "clouds," and "lawn," and completed her work; 12. The visitor chooses the watercolor style; 13. The visitor clicks Generate and creates a work of human-computer co-creation; Entering the discussion phase: 14. The agent guides the client to reflect on the work, asking questions such as “Do you want to describe your tree?” 15. The client shared her feelings and realized on her own that she and her boyfriend were like two different plants—independent but in need of mutual understanding; like Fig. 9 Stage 3 shown: Progress Reflection (retrospective thinking on creating works and describing information).

[0111] 16. The server reviews the visitor's interactive task history in the server-side application; 17. The service provider examined the original and co-created artworks, as well as the conversation records, and gained a deeper understanding of the client’s situation; 18. The agent prompts the client to consider the client's experience of arguing with his partner; 19. The server decided to address these insights in an upcoming meeting.

[0112] Through the above steps 1-19, we can gain an in-depth understanding of the client's current emotional problems with his or her partner, and through the execution of multiple interactive tasks, we can hopefully solve the emotional problems encountered by the client.

[0113] Based on the above embodiments, the present invention further provides a human-computer interaction device, whose principle block diagram can be shown as follows: Fig.10 As shown. The human-computer interaction device can be used to execute the human-computer interaction method based on asynchronous art creation provided in the above embodiment. For the sake of brevity, it is not repeated here. The human-computer interaction device includes: a processor, the processor is coupled to a memory, the memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions stored in the memory, so that the method in the above method embodiment is executed.

[0114] The present invention also provides a computer-readable storage medium on which computer instructions for implementing the method in the above method embodiment are stored.

[0115] For example, when the computer program is executed by a computer, the computer can implement the method in the above method embodiment.

[0116] The embodiment of the present application also provides a computer program product including instructions, which, when executed by a computer, enables the computer to implement the method in the above method embodiment.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

Claims

1. A human-computer interaction method based on asynchronous art creation, characterized in that: The following steps are involved: Real-time collection of original artwork and multimodal conversation information input by visitors according to the current interactive task; Using a combination of artificial intelligence models to dynamically analyze and process the original artwork and multimodal conversation information, respectively, to generate human-machine co-created artwork and conversation summaries; Feeding back the human-machine co-created artwork and the conversation summary to the server, in response to receiving a next interactive task generated by the server based on the conversation summary and the human-machine co-created artwork with the assistance of artificial intelligence, and feeding back the next interactive task to the visitor.

2. The human-computer interaction method based on asynchronous artistic creation according to claim 1, characterized in that: The real-time collection of original artworks and multimodal conversation information input by the visitor according to the current interactive task includes: Constructing a series of interactive questions based on the dialogue principles and example questions corresponding to the current interactive task; The interactive questions are fed back to the visitor in sequence, and the original artworks and multimodal conversation information corresponding to each of the interactive questions fed back by the visitor are collected in sequence.

3. The human-computer interaction method based on asynchronous artistic creation according to claim 1 or 2, characterized in that: The method of using the artificial intelligence model combination to dynamically analyze and process the original artwork and multimodal conversation information to generate human-machine co-created artwork and conversation summary includes: Analyzing information other than artistic creation information in the multimodal conversation information using the first module in the artificial intelligence model combination to generate several types of conversation instructions; Feeding back the conversation instruction to the visitor, and generating several types of conversation information in response to receiving the visitor's feedback information on the conversation instruction; All the session information is integrated to generate a session summary.

4. The human-computer interaction method based on asynchronous artistic creation according to claim 3, characterized in that: The first module in the artificial intelligence model combination is used to analyze the information other than the artistic creation information in the multimodal conversation information to generate several types of conversation instructions, including: Performing standardization processing and / or data conversion on the information other than the artistic creation information in the multimodal conversation information to obtain data in a target format; The first module in the artificial intelligence model combination is used to analyze the data in the target format to generate several types of conversation instructions.

5. The human-computer interaction method based on asynchronous artistic creation according to claim 1 or 2, characterized in that: The method of using the artificial intelligence model combination to dynamically analyze and process the original artwork and multimodal conversation information to generate human-machine co-created artwork and conversation summary includes: Analyzing the art creation information in the multimodal conversation information using the second module in the artificial intelligence model combination to generate art creation instructions and an art creation summary; The art creation instruction is fed back to the visitor, and the third module in the artificial intelligence model combination is used to dynamically analyze and process the art creation information fed back by the visitor according to the art creation instruction and the art creation summary to generate a human-machine co-created art work.

6. The human-computer interaction method based on asynchronous artistic creation according to claim 5, characterized in that: The method of using the third module in the artificial intelligence model combination to dynamically analyze and process the art creation information and the art creation summary fed back by the visitor according to the art creation instruction to generate a human-machine co-created art work includes: In response to receiving the art creation information fed back by the visitor according to the art creation instruction, obtaining art creation interface information; Using the third module in the artificial intelligence model combination to perform color segmentation on the art creation interface information to obtain information of a plurality of color modules; Based on the information of all the color modules and the artistic creation summary, a human-machine co-created artwork is generated.

7. The human-computer interaction method based on asynchronous artistic creation according to claim 1, characterized in that: The generation process of the next interactive task includes: Collect historical interaction results within a preset time period; Automatically generate a current interaction result based on the conversation summary and the human-machine co-created artwork; In response to receiving at least one of the information from the server based on the current interaction result, the human-machine co-created artwork and the historical interaction result, a next interaction task is generated with the assistance of artificial intelligence.

8. The human-computer interaction method based on asynchronous artistic creation according to claim 7, characterized in that: The collecting of historical interaction results within a preset time period includes: Collecting personal information of the visitor; Collecting the number of times the visitor performs the interactive task within the preset time period; Collecting a conversation summary of each interactive task performed by the visitor within the preset time period; Based on the personal information, the times and / or the session summary, a historical interaction result is generated.

9. The human-computer interaction method based on asynchronous artistic creation according to claim 7, characterized in that: The automatically generating a current interaction result based on the conversation summary and the human-machine co-created artwork includes: Collecting original artworks, human-computer co-created artworks and conversation summaries corresponding to several target time nodes in the current interactive task; Based on the original artwork, the human-machine co-created artwork and / or the conversation summary corresponding to all the target time nodes, the current interaction result is automatically generated.

10. The human-computer interaction method based on asynchronous artistic creation according to claim 7, characterized in that: Generate the next interactive task with the help of artificial intelligence, including: In response to receiving the conversation reflection content fed back by the visitor according to the current interaction result, generating optimized conversation principles and optimized example questions; Based on the optimized dialogue principles and the optimized example questions, the next interactive task is generated with the assistance of artificial intelligence.

11. The human-computer interaction method based on asynchronous artistic creation according to claim 1, characterized in that: The multimodal session information includes: At least one of audio data, video data, text data, artistic creation information, physiological information and environmental information.

12. A human-computer interaction system based on asynchronous art creation, characterized in that: The system comprises: The information collection module is used to collect original artworks and multimodal conversation information input by visitors according to the current interactive task in real time; An art creation module, for dynamically analyzing and processing the original artwork and multimodal conversation information using a combination of artificial intelligence models, to generate human-machine co-created artworks and conversation summaries; The interaction module is used to feed back the human-machine co-created artwork and the conversation summary to the server, and in response to receiving the next interaction task generated by the server based on the conversation summary and the human-machine co-created artwork with the assistance of artificial intelligence, feed back the next interaction task to the visitor.

13. A human-computer interaction device, characterized in that: include: A memory for storing executable instructions; A processor is used to call and run the executable instructions in the memory to execute the steps of the human-computer interaction method based on asynchronous artistic creation as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed by a processor, the human-computer interaction method based on asynchronous artistic creation as described in any one of claims 1-11 is implemented.

Images