A virtual human response interactive operation method

Abstract

The invention discloses a virtual person answering interactive operation method, comprising the steps of creating a front-end virtual person; the front-end virtual person recognizing a user's questioning gesture; the front-end virtual person receiving the questioning voice, converting it into text, and feeding back the questioning text content; the front-end virtual person recognizing the user's question ending gesture; the front-end virtual person searching for an answer according to the questioning text content, and displaying the answer content, and controlling the user's voice question recognition through the questioning gesture and the question ending gesture, which can greatly reduce the influence of environmental noise on voice recognition, thereby improving the accuracy of voice question recognition.

Description

[Technical field]

[0001] The invention relates to a virtual human response interactive operation method. [Background Technology]

[0002] Network security refers to the protection of the hardware, software and the data they carry in the network system to prevent them from being destroyed, modified or leaked due to accidental events or malicious acts, ensuring the continuous, stable and reliable operation of the system and maintaining the uninterrupted operation of network services. It mainly includes network equipment security, network information security, network software security, application security, information security and operational security.

[0003] In today's society, network information is developing rapidly, but network users' awareness of network security is relatively weak. Therefore, in the process of using the Internet, it is easy to cause personal information leakage, system infection and data loss, network fraud and financial loss, privacy invasion, account takeover, network infringement and copyright issues, social engineering attacks, and assistance to cybercrime. Therefore, it is urgent to publicize network security knowledge to network users and answer network security questions, so as to improve network users' awareness of network security.

[0004] Existing questions about network security policies and common sense are generally communicated and answered through interactive systems. Generally, questions about network security are input into the system in the form of voice. The system generally determines whether the voice question is finished by whether there is voice input. However, when performing voice recognition in a noisy environment, environmental noise is also easily recognized by the system. Therefore, it is difficult for the system to judge whether the voice question is finished, and it is easy to recognize environmental noise into the user's question, resulting in inaccurate question recognition and inaccurate feedback answers. At the same time, the operating interface of the existing network security interactive system is generally a touch-button interactive control interface, and the interface layout is monotonous and lacks interest. [Summary of the invention]

[0005] The present invention overcomes the shortcomings of the prior art and provides a virtual human response interaction operation method, which is applied to network security knowledge interaction, improves the accuracy of voice question recognition and the fun of human-computer interaction, and better realizes network security knowledge promotion and network security problem answering.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A virtual human response interactive operation method, characterized in that:

[0008] S1. Create a front-end virtual person;

[0009] S2, the front-end virtual human recognizes the user's question gesture;

[0010] S3, the front-end virtual person receives the voice of the question, converts it into text, and feeds back the text content of the question;

[0011] S4, the front-end virtual person recognizes the user's gesture to end the question;

[0012] S5. The front-end virtual person searches for the answer based on the text content of the question and displays the answer content.

[0013] The above-mentioned method for responding to interactive operations of a virtual human is characterized in that: the front-end virtual human in S1 is subjected to 3D modeling, rendering and synthesis processing through 3dmax, is subjected to texture drawing processing through Substance 3D Painter, is subjected to texture tone and texture modification processing through PS, is subjected to character model sculpting processing through Zbrush, is subjected to model development 2D UV processing through RizomUV, is subjected to character model preset processing through DAZ, is subjected to 3D game development processing through UE5, and is subjected to character bone binding and action adjustment processing through Maya.

[0014] The above-mentioned virtual human response interactive operation method is characterized in that: in S2 and S4, user gestures are captured and recognized through hand_action_server.

[0015] The above-mentioned virtual human answering interactive operation method is characterized in that: the questioning gesture in S2 is a gesture of opening the thumb, index finger and middle finger, and bending the ring finger and little finger; the ending questioning gesture in S4 is a gesture of clenching a fist.

[0016] The above-mentioned virtual human response interactive operation method is characterized in that: in S3, the question voice is converted into text through the STT service of the speech synthesis server, and the speech synthesis server and the front-end virtual human communicate through WebSocker.

[0017] A virtual human response interactive operation method as described above is characterized by: when the query answer in S5 is a text answer, the text is converted into voice, and the voice is played while controlling the front-end virtual human to make mouth movements that match the pronunciation of the text; when the query answer in S5 is a voice answer, the voice is played while controlling the front-end virtual human to make mouth movements that match the pronunciation of the text; when the query answer in S5 is a picture answer, the picture is displayed near the front-end virtual human; when the query answer in S5 is a video answer, the video is played near the front-end virtual human.

[0018] The above-mentioned virtual human response interactive operation method is characterized by: in S5, the GPL-LLM large language model of the AI ​​server is used to perform natural language processing of the question content, and the AI ​​server and the front-end virtual human query the answer in REST mode; in S5, the text is converted into speech through the TTS service of the speech synthesis server, and the speech synthesis server and the front-end virtual human communicate in WebSocker mode; the front-end virtual human's lip-sync training is performed through Wav2Lip of the virtual human action server, and the virtual human action server and the front-end virtual human communicate in REST mode.

[0019] The above-mentioned virtual human response interactive operation method is characterized by: further comprising S6, the front-end virtual human recognizes the user's page turning gesture, and turns the page of the displayed query answer according to the recognized page turning gesture.

[0020] The virtual human response interactive operation method as described above is characterized in that: in S6, user gestures are captured and recognized through hand_action_server, and the user's page turning gestures include a page turning gesture for turning to the first page, a page turning gesture for turning to the second page, a page turning gesture for turning to the third page, and a page turning gesture for turning to the fourth page.

[0021] The above-mentioned virtual human response interactive operation method is characterized in that: the page-turning gesture of turning to the first page is to extend the index finger, the page-turning gesture of turning to the second page is to extend the index finger and middle finger, the page-turning gesture of turning to the third page is to extend the index finger, middle finger and ring finger, and the page-turning gesture of turning to the fourth page is to extend the index finger, middle finger, ring finger and little finger.

[0022] The beneficial effects of the present invention are:

[0023] The present invention arranges a front-end virtual person to interact with network users. The front-end virtual person has a variety of anthropomorphic expressions and actions, which can greatly improve the interest of the interaction; at the same time, the user's voice question recognition is controlled through questioning gestures and ending questioning gestures, which can greatly reduce the impact of environmental noise on voice recognition, thereby improving the accuracy of voice question recognition; the user can also turn the page to view the displayed answers through the page turning gesture, further improving the interest of the interaction. [Drawings]

[0024] Figure 1 This is a schematic diagram of the response timing of the present invention;

[0025] Figure 2 Schematic diagram of various guiding gestures for users of the present invention. [Specific implementation method]

[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0027] It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly. In addition, the descriptions of "preferred", "sub-preferred", etc. in the present invention are only used for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "preferred" or "sub-preferred" may explicitly or implicitly include at least one such feature.

[0028] like Figure 1-2 As shown, a virtual human response interactive operation method includes:

[0029] S1. Create a front-end virtual person on the system of the network security interactive device, and the front-end virtual person is displayed on the display screen of the network security interactive device to interact with the user;

[0030] Specifically, 3d modeling, rendering and synthesis are performed through 3dmax, texture drawing is performed through Substance 3DPainter, texture tone and texture are modified through PS, character model sculpting is performed through Zbrush, model development 2D UV is performed through RizomUV, character model preset is performed through DAZ, 3D game development is performed through UE5, character bone binding and motion adjustment are performed through Maya, and the front-end virtual human is generated after integrated processing.

[0031] S2. The user makes a questioning gesture in front of the network security interactive device, the camera records the user's gesture, and the front-end virtual person recognizes the user's questioning gesture;

[0032] Specifically, the user gestures are captured and recognized through hand_action_server, and the questioning gesture is a gesture of opening the thumb, index finger and middle finger, and bending the ring finger and little finger.

[0033] S3, the user sends a voice question to the network security interactive device through voice, and the front-end virtual person receives the voice question, converts it into text, and feedbacks the text content of the question;

[0034] Specifically, the STT service of the speech synthesis server converts the question voice into text, and then feeds the converted question text content back to the network security interaction device, and displays the question text content through the front-end virtual person. The speech synthesis server and the front-end virtual person communicate through WebSocket.

[0035] S4. After the user finishes asking the question, he / she makes a gesture to end the question in front of the network security interactive device, and the front-end virtual person recognizes the gesture of the user to end the question;

[0036] Specifically, the user's gestures are captured and recognized through hand_action_server, and the ending question gesture is a fist-clenching gesture. By setting the questioning gesture and the ending question gesture to recognize the user's voice question, the impact of environmental noise on voice recognition can be reduced, thereby improving the accuracy of voice question recognition.

[0037] S5. The front-end virtual person searches for the answer based on the text content of the question and displays the answer content.

[0038] Specifically, the front-end virtual person uses the GPL-LLM large language model of the AI ​​server to perform natural language processing on the question content, extract keywords, and then query the answer based on the keywords on the AI ​​server, and feed the query answer back to the front-end virtual person. The AI ​​server and the front-end virtual person query the answer in REST mode;

[0039] Subsequently, the text in the query answer fed back to the front-end virtual person is converted into speech through the TTS service of the speech synthesis server and played through the network security interaction device. Among them, the speech synthesis server and the front-end virtual person communicate through the WebSocket method.

[0040] In the process of converting the text in the query answer into speech, the front-end virtual human is trained for lip-syncing through Wav2Lip of the virtual human action server, and the front-end virtual human is controlled to make corresponding mouth pronunciation movements according to the played speech, so that the front-end virtual human simulates pronunciation and interacts with the user. The virtual human action server and the front-end virtual human communicate in REST mode.

[0041] In this case, the query answer includes text answer, voice answer, picture answer or video answer. Among them, when the query answer is a text answer, the text is converted into voice, and the voice is played while controlling the front-end virtual person to make mouth movements that match the text pronunciation; when the query answer is a voice answer, the voice is played while controlling the front-end virtual person to make mouth movements that match the text pronunciation; when the query answer is a picture answer, the picture is displayed near the front-end virtual person on the display screen of the network security interaction device; when the query answer is a video answer, the video is played near the front-end virtual person on the display screen of the network security interaction device. By setting the front-end virtual person to interact with the network user and allowing the front-end virtual person to simulate pronunciation and feedback the query answer through the mouth, the fun of the interaction is greatly improved.

[0042] This case also includes

[0043] S6. When displaying the query answer, one or more of text answer, voice answer, picture answer and video answer can be displayed. Therefore, when the front-end virtual person displays the query answer in multiple categories, or the query answer of the same category contains multiple pages, all the query answers are arranged by page number and broadcast page by page. At this time, when the user needs to quickly turn to a specific page, he makes a page-turning gesture in front of the network security interaction device to turn the content page. The front-end virtual person recognizes the user's page-turning gesture and turns the displayed query answer to the corresponding page according to the recognized page-turning gesture;

[0044] Specifically, the user gestures are captured and recognized through hand_action_server, and the user page-turning gestures include page-turning gestures for turning to the first page, page-turning gestures for turning to the second page, page-turning gestures for turning to the third page, and page-turning gestures for turning to the fourth page. After the user's page-turning gesture is recognized, the query answer of the corresponding page is turned and displayed according to the page-turning gesture. Among them, the page-turning gesture for turning to the first page is a gesture of extending the index finger, the page-turning gesture for turning to the second page is a gesture of extending the index finger and the middle finger, the page-turning gesture for turning to the third page is a gesture of extending the index finger, the middle finger and the ring finger, and the page-turning gesture for turning to the fourth page is a gesture of extending the index finger, the middle finger, the ring finger and the little finger. The displayed answers are turned to a specific page for the user to view through the user's page-turning gesture, further improving the interactive fun.

[0045] In this case, the front-end virtual person is designed to be composed of a head, a torso, limbs and clothing; wherein, the face of the head is provided with expression muscles for simulating real-life expressions, and the expression muscles are arranged according to the structure of the human face. The facial muscles include the frontalis, orbicularis oculi, corrugator supercilii, depressor supercilii, nasal muscle, depressor septum nasalis, anterior auricular muscle, superior auricular muscle, posterior auricular muscle, transverse mentalis, risorius, zygomatic major, zygomatic minor, orbicularis oris, depressor anguli oris, levator labii superioris, levator labii ala nasalis superioris, depressor labii inferioris, levator anguli oris, buccinator and mentalis. At the same time, the network security interactive device presets the stretching states of various facial muscles when people are in emotional states such as joy, anger, sadness, laughter, seriousness and solemnity, and pairs different front-end virtual human expressions with different query answers, so that after the front-end virtual human obtains the query answer, it controls the corresponding facial muscles to simulate real-person expressions, and can intuitively express the nature of the query answer to the user who asks the question at the first time. At the same time, it cooperates with the front-end virtual human mouth to simulate the voice pronunciation action, so as to realize the simulation of communication and interaction between real people, thereby improving the fun of interaction and inducing the user's interest in browsing the query answer.

[0046] In this case, different actions and expressions can also be set at different working stages of the front-end virtual person. For example, when recognizing a user asking a question, the front-end virtual person's head, trunk and limbs cooperate to make a sideways movement with ears tilted and head forward, and put one hand in a trumpet-shaped listening posture next to the ear, and control all facial muscles to make a patient listening expression. For example, when recognizing that the user has finished asking a question and is in the process of querying the answer, the front-end simulated person's head, trunk and limbs cooperate to make a standing at attention movement, and put one hand in a V-shaped thinking posture under the chin, and control all facial muscles to make a thinking expression. For example, when displaying a picture to query the answer or playing a video to query the answer, the front-end simulated person's head, trunk and limbs cooperate to make an upward eight-shaped movement with both hands raised, and the query answer is broadcast between the two hands, and control all facial muscles to make a laughing expression. By matching different actions and expressions at different working stages of the front-end virtual person, the interactive fun is improved and the user's interest in asking questions and querying answers is induced, which facilitates the promotion and popularization of network security policies and knowledge.

[0047] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A virtual human response interactive operation method, characterized in that: Included S1. Create a front-end virtual person; S2, the front-end virtual human recognizes the user's question gesture; S3, the front-end virtual person receives the voice of the question, converts it into text, and feeds back the text content of the question; S4, the front-end virtual person recognizes the user's gesture to end the question; S5. The front-end virtual person searches for answers based on the text of the question and displays the answers; S6, the front-end virtual person recognizes the user's page-turning gesture, and turns the page of the displayed query answer according to the recognized page-turning gesture; In S2, S4 and S6, user gestures are captured and recognized through hand_action_server; the questioning gesture in S2 is a gesture of opening the thumb, index finger and middle finger, and bending the ring finger and pinky finger; the ending questioning gesture in S4 is a gesture of clenching a fist; the page turning gestures of the user in S6 include a page turning gesture of turning to the first page, a page turning gesture of turning to the second page, a page turning gesture of turning to the third page and a page turning gesture of turning to the fourth page, the page turning gesture of turning to the first page is a gesture of extending the index finger, the page turning gesture of turning to the second page is a gesture of extending the index finger and middle finger, the page turning gesture of turning to the third page is a gesture of extending the index finger, middle finger and ring finger, and the page turning gesture of turning to the fourth page is a gesture of extending the index finger, middle finger, ring finger and pinky finger; When the query answer in S5 is a text answer, the text is converted into voice, the voice is played and the front-end virtual person is controlled to make mouth movements that match the pronunciation of the text; when the query answer in S5 is a voice answer, the voice is played and the front-end virtual person is controlled to make mouth movements that match the pronunciation of the text; when the query answer in S5 is a picture answer, the picture is displayed near the front-end virtual person; when the query answer in S5 is a video answer, the video is played near the front-end virtual person; In S5, the GPL-LLM large language model of the AI ​​server is used to process the natural language of the question content, and the AI ​​server and the front-end virtual human use REST to query the answer; in S5, the TTS service of the speech synthesis server is used to convert text into speech, and the speech synthesis server and the front-end virtual human communicate through WebSocker; the Wav2Lip of the virtual human action server is used to perform lip-sync training on the front-end virtual human, and the virtual human action server and the front-end virtual human communicate through REST; The front-end virtual person is designed to be composed of a head, a trunk, limbs and clothing; wherein, expression muscles for simulating real-life expressions are provided on the face of the head, and the expression muscles are arranged according to the structure of the human face, and the expression muscles include frontalis, orbicularis oculi, corrugator supercilii, depressor supercilii, nasal muscle, depressor septum nasalis, anterior auricular muscle, superior auricular muscle, posterior auricular muscle, transverse mentalis, risorius, zygomaticus major, zygomaticus minor, orbicularis oris, depressor anguli oris, levator labii superioris, levator labii superioris nasalis alar muscle, depressor labii inferioris, levator anguli oris, buccinator and mentalis, and at the same time, the stretching states of the expression muscles of people in various emotional states of joy, anger, sadness, happy laughter, seriousness and solemnity are preset in the network security interactive device, and different expressions of the front-end virtual person are matched with different query answers, so that after the front-end virtual person obtains the query answer, the corresponding expression muscles are controlled to simulate real-life expressions, and at the same time, the front-end virtual person's mouth simulates speech pronunciation movements; Different actions and expressions are set in different working stages of the front-end virtual person. When recognizing the user's question, the front-end virtual person's head, torso and limbs cooperate to make the movement of leaning sideways, tilting ears and stretching the head forward, and puts one hand to the ear in a trumpet-shaped listening posture, and controls all facial muscles to make a patient listening expression; when recognizing the user's end of asking the question and in the process of querying the answer, the front-end simulated person's head, torso and limbs cooperate to make the movement of standing at attention, and puts one hand in a V-shaped thinking posture on the chin, and controls all facial muscles to make a thinking expression; when displaying pictures to query the answer or playing videos to query the answer, the front-end simulated person's head, torso and limbs cooperate to make the movement of raising both hands upward in an "eight" shape, and broadcasts the query answer between the two hands, and controls all facial muscles to make a laughing expression.

2. A virtual human response interactive operation method according to claim 1, characterized in that: In S1, the front-end virtual human is processed through 3dmax for 3D modeling, rendering and synthesis, Substance 3D Painter for texture painting, PS for modifying texture tone and texture, Zbrush for character model sculpting, RizomUV for model development 2D UV, DAZ for character model preset, UE5 for 3D game development, and Maya for character bone binding and action adjustment.

3. A virtual human response interactive operation method according to claim 1, characterized in that: In S3, the STT service of the speech synthesis server converts the question voice into text, and the speech synthesis server and the front-end virtual person communicate through WebSocket.

Images