A virtual diagnosis and treatment system and method

Through the virtual diagnosis and treatment system, using the five-dimensional library construction and adjustment, Blendshape expression controller and other technologies, a virtual character model that meets the needs of patients was generated, solving the problem of long time to build virtual images and difficult to relieve auditory hallucinations in the existing technology, and realizing the problem of long time to build virtual images in the existing technology. Psychological counseling and treatment effects.

CN112133409BActive Publication Date: 2024-11-22XINZHIXIN (HENAN) MEDICAL TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202011065073.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-30
Publication Date
2024-11-22
Estimated Expiration
2040-09-30

AI Technical Summary

Technical Problem

The prior art cannot effectively obtain virtual characters and auditory hallucination language to provide psychological counseling for patients, which makes it difficult to relieve auditory hallucination symptoms, and the virtual image construction time is long and cannot be used for scanning modeling.

Method used

Provide a virtual diagnosis and treatment system, including character portrait generation module, three-dimensional character model building module, situational diagnosis and treatment module, facial capture module, full-body motion capture module and voice changer module, construct and adjust through the five-feature library, Blendshape expression controller and transform action The controller generates a virtual character model that meets the needs of the patient, realizing interactive psychological treatment between the virtual character and the patient.

Benefits of technology

It realizes the rapid and accurate construction of a three-dimensional model that conforms to the patient's virtual image, and psychological counseling is carried out through the auditory sounds and expressions of virtual characters, improving the patient's psychological stress state, and overcoming the problems of manual construction time and unrealistic virtual image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112133409B_ABST
    Figure CN112133409B_ABST
Patent Text Reader

Abstract

The present invention relates to a virtual diagnosis and treatment system and method, which belongs to the technical field of psychological diagnosis and treatment devices, and solves the problem that the existing diagnosis and treatment devices cannot obtain virtual characters that fit the user and auditory hallucination language to provide psychological counseling for patients. The virtual diagnosis and treatment system includes: a character portrait generation module, which selects corresponding facial features from multiple facial feature libraries to assemble a complete character portrait based on the virtual character image described by the patient; a three-dimensional character model establishment module, which makes a Blendshape expression controller and a transform action controller based on the virtual portrait and generates a three-dimensional character model that simulates the virtual character; a scenario diagnosis and treatment module, which performs human-computer interaction between a psychotherapist, a patient and a virtual character model by loading a virtual scene module, a facial capture module, a full-body motion capture module and a voice changer module. It can obtain a three-dimensional character model that fits the user's virtual character image and simultaneously display the auditory hallucination sound and expression action of the virtual character to the patient, so as to achieve psychological counseling and treatment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of psychological diagnosis and treatment devices, and in particular to a virtual diagnosis and treatment system and method. Background Art

[0002] In psychiatry, auditory hallucination is a major perceptual disorder. Auditory hallucination is manifested as a false sense of reality based on the patient's perception without the intervention of any person or external stimulus. 75% of schizophrenia patients are diagnosed with auditory hallucinations. The drugs used to treat schizophrenia symptoms have a pharmacological effect of dopamine receptor blocking, and these drugs can relieve the symptoms of auditory hallucinations. However, clinical practice shows that medical procedures cannot effectively help patients with auditory hallucinations. Half of the auditory hallucinations will develop into chronic diseases. Despite the use of drug treatment, hallucinations will remain for months or even years. These symptoms are often the reason for hospitalization of patients with mental illness, causing patients to be disconnected from society.

[0003] So far, the existing technologies for the diagnosis, treatment and relief of mental illnesses and disorders are far from mature. There is a lack of devices and methods that can provide psychological counseling to patients through virtual characters and auditory hallucinations to improve their mental state. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a virtual diagnosis and treatment system and method to solve the problem that the existing diagnosis and treatment devices cannot obtain a virtual character that fits the user and the auditory hallucination language to provide psychological counseling for the patient.

[0005] On the one hand, an embodiment of the present invention provides a virtual diagnosis and treatment system, including: a character portrait generation module, which is used to select corresponding facial features from multiple facial feature libraries to assemble a complete character portrait according to the virtual character image described by the patient; a three-dimensional character model establishment module, which is used to make a Blendshape expression controller, a transform action controller and generate a three-dimensional character model simulating the virtual character based on the virtual portrait; a scenario diagnosis and treatment module, which is used to perform human-computer interaction between the psychotherapist, the patient and the virtual character model by loading the virtual scene module, the facial capture module, the whole body motion capture module and the voice changer module, wherein the virtual scene module is used to provide different virtual scenes according to the needs of the patient; a facial capture module, which is used to control the facial expression of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture value obtained from the face video; a whole body motion capture module, which is used to control the whole body movement of the three-dimensional character model in real time through the transform action controller according to the skeletal motion quaternion data obtained by multiple sensors; a voice changer module, which is used to perform pitch fundamental frequency voice change control on the original sound to convert it into a phantom sound simulating the voice of the virtual character.

[0006] The beneficial effects of the above technical solution are as follows: a three-dimensional character model that fits the user's virtual character image can be obtained. The psychotherapist controls the facial expressions and movements of the three-dimensional character model through the facial capture module and the full-body motion capture module, converts the original sound into an auditory hallucination sound that simulates the virtual character's voice through the voice changer module, and simultaneously displays the auditory hallucination sound and facial expressions and movements of the virtual character to the patient, thereby achieving psychological counseling and treatment and improving the patient's psychological stress state.

[0007] Based on further improvement of the above system, the virtual diagnosis and treatment system also includes a facial features library construction module for constructing multiple facial features libraries, and the facial features library construction module includes a face shape library construction submodule, a hair library construction submodule, an eyebrow library construction submodule, an eye library construction submodule, a nose library construction submodule and a mouth library construction submodule, wherein the face shape library construction submodule is used to remove other facial features and save the face shape based on multiple character image photos with different facial features in a cutout manner, and use the saved face shape to construct a face shape library with the different facial features; and the hair library construction submodule, the eyebrow library construction submodule, the eye library construction submodule, the nose library construction submodule and the mouth library construction submodule are used to respectively construct a hair library, an eyebrow library, an eye library, a nose library and a mouth library with different corresponding features in a cutout manner based on multiple character image photos with different facial features.

[0008] The beneficial effects of the above technical solution are as follows: the multiple facial feature libraries constructed by the facial feature library construction module can quickly and accurately obtain a virtual portrait that conforms to the virtual character image, thereby overcoming the defects that it takes a long time to manually construct a virtual image and that the virtual image imagined by the patient is a virtual image that does not exist in real life and cannot be scanned and modeled.

[0009] Based on further improvements of the above system, the virtual diagnosis and treatment system also includes a facial features adjustment module, which is used to adjust the facial features in the character portrait to obtain a virtual portrait that conforms to the virtual character image. The facial features adjustment module includes: an X-axis movement module, a Y-axis movement module, an X-axis scaling module and a Y-axis scaling module, wherein the X-axis movement module is used to move and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the X-axis direction according to the virtual character image; the Y-axis movement module is used to move and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the Y-axis direction according to the virtual character image; the X-axis scaling module is used to scale and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the X-axis direction according to the virtual character image; and the Y-axis scaling module is used to scale and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the Y-axis direction according to the virtual character image.

[0010] The beneficial effects of the above technical solution are as follows: by adjusting the facial features in the character portrait, a virtual portrait that conforms to the image of the virtual character is obtained.

[0011] Based on further improvements of the above system, the three-dimensional character model building module further includes: generating a basic model of the three-dimensional character and making the Blendshape expression controller and the transform action controller; based on the face recognition algorithm, analyzing the face area in the virtual portrait; converting the face area into basic measurement parameters of the face, and at the same time cutting out the image of the face area and performing head mapping on the basic model; the Blendshape expression controller receives the basic measurement parameters of the face, and dynamically controls the head shape of the basic model according to the basic measurement parameters of the face to maximize the match of the head features in the virtual portrait; and according to the virtual character described by the patient, controlling the height and shape of the torso and limbs of the basic model through the transform action controller.

[0012] Based on further improvements of the above system, the facial capture module is further configured to: capture the video of the psychotherapist's face in real time through a webcam; generate corresponding marker points at the positions of eyebrows, eyes, pupils, nose and mouth in combination with the facial capture algorithm; track the positions of the corresponding marker points in real time in the video of the psychotherapist's face; convert the positions of the corresponding marker points into facial capture values ​​in real time and store the facial capture values; and control multiple Blendshape control components of the three-dimensional character model in real time based on the facial capture values ​​to simulate the facial expressions of the psychotherapist.

[0013] Based on further improvement of the above system, the Blendshape expression controller includes multiple Blendshape control components, wherein the multiple Blendshape control components are used to respectively control the head height, head width, skull height, skull width, face width, face length, eyebrow height, eyebrow width, eyebrow height, eyebrow width, eye height, nose width, mouth height and mouth width.

[0014] Based on further improvements of the above system, the whole-body motion capture module is further configured as follows: the multiple sensors are respectively set at different positions of the psychotherapist, and the different positions include hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs; when the psychotherapist moves, the multiple sensors respectively sense the skeletal motion data of the different positions in real time and convert them into skeletal motion quaternion data; and according to the skeletal motion quaternion data, the hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs of the three-dimensional character module are controlled in real time through the transform motion controller to simulate the movements of the psychotherapist.

[0015] Based on further improvement of the above system, the virtual diagnosis and treatment system also includes a synchronization module and an output module, wherein the synchronization module is used to synchronize the auditory hallucination sound with the facial expression; and the output module is used to simultaneously display the auditory hallucination sound and the action expression of the virtual character to the patient.

[0016] Based on further improvement of the above system, the virtual diagnosis and treatment system also includes: a scale evaluation module, configured as follows: the patient fills out a psychological evaluation scale before each treatment; the patient's psychological state is evaluated based on the filled-in psychological evaluation scale; and the psychological treatment plan is flexibly adjusted according to the patient's psychological state.

[0017] On the other hand, an embodiment of the present invention provides a virtual diagnosis and treatment method including: according to the virtual character image described by the patient, selecting corresponding facial features from multiple facial feature libraries to assemble a complete character portrait; based on the virtual portrait, making a Blendshape expression controller, a transform action controller and generating a three-dimensional character model simulating the virtual character; by loading a virtual scene, facial capture, full-body motion capture and a voice changer, human-computer interaction is performed between the psychotherapist, the patient and the virtual character model, wherein the virtual scene is configured to provide different virtual scenes according to the needs of the patient; the facial capture is configured to control the facial expression of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture value obtained from the face video; the full-body motion capture is configured to control the whole-body movement of the three-dimensional character model in real time through the transform action controller according to the skeletal motion quaternion data obtained by multiple sensors; the voice changer is configured to perform pitch fundamental frequency voice change control on the original sound to convert it into an auditory hallucination sound simulating the voice of the virtual character.

[0018] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0019] 1. Multiple facial feature libraries constructed by the facial feature library construction module can quickly and accurately obtain a virtual portrait that matches the virtual character image, so as to overcome the defects of long time for manual construction of virtual images and the virtual image imagined by patients is a virtual image that does not exist in real life and cannot be scanned and modeled. Then, a three-dimensional character model that fits the user's virtual character image can be obtained through the virtual portrait.

[0020] 2. Adjust the facial features in the character portrait to obtain a virtual portrait that conforms to the image of the virtual character.

[0021] 3. The psychotherapist controls the facial expressions and movements of the three-dimensional character model through the facial capture module and the full-body motion capture module, and simultaneously displays the virtual character's auditory hallucinations, facial expressions and movements to the patient, thereby achieving psychological counseling and treatment and improving the patient's psychological stress state.

[0022] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0024] Figure 1 4 is a block diagram of a virtual diagnosis and treatment system according to an embodiment of the present invention.

[0025] Figure 2 4 is a schematic diagram of a virtual diagnosis and treatment system according to an embodiment of the present invention.

[0026] Figure 3 4 is a network topology diagram of a virtual diagnosis and treatment system according to an embodiment of the present invention.

[0027] Figure 4 Schematic diagram of the five sense organs library according to an embodiment of the present invention.

[0028] Figure 5 A schematic diagram of selecting corresponding facial features from multiple facial feature libraries to assemble a complete character portrait.

[0029] Figure 6 A schematic diagram for adjusting a character portrait to generate a virtual portrait that conforms to the image of a virtual character.

[0030] Figure 7 A schematic diagram of generating a three-dimensional model from a virtual portrait.

[0031] Figure 8 A schematic diagram of generating facial landmarks on a face.

[0032] Fig. 9 4 is a structural diagram of a voice changing module according to an embodiment of the present invention.

[0033] Fig.10 The virtual scene is a schematic diagram of an office scene.

[0034] Fig.11 The virtual scene is a schematic diagram of a school scene.

[0035] Fig.12 The virtual scene is a schematic diagram of an apartment.

[0036] Fig.13 The virtual scene is a schematic diagram of a street scene.

[0037] Fig.14 This is a patient management interface of a virtual diagnosis and treatment system according to an embodiment of the present invention.

[0038] Fig.15 This is a patient details interface of a virtual diagnosis and treatment system according to an embodiment of the present invention.

[0039] Fig.16 FIG. 4 is a flow chart of a voice changer according to an embodiment of the present invention.

[0040] Fig.17 Flow chart of a virtual diagnosis and treatment method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0042] A specific embodiment of the present invention discloses a virtual diagnosis and treatment system. Figure 1 The virtual diagnosis and treatment system includes: a character portrait generation module 102, which is used to select corresponding facial features from multiple facial feature libraries to assemble a complete character portrait according to the virtual character image described by the patient; a three-dimensional character model establishment module 104, which is used to make a Blendshape expression controller, a transform action controller and generate a three-dimensional character model simulating a virtual character based on the virtual portrait; a scenario diagnosis and treatment module 106, which is used to perform human-computer interaction between a psychotherapist, a patient and a virtual character model by loading a virtual scene module, a facial capture module, a full-body motion capture module and a voice changer module, wherein the virtual scene module 108 is used to provide different virtual scenes according to the patient's needs; a facial capture module 110, which is used to control the facial expression of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture value obtained from the face video; a full-body motion capture module 112, which is used to control the full-body movement of the three-dimensional character model in real time through the transform action controller according to the skeletal motion quaternion data obtained by multiple sensors; a voice changer module 114, which is used to perform pitch fundamental frequency voice change control on the original sound to convert it into a phantom sound simulating the virtual character sound.

[0043] Compared with the prior art, the virtual diagnosis and treatment system provided in this embodiment can obtain a three-dimensional character model that fits the user's virtual character image. The psychotherapist controls the facial expressions and movements of the three-dimensional character model through the facial capture module and the full-body motion capture module, converts the original sound into an auditory hallucination sound that simulates the virtual character's voice through the voice changer module, and simultaneously displays the auditory hallucination sound and facial expressions and movements of the virtual character to the patient, thereby achieving psychological counseling and treatment and improving the patient's psychological stress state.

[0044] In the following, reference will be made to Figures 1 to 13 , the virtual diagnosis and treatment system is described in detail. The virtual diagnosis and treatment system includes: a facial features library construction module, a facial features adjustment module, a character portrait generation module 102, a three-dimensional character model establishment module 104, a scenario diagnosis and treatment module 106, a synchronization module, an output module and a scale evaluation module.

[0045] refer to Figure 4 , the facial features library construction module is used to construct multiple facial features libraries. The facial features library construction module includes a face shape library construction submodule, a hair library construction submodule, an eyebrow library construction submodule, an eye library construction submodule, a nose library construction submodule and a mouth library construction submodule, wherein the face shape library construction submodule is used to remove other facial features and save the face shape based on multiple photos of characters with different facial features by cutting out, and to construct a face shape library with different facial features using the saved face shape. The hair library construction submodule, the eyebrow library construction submodule, the eye library construction submodule, the nose library construction submodule and the mouth library construction submodule are used to respectively construct a hair library, an eyebrow library, an eye library, a nose library and a mouth library with different corresponding features by cutting out based on multiple photos of characters with different facial features.

[0046] refer to Figure 5 The character portrait generation module 102 is used to select corresponding facial features from multiple facial feature libraries according to the virtual character image described by the patient to assemble a complete character portrait.

[0047] refer to Figure 6 The facial features adjustment module adjusts the facial features in the character portrait to obtain a virtual portrait that conforms to the virtual character image. Specifically, the facial features adjustment module includes: an X-axis movement module, a Y-axis movement module, an X-axis scaling module, and a Y-axis scaling module, wherein the X-axis movement module is used to move and adjust the face shape, hair, eyebrows, eyes, nose, and mouth in the character portrait in the X-axis direction according to the virtual character image; the Y-axis movement module is used to move and adjust the face shape, hair, eyebrows, eyes, nose, and mouth in the character portrait in the Y-axis direction according to the virtual character image; the X-axis scaling module is used to scale and adjust the face shape, hair, eyebrows, eyes, nose, and mouth in the character portrait in the X-axis direction according to the virtual character image; and the Y-axis scaling module is used to scale and adjust the face shape, hair, eyebrows, eyes, nose, and mouth in the character portrait in the Y-axis direction according to the virtual character image.

[0048] refer to Figure 7 , the three-dimensional character model building module 104 is used to make a blendshape expression controller, a transform action controller and generate a three-dimensional character model simulating a virtual character based on the virtual portrait. Specifically, the three-dimensional character model building module further includes: generating a basic model of the three-dimensional character and making a blendshape expression controller and a transform action controller; analyzing the face area in the virtual portrait based on the face recognition algorithm; converting the face area into basic measurement parameters of the face, and at the same time cutting out the image of the face area and mapping the head of the basic model (reference Figure 6), for example, facial details such as wrinkles and moles are mapped to the head to enhance the similarity between the virtual portrait and the three-dimensional character; the Blendshape expression controller receives the basic measurement parameters of the human face, and dynamically controls the head shape of the basic model according to the basic measurement parameters of the human face to maximize the matching of the head features in the virtual portrait; and according to the virtual character described by the patient, the height and shape of the body and limbs of the basic model are controlled by the transform action controller. The basic measurement parameters of the multiple faces include head height, head width, skull height, skull width, face width, face length, eyebrow height, eyebrow width, eyebrow height, nose width, mouth height and mouth width. The Blendshape controller includes multiple Blendshape control components, wherein the multiple Blendshape control components are used to control the head height, head width, skull height, skull width, face width, face length, eyebrow height, eyebrow width, eyebrow height, nose width, mouth height and mouth width respectively. In addition, the body and limbs can be controlled accordingly.

[0049] The situational diagnosis and treatment module 106 is used to carry out human-computer interaction between the psychotherapist, the patient and the virtual character model by loading the virtual scene module 108, the facial capture module 110, the whole body motion capture module 112 and the voice changer module 114, wherein the virtual scene module 108 is used to provide different virtual scenes according to the patient's needs; the facial capture module 110 is used to control the facial expressions of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture values ​​obtained from the face video; the whole body motion capture module 112 is used to control the whole body movements of the three-dimensional character model in real time through the transform motion controller according to the skeletal motion quaternion data obtained by multiple sensors; the voice changer module 114 is used to perform pitch fundamental frequency voice change control on the original sound to convert it into an auditory hallucination sound simulating the virtual character's voice.

[0050] For example, the virtual diagnosis and treatment system may have five preset treatment scenes, including: office, school, apartment, street, and default scene. The virtual scene module 108 selects an office according to the patient's needs for different treatment scenes (see Fig.10 )、School(reference Fig.11 ), Apartment (reference Fig.12 ), Street (reference Fig.13 ) and the default scene (virtual space with a solid color background). The treatment scenes can be continuously enriched in the later stage to form a large database of virtual scenes. Considering the limitations of hardware, treatment scenes can be added in a timely manner according to patient needs.

[0051] The facial capture module 110 is further configured to: capture the face of the psychotherapist in real time through a webcam; generate corresponding markers at the positions of eyebrows, eyes, pupils, nose and mouth in combination with a facial capture algorithm (refer to Figure 8 ); in the video of the face of the psychotherapist, real-time tracking of the position of the corresponding marker point; and real-time conversion of the position of the corresponding marker point into a facial capture value and storage of the facial capture value; and real-time control of multiple Blendshape control components of the three-dimensional character model based on the facial capture value to simulate the facial expression of the psychotherapist. The Blendshape expression controller includes multiple Blendshape control components, wherein the multiple Blendshape control components are used to respectively control the head height, head width, skull height, skull width, face width, face length, eyebrow height, eyebrow width, eyebrow height, eyebrow width, eyebrow height, nose width, mouth height and mouth width.

[0052] The whole-body motion capture module 112 is further configured as follows: a plurality of sensors are respectively arranged at different positions of the psychotherapist, including hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs; when the psychotherapist moves, the plurality of sensors respectively sense the skeletal motion data of different positions in real time and convert them into skeletal motion quaternion data; and according to the skeletal motion quaternion data, the hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs of the three-dimensional character module are controlled in real time through the transform motion controller to simulate the movements of the psychotherapist.

[0053] A synchronization module is used to synchronize the auditory hallucination sound with the facial expression; and an output module is used to simultaneously display the auditory hallucination sound and the action expression of the virtual character to the patient. Specifically, the auditory hallucination sound of the virtual character is played to the patient through a speaker, and the action expression of the virtual character is displayed to the patient through a display.

[0054] The scale assessment module is configured as follows: the patient fills in a psychological assessment scale before each treatment; the patient's psychological state is assessed based on the filled-in psychological assessment scale; and the psychological treatment plan is flexibly adjusted according to the patient's psychological state.

[0055] The present invention aims to solve the problem that medical procedures cannot effectively help patients with auditory hallucinations, and systematically intervene in the cognition and behavior of patients through an application program.

[0056] In the following, refer to Figures 2 to 13 , the virtual diagnosis and treatment system is described in detail by means of specific examples.

[0057] The virtual diagnosis and treatment system is a software diagnosis and treatment system based on the basic principles of avatar therapy, combining virtual reality technology and software development technology. Avatar therapy is a new technology that uses virtual reality to treat mental illness. The key to the technology is to simulate the virtual avatar of patients with mental illness through three-dimensional technology, use virtual characters as protagonists, and interact with patients under the real-time performance of professional psychotherapists, thereby alleviating patients' auditory hallucination symptoms and improving their mental state.

[0058] The virtual diagnosis and treatment system is a remote intervention mode that uses a computer interactive interface, clear operating steps, and highly structured multi-media interactive methods (animation, video, sound, three-dimensional modeling of character expressions and movements, etc.) to demonstrate the basic principles and methods of cognitive behavioral therapy. It can systematically intervene in the patient's cognition and behavior through applications.

[0059] The system mainly includes three modules: virtual portrait, scale assessment, and situational diagnosis and treatment. Figure 2 It is a schematic diagram of the virtual diagnosis and treatment system.

[0060] The virtual portrait description is through full communication between the patient and the therapist. With the help of virtual portrait software 3, a character image that matches the patient's auditory hallucination is drawn on the interface, and the corresponding picture is generated and available for download. The portrait software can quickly and accurately simulate the image of the patient's fantasy character by dividing the overall appearance of the person into five senses, selecting 8 typical subcategories for each sense, collecting and photographing typical images of people with the senses in each subcategory, and using technical means to segment the facial features represented by each subcategory of the facial features, and finally arranging and combining them to achieve the purpose of virtual portrait. Reference Figure 4 When operating the software, click on each facial feature in the facial features library to display the facial features on the right canvas. Figure 5 , select the face shape, hair, eyebrows, eyes, mouth, and nose in turn to assemble a complete character portrait. Figure 6 Each facial feature can be adjusted using the adjustment options on the right. Each facial feature can be moved and scaled in the X and Y axis directions to fit a richer character image.

[0061] Scale assessment is a doctor's timely assessment of the patient's psychological state based on the psychological assessment scale filled out by the patient, and the doctor can flexibly adjust the psychological treatment plan according to the patient's psychological state. The psychological assessment scale is built-in to the system. Before each treatment, the psychotherapist can assign a professional psychotherapy assessment scale to the patient. The system has built-in professional scales commonly used in psychotherapy to assess the patient's psychological state. The scale can be added by clicking on the scale in the scale list. The added scales are displayed in the selected scale list area in the order they were added. Click on the icon of a scale, and the specific assessment content of the scale will be displayed on the right. The patient fills in the scale in turn. During the filling process, the software will automatically continue to record in real time in the background according to the score measurement standard of the specific scale, and finally obtain the evaluation score of the scale after the patient completes the filling.

[0062] Situational diagnosis and treatment is achieved by loading virtual scenes 3, facial capture 4, full-body motion capture 5, voice changer 6 and other plug-ins (also known as modules) to achieve human-computer interaction among the therapist, the patient and the virtual character 1 (the patient's auditory hallucination object, that is, the person drawn by the virtual portrait). In the process of interaction, the therapist assumes the roles of the therapist himself and the virtual character 1, and incorporates psychological treatment methods. By integrating traditional psychological treatment methods with modern advanced technologies, auditory hallucinations are treated in order to alleviate the patient's auditory hallucination symptoms.

[0063] The virtual scene plug-in includes 5 preset treatment scenes in the software system (reference Figures 10 to 13 ) and a virtual image character, wherein the virtual image character is based on the virtual image description of the patient obtained by the virtual portrait software 2, and is generated by a one-click generation model method of a photo. The underlying principle of the one-click generation model of a photo is based on a face recognition algorithm. First, the algorithm can analyze the face area in the photo, and then convert it into the basic measurement parameters of the face according to the obtained face area. At the same time, the image of the face area is cut out through a specific mask, and then used as the head map of the generated character to improve the similarity between the virtual portrait of the photo and the three-dimensional character. One-click generation of a three-dimensional character model requires a basic model as a benchmark model. The basic model needs to be bound to the skeleton and the skin, and the face also needs to make a Blendshape animation controller. In addition, an additional head BlendShape control bar needs to be made. These control bars are used to receive the parameter features obtained after the facial recognition algorithm analyzes the picture, such as head width, nose width, eye height, mouth width height, etc., and then dynamically control the head shape of the basic model to match the head features in the portrait picture to the greatest extent. Through the above two steps, a head model similar to the photo model can be quickly generated, and the entire model making process can be controlled within one day.

[0064] The facial capture plug-in uses the Faceware facial capture algorithm. Through a simple WebCamera webcam, connected to a computer via USB, the webcam is used to capture a video of a human face in real time. Combined with the facial capture algorithm, marker points are generated at the eyebrows, eyes, pupils, mouth, nose and other locations. The marker points are tracked in real time and converted into value data, and the data generated by each control point is stored in real time. The numerical transmission of different facial features and muscle expressions obtained by the webcam through the facial capture algorithm is mapped to the corresponding facial Blendshape animation interface of a three-dimensional virtual character with the same structure. The data value of the real face can be captured by the facial capture algorithm to control the value of the Blendshape control bar that has been made in the three-dimensional model in real time, so as to achieve the performance and simulation of real people's facial expressions, and finally realize the function of being driven by real people and showing the performed expressions through virtual characters.

[0065] The full-body motion capture plug-in uses an inertial system to sense the psychotherapist's movements in real time. Specifically, the sensor movements of the psychotherapist wearing multiple sensors are sensed and converted into skeletal motion quaternion data. Specifically, the multiple sensors are configured as follows: multiple sensors are respectively set at different positions of the psychotherapist, including hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs; when the psychotherapist moves, multiple sensors respectively sense skeletal motion data at different positions in real time; and the skeletal motion data is converted into skeletal motion quaternion data and transmitted to the motion drive module. The skeletal motion quaternion data is mapped in real time and transmitted to the Transform controller of the three-dimensional character skeleton to realize the full-body motion capture function.

[0066] refer to Fig. 9The voice changer plug-in changes the therapist's voice to meet the voice requirements of different patients for the avatar. The voice changer interface includes a microphone switch, a pitch adjustment slider, a Formant base frequency adjustment slider, an original voice function, a voice change function, a save voice preset, and a quick preset button. Click the microphone switch to turn the microphone on or off; slide the Pitch tone adjustment slider to change the tone of the sound; slide the Formant base frequency adjustment slider to control the frequency of the sound; click the original sound function button to pop up the therapist's real-time image on the left side of the screen, and turn off the voice changing function of the software, and at the same time turn off the facial capture and full-body capture functions of the software, so that the therapist can intervene in the three-party treatment as a therapist; click the voice changing function button to turn off the real-time image of the therapist on the left, connect the facial capture and full-body capture function modules, and turn on the voice changer at the same time, so that the therapist can intervene in the treatment as the patient's virtual image; click the Save Voice Preset button to save the current voice preset value. The next time you enter the diagnosis and treatment stage, the software will automatically read the preset value and change the voice according to this voice; click the Quick Preset button to change the boy's virtual voice into a girl's, and the girl's virtual voice into a boy's, breaking the gender limitations of the therapist, allowing male therapists to play the female virtual image, and female therapists to play the voice of the male virtual image.

[0067] refer to Fig.16 The voice changer mainly includes: an original sound source sound-emitting submodule, which is used to receive the original sound after modulation and combination; a sound source first receiving submodule: which is used to receive the original sound after modulation and combination, and modulate it into a specific wireless signal frequency band for transmission. The sound source relay submodule is used to receive the radio signal of a specific frequency band transmitted by the first sound source receiving module, decode and modulate it, and then input it into the pitch domain baseband voice changing module; the pitch and baseband voice changing submodule receives the demodulated signal of the sound source relay module, and performs pitch and baseband change processing on the audio according to the input parameters; the pitch and baseband control submodule is used to provide parameter adjustment of pitch and baseband visually, and save the parameter preset and transmit the parameter; the changed sound source output submodule is used to receive the result of the voice changing module and convert the digital signal into an analog signal, and output the changed sound signal through the audio playback device; the three-party diagnosis and treatment switching submodule is used to switch between the changed voice state and the original voice state. Based on the principles of CBT diagnosis and treatment and exposure treatment, the psychotherapist selects the timing of switching between the original voice and the changed voice, and displays the therapist's screen in the original voice stage to provide supportive encouragement to patients with refractory verbal auditory hallucinations; and the script treatment module for refractory verbal auditory hallucinations: based on the clinical treatment theory of refractory verbal auditory hallucinations and the case analysis of the etiology of specific individuals, a diagnosis and treatment script is prepared as a linear guiding principle for the diagnosis and treatment process.

[0068] The voice changer module of the embodiment of the present invention obtains the voice of the psychotherapist in real time and superimposes the modulated sound, and then uses the pitch fundamental frequency voice change control and the processing of the voice change module to finally output the auditory hallucination sound that meets the needs of patients with refractory verbal auditory hallucination. In combination with psychotherapy techniques, the purpose of improving and treating the symptoms of refractory verbal auditory hallucination is achieved.

[0069] The device of the present invention is simple and easy to operate, and opens up new treatment ideas and implementation means for non-drug and non-physical stimulation intervention psychological diagnosis and treatment of refractory verbal auditory hallucinations.

[0070] Specific implementation plan:

[0071] The software interface of the diagnosis and treatment system includes a virtual portrait interface (reference Figure 4 ), patient management interface (reference Fig.14 ), patient details interface (reference Fig.15 ), scale assessment interface, scenario treatment preset interface and data center interface. Figure 2 It is the network topology diagram of the virtual diagnosis and treatment system.

[0072] refer to Figure 4 ,The virtual portrait interface can draw a character image that matches the patient’s auditory hallucination image and generate ,corresponding pictures for download.

[0073] refer to Fig.14 The patient management module can record and save the detailed information of patients, and realize basic functions such as adding, deleting, modifying and searching patients. Through this module, therapists can view all the information of a specific patient at any time, and obtain the number of treatments and stage status of the patient through the interface visualization.

[0074] refer to Fig.15 The patient details interface is responsible for displaying the previously entered patient details, and can also display the patient's treatment record report during each treatment process. It also supports unified management of local file uploads, and can upload other patient diagnosis and treatment reports.

[0075] The scale evaluation interface allows you to select professional scales commonly used in psychotherapy to evaluate the patient's psychological state. Click on the scale in the scale list to add a scale. The added scales are displayed in the selected scale list area in the order they were added. Click on the icon of a scale, and the specific evaluation content of the scale will be displayed on the right. The patient fills in the scale in turn. During the filling process, the software will automatically continue to record in real time in the background according to the score measurement standard of the specific scale, and finally obtain the evaluation score of the scale after the patient completes the filling.

[0076] The scene therapy preset interface includes 5 treatment scenes and virtual image characters preset in the software system. The 5 treatment scenes are office (reference Fig.10)、School(reference Fig.11 ), Apartment (reference Fig.12 ), Street (reference Fig.13 ) and default scenes; the default scene is a virtual space with a solid color background, which is responsible for most scene treatment work. The treatment scenes can be continuously enriched in the later stage, but the limitations of the hardware should be considered and added in due course. The virtual image character is modeled based on the virtual portrait and displayed in the form of picture thumbnails (reference Figure 4 ), which makes it easier for psychotherapists and patients to quickly select treatment characters and treatment scenes through visualization. After selecting the scene and character, you can add a scene and character treatment preset based on the current patient through the Add button. Because the virtual character of a patient in clinical practice may contain multiple characters, this option is set to meet the needs of patients with multiple virtual images. After generating the preset, you can click Start Treatment to enter the scene treatment stage.

[0077] The data center interface is used to display the visual data information of patients. The display content includes the scale score and its curve change chart. It includes the patient's treatment stage, treatment days, treatment duration, user virtual portrait, etc. In addition, in the overall information display interface, you can also see the number of patients registered and added to the current software system, the types of patients' diseases and age distribution charts and other big data statistical information. It provides data support for therapists and software users to analyze the incidence of diseases and age distribution scientifically.

[0078] When using the system, first log in to the system through the account and password on the system login interface, then add patient information through the patient management interface, and finally enter the treatment program. Figure 3 When conducting intervention therapy, it is recommended to conduct the treatment in two separate rooms, with the therapist in one room and the patient in the other. The room where the patient is located is set up with a screen that can display the patient's fantasy avatar image, which is directly driven by the therapist. The treatment process proceeds in the following stages:

[0079] (1) In the first (offline) treatment, the therapist communicates with the patient and accurately describes and portrays the patient's specific behavior and fantasy virtual character to generate the image of the virtual character.

[0080] (2) The second step is to use 3D modeling technology to transform the virtual character into a 3D person and create specific animations and environments.

[0081] (3) In the subsequent intervention treatments, the therapist uses real-time motion and expression capture equipment in conjunction with synthetic speech technology to drive the virtual character to perform corresponding therapeutic behaviors, achieve psychological counseling and treatment, and improve the patient's psychological stress state.

[0082] Beneficial effects brought by the technical solution of the present invention

[0083] The virtual diagnosis and treatment system can ultimately comprehensively regulate the patient's stress state, reduce their negative emotions such as fear, anxiety, and depression, and help them build confidence in fighting the disease. The intervention content is customized and easy to operate. The patient only needs to conduct psychological interactive intervention with the virtual character driven by the therapist. Stress regulation for patients can reduce the patient's stress level and psychological burden to the greatest extent from a professional perspective of mental health.

[0084] When using the system, first log in to the system through the account and password on the system login interface, then add the patient information through the patient management interface, and finally enter the treatment program. When conducting intervention treatment, it is recommended to conduct treatment in two separate rooms, with the therapist in one room and the patient in the other room. The room where the patient is located is equipped with a screen that can display the patient's fantasy avatar image, which is directly driven by the therapist. The treatment process is carried out in the following stages.

[0085] The main task of the first treatment is to make a virtual portrait, that is, to draw the patient's auditory hallucination character on the software interface. In the communication between the therapist and the patient, the characteristics of the patient's auditory hallucination object are understood, and according to the patient's description, the corresponding pictures are called from the facial features library. The combination mode of the facial features pictures is continuously adjusted through the function buttons to generate a virtual character picture that conforms to the patient's auditory hallucination. Finally, the data is saved and can be downloaded. After the virtual portrait is completed, the modeler generates a similar three-dimensional character model based on the virtual portrait.

[0086] After completing the virtual portrait, the therapist can enter the scale assessment module. This module evaluates the patient's current psychological state by selecting a scale from the scale library. Click on the scale assessment to display the existing scales in the scale library, and click on the intended scale to add the scale to the taskbar. The added scales are displayed in the selected scale list area in the order they were added. Click on a scale to display the specific assessment content of the scale on the right. The patient fills in the scale in turn. During the filling process, the software will automatically record the data in real time, and the system background will convert the data according to the preset scale scoring method. After the patient fills in the scale, the evaluation results of the scale can be obtained, and a scale evaluation report will be generated for download and printing.

[0087] After completing the scale assessment, you can enter the patient details interface to see the patient's detailed information, including basic information and completed scale results. In addition, the patient's treatment record report during each treatment process can also be displayed. At the same time, it supports unified management of local file uploads and can upload other diagnosis and treatment reports of patients.

[0088] Referring to the results of the scale assessment and combining the therapist's detailed understanding of the patient, the therapist will develop a scenario-based treatment plan that meets the patient's actual needs. Scenario-based treatment is achieved by loading virtual scenes, facial capture, full-body motion capture, voice changer and other plug-ins to achieve human-computer interaction between the therapist, the patient and the avatar. The scenario-based treatment interface includes rich treatment scenes and virtual image characters preset in the software system. The treatment scenes include offices, schools, apartments, streets, etc. The virtual image characters are modeled based on virtual portraits and displayed in the form of picture thumbnails, which facilitates psychotherapists and patients to quickly select treatment characters and treatment scenes through visual means. After selecting the scene and character, you can add a scene and character treatment preset based on the current patient through the Add button. Because the virtual character of a patient in the clinic may contain multiple characters, this option is set to meet the needs of patients with multiple virtual images. After generating the preset, the patient can choose the web PC or VR end according to his or her preferences, click Start Treatment, and enter the scenario treatment stage. During the diagnosis and treatment process, the therapist assumes two roles: the therapist himself and the avatar. He can drive the virtual model in real time. By integrating traditional psychotherapy methods with modern advanced technologies, he works on symptoms such as auditory hallucinations and delusions in order to alleviate the patient's psychotic symptoms and improve his physical and mental state.

[0089] In addition, the data center module is used to intuitively display the patient's visual data information, including: virtual portraits, scale assessment scores and their changing trends, completed treatment stages, treatment days and treatment duration, etc. In the overall information display interface, you can also see the number of patients registered and added to the current software system, the types of patients' diseases and age distribution charts and other big data statistical information. It provides data support for therapists and software users to analyze the incidence of diseases and age distribution scientifically.

[0090] After each treatment is completed, click the "Exit" button to exit the system.

[0091] Another specific embodiment of the present invention discloses a virtual diagnosis and treatment method. Fig.17A virtual diagnosis and treatment method includes: step S1702, selecting corresponding facial features from multiple facial feature libraries to assemble a complete character portrait according to the virtual character image described by the patient; step S1704, making a Blendshape expression controller and a transform motion controller based on the virtual portrait and generating a three-dimensional character model simulating the virtual character; step S1706, performing human-computer interaction between a psychotherapist, a patient and a virtual character model by loading a virtual scene, facial capture, full-body motion capture and a voice changer, wherein the virtual scene is configured to provide different virtual scenes according to the patient's needs; the facial capture is configured to control the facial expression of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture value obtained from the face video; the full-body motion capture is configured to control the whole-body movement of the three-dimensional character model in real time through the transform motion controller according to the skeletal motion quaternion data obtained by multiple sensors; the voice changer is configured to perform pitch fundamental frequency voice change control on the original sound to convert it into an auditory hallucination sound simulating the virtual character sound.

[0092] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0093] 1. Multiple facial feature libraries constructed by the facial feature library construction module can quickly and accurately obtain a virtual portrait that matches the virtual character image, so as to overcome the defects of long time for manual construction of virtual images and the virtual image imagined by patients is a virtual image that does not exist in real life and cannot be scanned and modeled. Then, a three-dimensional character model that fits the user's virtual character image can be obtained through the virtual portrait.

[0094] 2. Adjust the facial features in the character portrait to obtain a virtual portrait that conforms to the image of the virtual character.

[0095] 3. The psychotherapist controls the facial expressions and movements of the three-dimensional character model through the facial capture module and the full-body motion capture module, and simultaneously displays the virtual character's auditory hallucinations, facial expressions and movements to the patient, thereby achieving psychological counseling and treatment and improving the patient's psychological stress state.

[0096] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0097] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A virtual diagnosis and treatment system, characterized in that: include: The facial features library construction module, facial features adjustment module, character portrait generation module, three-dimensional character model establishment module, scenario diagnosis and treatment module, synchronization module, output module and scale evaluation module; The character portrait generation module is used to select corresponding facial features from multiple facial feature libraries according to the virtual character image described by the patient to assemble a complete character portrait; The three-dimensional character model building module is used to make a Blendshape expression controller and a transform action controller based on a virtual portrait and generate a three-dimensional character model simulating the virtual character, wherein the virtual portrait is a character portrait that conforms to the patient's auditory hallucination image; The scenario diagnosis and treatment module is used to perform human-computer interaction between the psychotherapist, the patient and the virtual character model by loading a virtual scene module, a facial capture module, a full-body motion capture module and a voice changer module, wherein the virtual scene module is used to provide different virtual scenes according to the patient's needs; the facial capture module is used to control the facial expression of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture value obtained from the face video; the full-body motion capture module is used to control the full-body movement of the three-dimensional character model in real time through the transform motion controller according to the skeletal motion quaternion data obtained by multiple sensors, wherein the full-body motion capture module further The configuration is as follows: the multiple sensors are respectively arranged at different positions of the psychotherapist, the different positions including hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs; when the psychotherapist moves, the multiple sensors respectively sense the skeletal motion data of the different positions in real time and convert them into skeletal motion quaternion data; and according to the skeletal motion quaternion data, the hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs of the three-dimensional character module are controlled in real time through the transform motion controller to simulate the movements of the psychotherapist; a voice changer module is used to perform pitch fundamental frequency voice change control on the original sound to convert it into an auditory hallucination sound simulating the voice of the virtual character; The synchronization module is used to synchronize the auditory hallucination sound and the facial expression of the three-dimensional character model; The output module is used to simultaneously display the phantom sounds and whole body movements of the three-dimensional character model to the patient; The facial features library construction module is used to construct multiple facial features libraries, and the facial features library construction module includes a face shape library construction submodule, a hair library construction submodule, an eyebrow library construction submodule, an eye library construction submodule, a nose library construction submodule and a mouth library construction submodule, wherein the face shape library construction submodule is used to remove other facial features in a cutout manner to save the face shape based on multiple character image photos with different facial features, and to construct a face shape library with the different facial features using the saved face shape; and the hair library construction submodule, the eyebrow library construction submodule, the eye library construction submodule, the nose library construction submodule and the mouth library construction submodule are used to respectively construct a hair library, an eyebrow library, an eye library, a nose library and a mouth library with different corresponding features in a cutout manner based on multiple character image photos with different facial features; The facial features adjustment module is used to adjust the facial features in the character portrait to obtain a virtual portrait that conforms to the virtual character image. The facial features adjustment module includes: an X-axis movement module, a Y-axis movement module, an X-axis scaling module and a Y-axis scaling module, wherein the X-axis movement module is used to move and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the X-axis direction according to the virtual character image; the Y-axis movement module is used to move and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the Y-axis direction according to the virtual character image; the X-axis scaling module is used to scale and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the X-axis direction according to the virtual character image; and the Y-axis scaling module is used to scale and adjust the face shape, hair, eyebrows, eyes, nose and mouth in the character portrait in the Y-axis direction according to the virtual character image; The three-dimensional character model building module further includes: generating a basic model of the three-dimensional character and making the Blendshape expression controller and the transform action controller; analyzing the face area in the virtual portrait based on the face recognition algorithm; converting the face area into basic measurement parameters of the face, and at the same time cutting out the image of the face area and performing head mapping on the basic model; the Blendshape expression controller receives the basic measurement parameters of the face, and dynamically controls the head shape of the basic model according to the basic measurement parameters of the face to maximize the match of the head features in the virtual portrait; and controlling the height, body and limb shapes of the basic model through the transform action controller according to the virtual character described by the patient.

2. The virtual diagnosis and treatment system according to claim 1, characterized in that: The facial capture module is further configured to: capture a video of the psychotherapist's face in real time through a webcam; generate corresponding marker points at the positions of eyebrows, eyes, pupils, nose and mouth in combination with the facial capture algorithm; track the positions of the corresponding marker points in real time in the video of the psychotherapist's face; convert the positions of the corresponding marker points into facial capture values ​​in real time and store the facial capture values; and control multiple Blendshape control components of the three-dimensional character model in real time based on the facial capture values ​​to simulate the facial expressions of the psychotherapist.

3. The virtual diagnosis and treatment system according to claim 2, characterized in that: The Blendshape expression controller includes multiple Blendshape control components, wherein the multiple Blendshape control components are used to respectively control head height, head width, skull height, skull width, face width, face length, eyebrow height, eyebrow width, eyebrow height, eyebrow width, eye height, nose width, mouth height and mouth width.

4. The virtual diagnosis and treatment system according to claim 1, characterized in that: Also includes: A scale assessment module is configured as follows: the patient fills in a psychological assessment scale before each treatment; The psychological state of the patient is assessed according to the filled-in psychological assessment scale; and the psychological treatment plan is flexibly adjusted according to the psychological state of the patient.

5. A virtual diagnosis and treatment method implemented based on the virtual diagnosis and treatment system according to any one of claims 1 to 4, characterized in that: include: According to the patient's description of the virtual character image, the corresponding facial features are selected from multiple facial feature libraries to assemble a complete character portrait; Based on the virtual portrait, a Blendshape expression controller and a transform motion controller are produced and a three-dimensional character model simulating the virtual character is generated, wherein the virtual portrait is a character portrait that conforms to the patient's auditory hallucination image; by loading a virtual scene, facial capture, full-body motion capture and a voice changer, human-computer interaction is performed between the psychotherapist, the patient and the virtual character model, wherein the virtual scene is configured to provide different virtual scenes according to the needs of the patient; the facial capture is configured to control the facial expression of the three-dimensional character model in real time through the Blendshape expression controller according to the facial capture values ​​obtained from the face video; the full-body motion capture is configured to control the full-body motion of the three-dimensional character model in real time through the transform motion controller according to the skeletal motion quaternion data obtained by multiple sensors, wherein the skeletal motion quaternion data obtained by multiple sensors is obtained through the transform motion controller The controller controls the whole body movements of the three-dimensional character model in real time, further comprising setting the multiple sensors at different positions of the psychotherapist, the different positions including hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs; when the psychotherapist moves, the multiple sensors respectively sense the skeletal motion data of the different positions in real time and convert them into skeletal motion quaternion data; and according to the skeletal motion quaternion data, the hands, forearms, upper arms, shoulders, head, back, chest, feet, calves and thighs of the three-dimensional character module are controlled in real time through the transform motion controller to simulate the movements of the psychotherapist; the voice changer is configured to perform pitch fundamental frequency voice change control on the original sound to convert it into an auditory sound simulating the voice of the virtual character; the auditory sound of the three-dimensional character model is synchronized with the facial expression; and the auditory sound and whole body movements of the three-dimensional character model are simultaneously displayed to the patient.

Citation Information

Patent Citations

  • Real-time virtual idol live-broadcasting method and system

    CN107197385A

  • Three-dimensional interactive teaching system based on virtual reality technology

    CN107248342A

  • Virtual reality (VR) mirror image therapeutic instrument

    CN108648796A

  • Interaction method, device, server, storage medium and sand table treatment system

    CN109584992A

  • System and method for producing three-dimensional moving picture authoring tool supporting synthesis of motion, facial expression, lip synchronizing and lip synchronized voice of three-dimensional character

    US20020024519A1