Session interaction method and system and intelligent wearable device

Abstract

The embodiment of the invention provides a session interaction method and system and intelligent wearable equipment, and the method comprises the steps: collecting first visual angle data of a first user through first intelligent wearable equipment, collecting an eye image of the first user, and extracting the eye feature information; generating first virtual character data according to a pre-acquired original image containing the face of the first user and the eye feature information, so as to obtain third view angle data of the first user according to the first virtual character data; the first intelligent wearable device receives second visual angle data of the first user; forming display of a first session interface based on the first view angle data, the second view angle data and the third view angle data of the first user; and the first intelligent wearable device sends the first virtual character data and / or the first visual angle data of the first user to the second intelligent wearable device. According to the invention, the first virtual character with the face and expression of the first user is generated, and the real expression and action of the user can be reflected more truly.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a conversation interaction method, system and smart wearable device. Background Art

[0002] With the rapid development of wearable technology and virtual reality (VR) technology, smart wearable devices have become an indispensable part of people's daily life and work. Head-mounted smart wearable devices, such as smart glasses and VR helmets, not only provide convenient information acquisition and entertainment functions, but also show great potential in remote collaboration, education and training, medical health and other fields. In related technologies, conversational interaction methods mainly rely on text and voice, lacking intuitive visual feedback and emotional expression. This may lead to invalid communication results or low credibility. For example, in a remote meeting, participants cannot judge other people's approval or disapproval by facial expressions, which may lead to misunderstandings and communication barriers. When wearing head-mounted smart wearable devices for interaction, users usually cannot capture their complete facial expressions and demeanor. Since they cannot capture their complete facial information, the two parties in the interaction cannot judge each other's emotions and intentions by facial expressions and demeanor, resulting in a significant reduction in communication effectiveness. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present disclosure is to provide a conversation interaction method, system and smart wearable device to solve the problems in the related technology.

[0004] A first aspect of the present disclosure provides a conversation interaction method, wherein the method is applied to a conversation interaction between a first user using a first head-mounted smart wearable device and a second user using a second head-mounted smart wearable device that communicates and interacts with the first head-mounted wearable device;

[0005] The conversation interaction method comprises:

[0006] Collecting the first perspective data of the first user through the first smart wearable device, collecting the eye image of the first user and extracting eye feature information;

[0007] The first smart wearable device generates first virtual character data based on a pre-set modeling model, according to a pre-acquired original image containing the face of the first user and the eye feature information, so as to obtain third-viewing angle data of the first user according to the first virtual character data; the third-viewing angle data is used to display a first virtual character having the face and facial expression of the first user; the facial expression is generated based on the eye feature information and changes accordingly;

[0008] The first smart wearable device receives second perspective data of a first user, where the second perspective data includes second virtual character data of the second user and / or first perspective data of the second user;

[0009] The first smart wearable device forms a display of a first conversation interface based on the first perspective data, the second perspective data, and the third perspective data of the first user, wherein the first conversation interface includes: a first display area for displaying the second virtual character, a second display area for displaying the first virtual character, and a first perspective image for displaying the second virtual character in the first display area or in an area other than the first display area and the second display area, wherein the first perspective image is generated based on the first perspective data;

[0010] The first smart wearable device sends the first virtual character data and / or the first perspective data of the first user to the second smart wearable device, so as to form a session on the second smart wearable device and display a second session interface coordinated with the first session interface.

[0011] In an embodiment of the first aspect, generating the first virtual character data according to the pre-acquired original image containing the first user and the eye feature information further includes:

[0012] Collecting the user's voice information in real time through the first smart wearable device;

[0013] The smart wearable device generates a mouth expression included in the third-view data based on the modeling model and the voice information; the mouth expression is generated based on the voice information and changes accordingly.

[0014] In an embodiment of the first aspect, the first smart wearable device generates first virtual character data based on a preset modeling model and according to a pre-acquired original image including a face of the first user and the eye feature information, including:

[0015] Detecting a first eye key point in the original image, and determining an eye area according to the first eye key point;

[0016] A second eye key point in the eye feature information is extracted, and the second eye key point is matched with the eye region, so that the eye region changes accordingly according to the second eye key point.

[0017] In an embodiment of the first aspect, the expression information also includes other facial expressions that match the simulated eye expressions.

[0018] In an embodiment of the first aspect, the first perspective image includes the first perspective data of the first user, or the display area of ​​the first session interface of the first smart wearable device, or includes the first perspective data and the second display area.

[0019] In an embodiment of the first aspect, after collecting the first perspective data of the first user through the first smart wearable device, collecting the eye image of the first user and extracting the eye feature information, the method further includes:

[0020] Extracting eye key points according to the eye feature information, and determining changes in the user's eye features according to the key points based on a preset eye key point model in a healthy state;

[0021] Based on a preset biological rehabilitation theoretical model, the health condition and / or disease condition of the user's eyes is obtained according to the changes;

[0022] A diagnosis report is generated based on the health condition and / or disease condition for reference by the staff.

[0023] A second aspect of the present disclosure provides a conversation interaction device, characterized in that it is applied to a conversation interaction between a first user using a first head-mounted smart wearable device and a second user using a second head-mounted smart wearable device that communicates and interacts with the first head-mounted wearable device;

[0024] The conversation interaction device comprises:

[0025] A collection module, which collects the first perspective data of the first user through the first smart wearable device, collects the eye image of the first user and extracts eye feature information;

[0026] a construction module, communicatively connected to the acquisition module, for generating first virtual character data through the first smart wearable device based on a preset modeling model, according to a pre-acquired original image containing the first user and the eye feature information, so as to obtain third-viewing angle data of the first user according to the first virtual character data; the third-viewing angle data is used for displaying a first virtual character having the face and expression of the first user; the expression is generated based on the eye feature information and changes accordingly;

[0027] A receiving module, configured to receive second perspective data of a first user through the first smart wearable device, where the second perspective data includes second virtual character data of the second user and / or first perspective data of the second user;

[0028] a display module, communicatively connected to the construction module and the receiving module, and configured to form a display of a first conversation interface based on the first perspective data, the second perspective data, and the third perspective data of the first user through the first smart wearable device, wherein the first conversation interface includes: a first display area for displaying the second virtual character, a second display area for displaying the first virtual character, and a first perspective image for displaying the second virtual character in the first display area or in an area other than the first display area and the second display area, wherein the first perspective image is generated based on the first perspective data;

[0029] An interaction module is communicatively connected to the display module, and is used to send the first virtual character data and / or the first perspective data of the first user to the second smart wearable device through the first smart wearable device, so as to form a conversation on the second smart wearable device and display a second conversation interface coordinated with the first conversation interface.

[0030] A third aspect of the present disclosure provides a smart wearable device, characterized in that it includes:

[0031] A collection device, used to collect the user's eye image, first-person perspective data and the user's voice information;

[0032] A central processing unit, which is in communication with the acquisition device, applies the conversation interaction method described in any one of the above embodiments to obtain a first virtual character that matches the first user;

[0033] A display, connected to the central processing unit for displaying a first conversation interface;

[0034] The audio player is used to play the audio of the second user interacting with the first user.

[0035] A fourth aspect of the present disclosure provides a computer-readable storage medium, characterized in that program instructions are stored therein, and the program instructions are executed to execute the conversation interaction method described in any of the above embodiments.

[0036] A fourth aspect of the present disclosure provides a computer program product, characterized in that it includes: program instructions for executing the conversation interaction method described in any of the above embodiments.

[0037] Beneficial effects of the present disclosure: In the present disclosure, eye images of a user wearing a smart wearable device are collected to construct a virtual character corresponding to the user's face, so that the user can interact through the virtual character. The virtual character can accurately reflect the user's real expression and demeanor, thereby improving the effectiveness and credibility of communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1A schematic diagram showing the interaction between a first smart wearable device and a second smart wearable device in an embodiment of the present disclosure.

[0039] Figure 2 A schematic diagram showing a first user wearing a first smart wearable device in an embodiment of the present disclosure.

[0040] Figure 3 A flowchart of a conversation interaction method in an embodiment of the present disclosure is shown.

[0041] Figure 4 A flow chart showing the generation of the first virtual character data based on the pre-acquired original image containing the first user and the eye feature information in a conversation interaction method in one embodiment of the present disclosure is shown.

[0042] Figure 5 A schematic diagram showing a first conversation interface in an embodiment of the present disclosure.

[0043] Figure 6 A schematic diagram showing a flow chart of generating first virtual character data based on user voice information in a conversation interaction method in another embodiment of the present disclosure.

[0044] Figure 7 A schematic diagram of a process of performing health detection based on user eye feature information in a conversation interaction method in one embodiment of the present disclosure is shown.

[0045] Figure 8 A module schematic diagram of a conversation interaction device in an embodiment of the present disclosure is shown.

[0046] Fig. 9 A structural block diagram of a first wearable device in an embodiment of the present disclosure is shown.

[0047] Fig.10 A module schematic diagram of a computer device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0048] The following is an explanation of the embodiments of the present disclosure by specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the information disclosed by the present disclosure. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in the present disclosure can also be modified or changed in various ways according to different viewpoints and application modules without departing from the spirit of the present disclosure. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0049] The following is a detailed description of the embodiments of the present disclosure with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. The present disclosure can be embodied in many different forms and is not limited to the embodiments described herein.

[0050] In the representations of the present disclosure, the reference terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" and the like mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials or characteristics represented may be combined in any one or a group of embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples represented in the present disclosure and the features of different embodiments or examples, unless they are mutually contradictory.

[0051] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of the present disclosure, "a group" means two or more, unless otherwise clearly and specifically defined.

[0052] In order to clearly describe the present disclosure, components not related to the description are omitted, and the same reference numerals are given to the same or similar components throughout the specification.

[0053] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0054] Although the terms first, second, etc. are used to represent various elements in this article in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, modules, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or a group of other features, steps, operations, elements, modules, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0055] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present disclosure. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0056] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with relevant technical literature and the message of the present disclosure, and shall not be overly interpreted as ideal or very formal meanings unless defined.

[0057] In related technologies, conversational interaction mainly relies on text and voice, lacking intuitive visual feedback and emotional expression. When interacting with head-mounted smart wearable devices, users are usually unable to capture their complete facial expressions and demeanor. Since they cannot capture their complete facial information, the two parties in the interaction cannot judge each other's emotions and intentions through facial expressions and demeanor, which greatly reduces the communication effect.

[0058] In the present disclosure, eye images of a user wearing a smart wearable device are collected to construct a virtual character corresponding to the user's face, so that the user can interact through the virtual character. The virtual character can accurately reflect the user's real expression and demeanor, thereby improving the effectiveness and credibility of communication.

[0059] Figure 1 A schematic diagram showing the interaction between a first smart wearable device and a second smart wearable device in an embodiment of the present disclosure.

[0060] Figure 2 A schematic diagram showing a first user wearing a first smart wearable device in an embodiment of the present disclosure.

[0061] exist Figure 1 In the example, the first user 110 wears a first smart wearable device 120 , and the second user 220 wears a second smart wearable device 210 , and a session interaction is established between the first smart wearable device 120 and the second smart wearable device 210 . Figure 2 In the example, the first user 110 wears a smart device, and the second user 220 wears a smart wearable device. Figure 2 Because when the user wears the smart wearable device, the two parties cannot see each other's expressions, and thus cannot judge the authenticity of the information, which ultimately leads to invalid communication results or low credibility. Figure 2 In the example, when the first smart wearable device 120 is worn, Figure 1 The process in the example completes the interaction while mapping the wearer's expression on the virtual character, so that both parties can judge the other party's state when speaking by observing the virtual character's expression, making the communication more credible.

[0062] Specifically, in Figure 1 In the example, the first user 110 wearing the first smart wearable device 120 sends the first virtual character data including the expression of the first user 110 to the second user 220 wearing the second smart wearable device 210, so as to present the first virtual character associated with the first user 110 on the second smart wearable device 210. The second user 220 is the same as the first user 110. In some embodiments, the smart wearable device can not only display the virtual characters of both users, but also send the first perspective data to the other party according to the user's needs, so as to display the first perspective image that the user can see.

[0063] Furthermore, Figure 3 A flowchart of a conversation interaction method in an embodiment of the present disclosure is shown.

[0064] In one embodiment of the present disclosure, a conversation interaction method is provided, wherein the method is applied to a conversation interaction between a first user 110 using a first head-mounted smart wearable device 120 and a second user 220 using a second head-mounted smart wearable device 210 that communicates and interacts with the first head-mounted wearable device;

[0065] exist Figure 3 In an embodiment, the conversation interaction method includes:

[0066] Step S110: collecting first perspective data of the first user 110 through the first smart wearable device 120, collecting eye images of the first user 110 and extracting eye feature information.

[0067] Specifically, by Figure 2 The head-mounted first smart wearable device 120 in the example performs data collection, such as smart glasses or a VR helmet, wherein the first smart wearable device 120 is equipped with a high-resolution camera and an eye tracking sensor. The camera is used to collect the first-view data of the first user 110, that is, the scene image currently seen by the user. The eye tracking sensor is used to collect the eye image of the first user 110 and extract eye feature information, including the degree of opening and closing of the eyes, the direction of rotation of the eyeballs, the shape of the eyelids, etc.

[0068] After the user wears the smart wearable device, the camera captures the scene in front of the user in real time and generates first-person perspective data. This data can be a continuous video stream or a high-frame rate image sequence. The eye tracking sensor captures the user's eye image in real time and extracts eye feature information through image processing technology. For example, a convolutional neural network (CNN) is used to extract the degree of eye opening, the direction of eyeball rotation, the shape of the eyelid, etc.

[0069] Furthermore, extracting eye feature information based on the user's eye image includes: first, using a high-resolution camera and an eye tracking sensor of a head-mounted smart wearable device to collect the user's eye image in real time. Then, preprocessing the collected eye image, including denoising (such as Gaussian filtering, median filtering, etc.) and standardization, to improve image quality and ensure uniform data format. Subsequently, the degree of eye opening and closing is extracted by calculating the distance between the upper and lower eyelids, the direction of eyeball rotation is extracted by detecting the position of the pupil in the eye area, the shape characteristics of the eyelid are extracted by detecting the curve shape of the eyelid, the blinking frequency is extracted by analyzing the opening and closing changes of the eyes in continuous frames, and the activity characteristics of the eye muscles are extracted by detecting the muscle activities around the eyes, such as frowning, raising eyebrows, etc. Finally, the extracted multiple features (such as the degree of eye opening and closing, the direction of eyeball rotation, the shape of the eyelid, etc.) are fused into a feature vector, and the feature vector is normalized to ensure the weight balance between the features, thereby generating complete eye feature information.

[0070] Step S120, the first smart wearable device 120 generates first virtual character data based on a preset modeling model, according to a pre-acquired original image containing the face of the first user 110 and the eye feature information, so as to obtain third-perspective data of the first user 110 based on the first virtual character data; the third-perspective data is used to display a first virtual character having the face and facial expression of the first user 110; the facial expression is generated based on the eye feature information and changes accordingly.

[0071] Specifically, in some embodiments, a modeling model is pre-set, and the model can be a multi-layer neural network based on deep learning, such as a combination of a convolutional neural network (CNN) and a generative adversarial network (GAN). The model needs to be pre-trained, and the training data includes a large number of face images and corresponding eye feature information to ensure that the model can accurately generate virtual character data. An original image containing the face of the first user 110 is obtained in advance, and the original image can be a clear image in a photo, video, or historical session uploaded by the user. The original image is pre-processed, including operations such as denoising and standardization, to improve the image quality. A face model of the user is established based on the original image, and then the eye feature information extracted in step S110 (such as the degree of opening and closing of the eyes, the direction of rotation of the eyeball, the shape of the eyelid, etc.) is input into the modeling model. The modeling model generates the expression of the virtual character based on the eye feature information. For example, if the eye feature information shows that the user's eyes are open and closed to a large extent and the eyelid shape is upturned, then the eyes of the virtual character will also be adjusted to a similar degree of opening and closing and eyelid shape, showing a happy expression. According to the eye feature information collected in real time, the expression of the virtual character is dynamically updated so that it changes with the user's real expression. Or in some embodiments, the virtual character can also be obtained in the following manner: by aligning the eye feature information with the eye area in the original image, the change of the eye area in the original image is consistent with the change of the user's eyes. According to the eye feature information collected in real time, the eye area in the original image is dynamically updated so that it changes with the user's real eyes. For example, if the user's eyes open and close more, the eyes in the original image will also become larger accordingly; if the user's eyeball rotation direction changes, the eyeball in the original image will also rotate accordingly. Among them, for the first user 110, the first virtual character is the third perspective data of the first user 110, and the third perspective data is used to display the first virtual character with the face and expression of the first user 110. The second user 220 end is the same as the first user 110 end, so the first user 110 end is taken as an example, and the second user 220 end is not repeated too much.

[0072] further, Figure 4A flow chart of generating the first virtual character data according to the pre-acquired original image including the first user 110 and the eye feature information in a conversation interaction method in one embodiment of the present disclosure is shown.

[0073] Therefore, in Figure 4 In the embodiment, the process of generating the first virtual character data by the first smart wearable device 120 based on the preset modeling model and the pre-acquired original image containing the face of the first user 110 and the eye feature information in step S120 includes:

[0074] Step S121, detecting a first eye key point in the original image, and determining an eye area according to the first eye key point;

[0075] Step S122, extracting a second eye key point from the eye feature information, and matching the second eye key point with the eye region, so that the eye region changes accordingly according to the second eye key point.

[0076] Specifically, in some embodiments, in step S121, a pre-trained facial key point detection model is used, and the facial key point detection model is taken from a facial detection model library, and the facial detection model library can be self-built or can be an existing facial key point detection model that can be directly used, such as a facial key point detection model in a Dlib library. The key points of the eyes in the original image are extracted, including the corners of the eyes, the edges of the eyelids, etc. These key points are used to determine the areas of the left eye and the right eye, and a polygonal area is drawn by the coordinates of these key points to cover the entire eye.

[0077] In step S122, the second eye key points are obtained from the eye feature information extracted in step S110, including the degree of eye opening, the direction of eyeball rotation, the shape of the eyelid, etc. These second eye key points are matched with the eye area in the original image, and the changes in the eye area in the original image are made consistent with the user's real eye changes by adjusting the size and position of the eye area. For example, if the second eye key points show that the degree of eye opening of the user becomes larger, the eyes in the original image will also become larger accordingly; if the direction of eyeball rotation changes, the eyeball in the original image will also rotate accordingly. In this way, the expression and movement of the virtual character are ensured to be consistent with the real state of the user, so as to provide a video conversation interaction experience between users that is as close to the real one as possible.

[0078] exist Figure 3 In the example, in step S130, the first smart wearable device 120 receives second perspective data of the first user 110, where the second perspective data includes second virtual character data of the second user 220 and / or first perspective data of the second user 220.

[0079] Specifically, in some embodiments, when the first smart wearable device 120 and the second smart wearable device 210 are in the same local area network, the first smart wearable device 120 receives the second perspective data from the second smart wearable device 210 through a wireless communication module (such as Wi-Fi, Bluetooth, etc.). The second perspective data may include the second virtual character data of the second user 220 (the generation principle may be the same as the first virtual character data), and the second virtual character data includes the virtual image of the second user 220 and its expression information, which is used to display the second virtual character in the conversation interface of the first user 110. In addition, the second perspective data may also include the first perspective data of the second user 220, that is, the scene images currently seen by the second user 220, which can be transmitted to the conversation interface of the first user 110 in real time, so that the first user 110 can understand the current field of view of the second user 220. By receiving and processing these second perspective data, the first smart wearable device 120 can provide users with a richer and more realistic conversation interaction experience.

[0080] When the first smart wearable device 120 and the second smart wearable device 210 are not in the same local area network, the first smart wearable device 120 and the second smart wearable device 210 can communicate remotely, for example, through a 4G / 5G mobile network. The second smart wearable device 210 uploads the second virtual character data and / or the first perspective data of the second user 220 to the cloud server. The cloud server receives and stores this data, and then forwards it to the first smart wearable device 120. The first smart wearable device 120 downloads the second virtual character data and / or the first perspective data of the second user 220 from the cloud server and displays it in the conversation interface. In this way, even if the two devices are not in the same local area network, the first smart wearable device 120 can still obtain and display the virtual image of the second user 220 and its current field of view in real time, ensuring the smoothness and naturalness of the conversation process.

[0081] Figure 5 A schematic diagram of a first conversation interface 310 in an embodiment of the present disclosure is shown.

[0082] exist Figure 3 In the example, the first smart wearable device 120 forms a display of a first conversation interface 310 based on the first perspective data, the second perspective data and the third perspective data of the first user 110, and the first conversation interface 310 includes: a first display area 311 for displaying the second virtual character, a second display area 312 for displaying the first virtual character, and a first perspective image for switching with the second virtual character to be displayed in the first display area 311 or in an area outside the first display area 311 and the second display area 312, and the first perspective image is generated based on the first perspective data.

[0083] Specifically, in some embodiments, Figure 5 As shown in the example, the first smart wearable device 120 first receives and processes the first perspective data of the first user 110, the second perspective data of the second user 220 (including the second virtual character data of the second user 220 and / or the first perspective data of the second user 220), and the third perspective data generated according to the eye feature information of the first user 110. The first display area 311 in the first conversation interface 310 is used to display the second virtual character of the second user 220, and to display the virtual image of the second user 220 and its expressions and actions. The second display area 312 is used to display the virtual character of the first user 110, and to display the virtual image of the first user 110 and its expressions and actions, which are generated based on the eye feature information of the first user 110. The first perspective image is used to display the first perspective image of the first user 110, that is, the scene image currently seen by the first user 110. The user can choose to display the first perspective image in the area outside the first display area 311 and the second display area 312, or switch with the second virtual character to display in the first display area 311. In this way, the first conversation interface 310 can simultaneously display the avatars of the second user 220 and the first user 110, and can also selectively display the current field of view of the first user 110. Users can flexibly switch and view different perspectives as needed, which enhances the interactivity and immersion of the conversation.

[0084] Optionally, the first perspective image includes first perspective data of the first user 110 , or a display area of ​​a first session interface 310 of the first smart wearable device 120 , or includes the first perspective data and the second display area 312 .

[0085] In some embodiments, the switching method can be through eye movement, screen or button touch operation, voice command or gesture recognition, etc., to select the switching perspective.

[0086] First-person images can be displayed in a variety of ways:

[0087] After switching, the third display area can display one of the following:

[0088] The other party's first-person perspective image: displays the first-person perspective data captured by the second user 220 through his smart wearable device, that is, the actual scene currently seen by the second user 220. This enables the first user 110 to understand the environment and field of view currently located by the second user 220.

[0089] The image on the other party's display screen: the image displayed on the display screen of the smart wearable device worn by the second user 220, that is, what the second user 220 can see. This includes the virtual character of the second user 220, the virtual character of the first user 110, and any other content displayed on the second user 220's device.

[0090] exist Figure 3 In the embodiment, step S140, the first smart wearable device 120 sends the first virtual character data and / or the first perspective data of the first user 110 to the second smart wearable device 210, so as to form a session on the second smart wearable device 210 to display a second session interface coordinated with the first session interface 310.

[0091] Specifically, in some embodiments, the first smart wearable device 120 transmits the first virtual character data and / or the first perspective data of the first user 110 to the second smart wearable device 210 through the wireless communication module. After receiving the data, the second smart wearable device 210 forms a second conversation interface that matches the first conversation interface 310 according to the data. The area in the second conversation interface is the same as that in the first conversation interface 310, and no further details are given here.

[0092] In order to make the expression of the first virtual character of the first user 110 more closely fit the user, the first smart wearable device 120, in the process of generating the first virtual character data, not only collects the user's voice information in real time based on the pre-acquired original image containing the face of the first user 110 and eye feature information.

[0093] therefore, Figure 6 A schematic diagram showing a flow chart of generating first virtual character data based on user voice information in a conversation interaction method in another embodiment of the present disclosure.

[0094] Optional, in Figure 6 In the example, generating the first virtual character data according to the pre-acquired original image including the first user 110 and the eye feature information further includes:

[0095] Step S150, collecting the user's voice information in real time through the first smart wearable device 120;

[0096] Step S160: The smart wearable device generates a mouth expression included in the third-view data based on the modeling model and the voice information; the mouth expression is generated based on the voice information and changes accordingly.

[0097] Specifically, in some embodiments, the first smart wearable device 120 collects the user's voice information in real time through a built-in microphone. These voice information can be the user's speech sound. The collected voice information is processed using a pre-trained voice recognition model to extract voice feature parameters such as pitch, volume, and speech speed. Based on the extracted voice feature parameters, a modeling model is used to generate corresponding mouth expressions. For example, according to the user's pronunciation characteristics, the corresponding mouth opening and closing degree, lip shape changes, etc. are generated. The generated mouth expression is merged with the previously generated facial expression (such as the opening and closing degree of the eyes, the direction of eyeball rotation, etc.) to generate complete third-view data. According to the real-time collected voice information, the mouth expression in the third-view data is dynamically updated so that it changes with the user's voice. For example, when the user speaks, the mouth of the virtual character will open and close accordingly to simulate real mouth movements.

[0098] Optionally, the expression information also includes other facial expressions that match the simulated eye expressions, which may include mouth expressions. In addition to the eye and mouth expressions, the modeling model also generates other facial expressions that match the eye expressions. For example, based on the user's voice information and eye feature information, the raising or lowering of eyebrows, cheek muscle activity, etc. are generated. Based on the real-time collected voice information and eye feature information, the other facial expressions of the virtual character are dynamically updated to be consistent with the eye and mouth expressions. For example, when the user smiles, the virtual character's eyes will narrow, eyebrows will rise, and the corners of the mouth will curl up.

[0099] Figure 7 A schematic diagram of a process of performing health detection based on user eye feature information in a conversation interaction method in one embodiment of the present disclosure is shown.

[0100] Optional, in Figure 7 In the example, after collecting the first perspective data of the first user 110 through the first smart wearable device 120, collecting the eye image of the first user 110 and extracting the eye feature information, the method further includes:

[0101] Step S170, extracting eye key points according to the eye feature information, and determining changes in the user's eye features according to the key points based on a preset eye key point model in a healthy state;

[0102] Specifically, in some embodiments, a pre-trained key point detection model (such as a facial key point detection model in the Dlib library) is used to extract eye key points from the collected eye images, including the corners of the eyes, the edge of the eyelids, the position of the pupil, etc. The extracted eye key points are compared with the preset eye key point model in a healthy state to determine the changes in the user's eye features. For example, the degree of eye opening and closing, the range of eyeball rotation, the shape of the eyelids, etc. are compared to identify deviations from the healthy state.

[0103] Step S180, based on a preset biological rehabilitation theoretical model, obtain the health condition and / or disease condition of the user's eyes according to the changes.

[0104] Specifically, the preset biological rehabilitation theoretical model is used to evaluate the user's health status according to the changes in eye characteristics. For example, if the degree of eye opening and closing is abnormally reduced, it may indicate fatigue or lack of sleep; if the eyeballs are not flexible, it may indicate eye disease. According to the changes in eye characteristics, combined with the biological rehabilitation theoretical model, it is determined whether the user suffers from specific eye diseases, such as dry eyes, glaucoma, etc.

[0105] Step S190, generating a diagnosis report based on the health condition and / or disease condition for reference by staff.

[0106] Specifically, the evaluation results and diagnostic conclusions are summarized to generate a diagnostic report. The report includes changes in the user's eye characteristics, health assessment results, possible disease conditions, and recommended treatment plans or suggestions for further examinations. The generated diagnostic report is sent to medical staff or the user himself for reference and to take corresponding health management and treatment measures.

[0107] In another embodiment, health detection can be used not only for physical health assessment, but also for mental health monitoring. In addition to the key points related to physical health mentioned above, you can also pay attention to indicators closely related to psychological state, such as changes in pupil size, blinking frequency, and eye concentration. Compared with the preset model under the mental health state, the difference from the normal mental state is identified, such as excessive tension may cause pupil dilation, depression may cause dull eyes, etc. Utilize the psychological theory model to evaluate the user's mental state according to the changes in eye characteristics. For example, frequent blinking and rapid eye movement may indicate that the user is in an anxious state; while long-term eye gaze or lack of eye contact may be a sign of depression. By analyzing the user's eye expression, the user's emotional state, such as happiness, sadness, anger, etc., can be identified, which is very helpful for understanding the user's emotional fluctuations. Comprehensively evaluate the results and generate a report containing the user's mental state. The report should include the user's current mental state, possible cause analysis, improvement suggestions, etc. Provide suggestions based on the user's mental state, such as relaxation techniques, meditation exercises, and recommendations for professional psychological counseling.

[0108] Figure 8 A module schematic diagram of a conversation interaction device in an embodiment of the present disclosure is shown.

[0109] In another embodiment of the present disclosure, a conversation interaction apparatus is provided, wherein the apparatus is applied to a conversation interaction between a first user 110 using a first head-mounted smart wearable device 120 and a second user 220 using a second head-mounted smart wearable device 210 that communicates and interacts with the first head-mounted wearable device;

[0110] The conversation interaction device comprises:

[0111] The acquisition module 401 acquires the first viewing angle data of the first user 110 through the first smart wearable device 120, acquires the eye image of the first user 110 and extracts the eye feature information;

[0112] The construction module 402 is communicatively connected to the acquisition module 401, and is used to generate first virtual character data based on a pre-set modeling model through the first smart wearable device 120, according to the pre-acquired original image containing the first user 110 and the eye feature information, so as to obtain third-viewing angle data of the first user 110 according to the first virtual character data; the third-viewing angle data is used to display a first virtual character having the face and expression of the first user 110; the expression is generated based on the eye feature information and changes accordingly;

[0113] A receiving module 403 is configured to receive second perspective data of the first user 110 through the first smart wearable device 120, where the second perspective data includes second avatar data of the second user 220 and / or first perspective data of the second user 220;

[0114] a display module 404, which is communicatively connected to the construction module 402 and the receiving module 403, and is used to form a display of a first conversation interface 310 based on the first perspective data, the second perspective data, and the third perspective data of the first user 110 through the first smart wearable device 120, wherein the first conversation interface 310 includes: a first display area 311 for displaying the second virtual character, a second display area 312 for displaying the first virtual character, and a first perspective image for displaying the second virtual character in the first display area 311 or in an area other than the first display area 311 and the second display area 312, wherein the first perspective image is generated based on the first perspective data;

[0115] The interaction module 405 is communicatively connected to the display module 404, and is used to send the first virtual character data and / or the first perspective data of the first user 110 to the second smart wearable device 210 through the first smart wearable device 120, so as to form a session on the second smart wearable device 210 to display a second session interface coordinated with the first session interface 310.

[0116] For the first smart wearable device 120, the conversation interaction device is arranged in the first smart wearable device 120, and the conversation interaction device is a virtual device of the above-mentioned conversation interaction method, so it will not be described in detail.

[0117] Fig. 9 A structural block diagram of a first wearable device in an embodiment of the present disclosure is shown.

[0118] In another embodiment of the present disclosure, a smart wearable device is provided, wherein Fig. 9 In the example, the smart wearable device includes:

[0119] A collection device 121, used to collect the user's eye image, first-person perspective data and the user's voice information;

[0120] The central processor 122 is connected to the acquisition device 121 for communication, and applies the conversation interaction method described in any one of the above embodiments to obtain a first virtual character matching the first user 110;

[0121] The display 123 is connected to the central processor 122 for displaying the first conversation interface 310;

[0122] The audio player 124 is used to play the audio of the second user 220 interacting with the first user 110 .

[0123] Specifically, in some embodiments, the acquisition device 121 includes sensors such as a high-resolution camera and a microphone, which can capture the user's eye images, the scene images currently seen by the user (first-person perspective data), and the user's voice information in real time.

[0124] The central processor 122 is connected to the acquisition device 121 in communication, and applies the conversation interaction method described in any one of the above embodiments to obtain a first virtual character matching the first user 110. The central processor 122 is the conversation interaction device in the above embodiment, which may include: eye image processing: using a pre-trained key point detection model to extract eye feature information, such as the degree of eye opening, the direction of eyeball rotation, the shape of eyelids, etc. First perspective data processing: processing the user's first perspective data to generate a first perspective image. Voice information processing: collecting the user's voice information in real time through a built-in microphone, using a pre-trained voice recognition model to extract voice feature parameters, and generating corresponding mouth expressions. Virtual character generation: based on the above processing results, using a modeling model to generate first virtual character data matching the first user 110, including facial expressions, mouth expressions, etc. The first conversation interface 310 is displayed on the display 123, and the audio of the second user 220 is played through the sound player 124, providing a rich and natural conversation interaction experience.

[0125] Fig.10 A module schematic diagram of a computer device according to an embodiment of the present disclosure is shown.

[0126] like Fig.10 As shown in the example, a schematic diagram of the structure of a computer device in an embodiment of the present disclosure is shown.

[0127] The computer device 500 may be exemplified as a processing terminal in a cloud platform, such as a server, a desktop computer, a laptop computer, a tablet computer, a smart phone or other terminals.

[0128] The computer device 500 includes a bus 501, a processor 502, and a memory 503. The processor 502 and the memory 503 can communicate with each other through the bus 501. The memory 503 can store program instructions. The processor 502 implements the steps of the conversation interaction method in the previous embodiment by running the program instructions in the memory 503, for example Figure 1 .

[0129] The bus 501 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, although only one thick line is used in the figure, it does not mean that there is only one bus or one type of bus.

[0130] In some embodiments, the processor 502 may be implemented by a central processing unit (CPU), a microprocessor unit (MCU), a system on chip (System On Chip), or a field programmable logic array (FPGA), etc. The memory 503 may include a volatile memory (Volatile Memory) for temporary storage of data when running a program, such as a random access memory (Random Access Memory, RAM).

[0131] The memory 503 may also include a non-volatile memory (non-volatile memory) for data storage, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state disk (SSD).

[0132] In some embodiments, the computer device 500 may further include a communicator 504. The communicator 504 is used to communicate with the outside. In a specific example, the communicator 504 may include one or a group of wired and / or wireless communication circuit modules. For example, the communicator 504 may include one or more of a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include: for example, near field communication (NFC) technology, infrared (IR) technology, Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc. One or more.

[0133] In an embodiment of the present disclosure, a computer-readable storage medium may be provided, storing program instructions, which implement the conversation interaction method in any of the previous embodiments when executed.

[0134] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or are implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA).

[0135] A computer program product may also be provided in an embodiment of the present disclosure, including: program instructions for executing any of the above-mentioned conversation interaction methods.

[0136] The above embodiments are merely illustrative of the principles and effects of the present disclosure, and are not intended to limit the present disclosure. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present disclosure. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in the present disclosure shall still be covered by the protection scope of the present disclosure.

Claims

1. A conversation interaction method, characterized in that: Applicable to conversation interaction between a first user using a first head-mounted smart wearable device and a second user using a second head-mounted smart wearable device that communicates and interacts with the first head-mounted wearable device; The conversation interaction method comprises: Collecting the first perspective data of the first user through the first smart wearable device, collecting the eye image of the first user and extracting eye feature information; The first smart wearable device generates first virtual character data based on a pre-set modeling model, according to a pre-acquired original image containing the face of the first user and the eye feature information, so as to obtain third-viewing angle data of the first user according to the first virtual character data; the third-viewing angle data is used to display a first virtual character having the face and facial expression of the first user; the facial expression is generated based on the eye feature information and changes accordingly; The first smart wearable device receives second perspective data of a first user, where the second perspective data includes second virtual character data of the second user and / or first perspective data of the second user; The first smart wearable device forms a display of a first conversation interface based on the first perspective data, the second perspective data, and the third perspective data of the first user, wherein the first conversation interface includes: a first display area for displaying the second virtual character, a second display area for displaying the first virtual character, and a first perspective image for displaying the second virtual character in the first display area or in an area other than the first display area and the second display area, wherein the first perspective image is generated based on the first perspective data; The first smart wearable device sends the first virtual character data and / or the first perspective data of the first user to the second smart wearable device, so as to form a session on the second smart wearable device and display a second session interface coordinated with the first session interface.

2. The conversation interaction method according to claim 1, characterized in that: Generating the first virtual character data according to the pre-acquired original image containing the first user and the eye feature information further includes: Collecting the user's voice information in real time through the first smart wearable device; The smart wearable device generates a mouth expression included in the third-view data based on the modeling model and the voice information; the mouth expression is generated based on the voice information and changes accordingly.

3. The conversation interaction method according to claim 1, characterized in that: The first smart wearable device generates the first virtual character data based on the preset modeling model and the pre-acquired original image containing the first user's face and the eye feature information, including: Detecting a first eye key point in the original image, and determining an eye area according to the first eye key point; A second eye key point in the eye feature information is extracted, and the second eye key point is matched with the eye region, so that the eye region changes accordingly according to the second eye key point.

4. The conversation interaction method according to claim 1, characterized in that: The expression information also includes other facial expressions that match the simulated eye expressions.

5. The conversation interaction method according to claim 1, characterized in that: The first perspective image includes the first perspective data of the first user, or the display area of ​​the first session interface of the first smart wearable device, or includes the first perspective data and the second display area.

6. The conversation interaction method according to claim 1, characterized in that: After collecting the first perspective data of the first user through the first smart wearable device, collecting the eye image of the first user and extracting the eye feature information, the method further includes: Extracting eye key points according to the eye feature information, and determining changes in the user's eye features according to the key points based on a preset eye key point model in a healthy state; Based on a preset biological rehabilitation theoretical model, the health condition and / or disease condition of the user's eyes is obtained according to the changes; A diagnosis report is generated based on the health condition and / or disease condition for reference by the staff.

7. A conversation interaction device, characterized in that: Applicable to conversation interaction between a first user using a first head-mounted smart wearable device and a second user using a second head-mounted smart wearable device that communicates and interacts with the first head-mounted wearable device; The conversation interaction device comprises: A collection module, which collects the first perspective data of the first user through the first smart wearable device, collects the eye image of the first user and extracts eye feature information; a construction module, communicatively connected to the acquisition module, for generating first virtual character data through the first smart wearable device based on a preset modeling model, according to a pre-acquired original image containing the first user and the eye feature information, so as to obtain third-viewing angle data of the first user according to the first virtual character data; the third-viewing angle data is used for displaying a first virtual character having the face and expression of the first user; the expression is generated based on the eye feature information and changes accordingly; A receiving module, configured to receive second perspective data of a first user through the first smart wearable device, where the second perspective data includes second virtual character data of the second user and / or first perspective data of the second user; a display module, communicatively connected to the construction module and the receiving module, and configured to form a display of a first conversation interface based on the first perspective data, the second perspective data, and the third perspective data of the first user through the first smart wearable device, wherein the first conversation interface includes: a first display area for displaying the second virtual character, a second display area for displaying the first virtual character, and a first perspective image for displaying the second virtual character in the first display area or in an area other than the first display area and the second display area, wherein the first perspective image is generated based on the first perspective data; An interaction module is communicatively connected to the display module, and is used to send the first virtual character data and / or the first perspective data of the first user to the second smart wearable device through the first smart wearable device, so as to form a conversation on the second smart wearable device and display a second conversation interface coordinated with the first conversation interface.

8. A smart wearable device, characterized in that: include: A collection device, used to collect the user's eye image, first-person perspective data and the user's voice information; A central processing unit, in communication with the acquisition device, and applying the conversation interaction method according to any one of claims 1 to 6 to obtain a first virtual character matched to the first user; A display, connected to the central processing unit for displaying a first conversation interface; The audio player is used to play the audio of the second user interacting with the first user.

9. A computer-readable storage medium, characterized in that: Program instructions are stored, and the program instructions are executed to perform the conversation interaction method according to any one of claims 1 to 6.

10. A computer program product, characterized in that include: Program instructions for executing the conversation interaction method as described in any one of claims 1-6.

Images