VR real-time communication interaction system and method combining 5G, expression tracking and beautification

Abstract

The present invention discloses a VR real-time communication interaction system and method combining 5G, expression tracking and beautification. The system is provided with a cloud server, multiple cluster processing devices, multiple communication rooms and multiple user interaction devices. The cloud server is respectively connected to the multiple cluster processing devices, and each cluster processing device is respectively connected to multiple adjacent user interaction devices, thereby forming multiple regional processing stations. Each user interaction device is arranged in a corresponding communication room. The cluster processing device is used to match and combine the facial model with the universal body model to obtain a user simulation model, and the user simulation model is forwarded to the communication party with which the communication connection has been established through the cloud server. The present invention models the user and environment that establish the communication interaction and restores the expression and posture of the communication party to produce a stronger sense of substitution in the interaction.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a VR real-time communication interaction system and method combining 5G, expression tracking and beautification. Background Art

[0002] VR refers to virtual reality technology, which mainly adopts the integration of multiple technologies, including but not limited to computer vision technology, wide-angle display technology, dynamic motion capture technology, etc. With VR, we can create and experience a virtual environment that simulates the real environment generated by a computer, providing people with multi-dimensional perceptual information such as vision, hearing, and touch, making people feel involved.

[0003] In recent years, with the continuous development and innovation of VR technology, VR technology has also been widely used in various fields, such as in game entertainment to make up for the lack of 3D immersion in games, and in the field of medical simulation training to provide real medical training scenarios, etc. However, in general, most of the VR usage scenarios on the market are based on the personal experience of a single user, lacking usage scenarios for multiple users, and lacking online interaction and real-time communication between different users. In addition, in today's era, an instant messaging and interactive device based on VR technology that meets multi-user usage scenarios, compared to instant messaging and interactive devices such as smartphones, can undoubtedly provide users with more dimensional perceptual information, allowing users to feel immersive, and also meet the various entertainment and communication needs of different users in two places.

[0004] In the existing research on VR instant communication and interactive devices, such as street phone booth VR sharing device (patent application number: CN202020822636.5) and a VR-based viewing method, device and system (patent application number: CN202011182157.2), the former is only an instant communication and interactive device based on an existing phone booth combined with VR glasses and a small number of sensors, which only meets communication needs but is difficult to provide entertainment content, while the latter only provides virtual scenes. Moreover, most of the current patents rarely use 5G technology as their main wireless communication means, which is difficult to meet the needs of multiple users to be online instantly. Summary of the invention

[0005] In order to overcome the defects and shortcomings of the prior art, the first purpose of the present invention is to provide a VR real-time communication interaction system that combines 5G, expression tracking and beauty. The system models the users and environment that establish communication interactions and accurately restores the expressions and postures of the communicating parties to create a stronger sense of immersion in the interaction, while meeting the various needs of multi-user communication, entertainment, etc.

[0006] The second object of the present invention is to provide a VR real-time communication interaction method that combines 5G, expression tracking and beautification.

[0007] In order to achieve the above first purpose, the present invention adopts the following technical solutions:

[0008] A VR real-time communication and interaction system combining 5G, expression tracking and beautification, the system is provided with a cloud server, a plurality of cluster processing devices, a plurality of communication rooms and a plurality of user interaction devices, the cloud server is respectively connected to the plurality of cluster processing devices, each cluster processing device is respectively connected to a plurality of adjacent user interaction devices, thereby forming a plurality of regional processing stations, each user interaction device is arranged in a corresponding communication room, the user interaction device comprises a first image acquisition device, a user processor, a head-mounted VR interaction device and a first communication module, the user processor is respectively connected to the first image acquisition device and the first communication module, the first communication module is also connected to the cluster processing device, and the user processor transmits data with the cluster processing device through the first communication module;

[0009] The head-mounted VR interactive device is provided with a display device, a second image acquisition device, a micro-electrode sheet and a second communication module, wherein the micro-electrode sheet, the display device and the second image acquisition device are respectively connected to the second communication module, and the second communication module is also wirelessly connected to the first communication module;

[0010] The microelectrode sheet is used to collect EOG signals;

[0011] The first image acquisition device is used to acquire an environment image including a user's limbs, and the second image acquisition device is used to acquire a user's facial image, and the second image acquisition device is arranged on a head-mounted VR interaction device;

[0012] The user processor is used to create a general body model and a facial model, identify the user's posture and expression to obtain posture tracking information and expression recognition type, wherein the posture information includes head posture information, hand posture information and limb posture information;

[0013] The cluster processing device is used to receive a general body model, a facial model, posture tracking information and expression recognition type, match and combine the facial model with the general body model to obtain a user simulation model, and forward the user simulation model to a communication party with which a communication connection has been established through a cloud server;

[0014] The cloud server is used to forward data and pre-build the environment scene model and user simulation model;

[0015] The display device is used to display the environment scene model and the user simulation model;

[0016] The cluster processing device connected to the communication party updates the posture and expression of the user simulation model according to the posture tracking information and the expression recognition type.

[0017] As a preferred technical solution, the cloud server is used to match the communication party that establishes a connection with the current user, and the communication party is one or more communication users that have established communication interaction with the current user;

[0018] The user processor receives a preset environment scene type input by a user, and forwards it to a cloud server through a cluster processing device. The cloud server stores a plurality of preset environment scene types. The cloud server generates an environment scene model according to the preset environment scene type selected by the user through the user processor, matches the communication party connected to the current user, and sends the environment scene model, user simulation model, posture tracking information, and expression recognition type to the cluster processing device connected to the communication party.

[0019] As a preferred technical solution, the user interaction device further includes a laser sensor, which is arranged in the communication room and connected to the user processor;

[0020] The laser sensor is used to locate the position of the user in the communication room. The user processor obtains the user position through the laser sensor, and transmits the user position to the cloud server through the cluster processing device through the first communication module. The cloud server forwards the user position to the cluster processing device connected to the communication party. The communication party is one or any multiple communication users who have established communication interaction with the current user. The cluster processing device connected to the communication party updates the relative position of the user simulation model in the environmental scene model according to the user position, and sends the position update information to the user processor of the communication party.

[0021] As a preferred technical solution, the user interaction device also includes a photosensitive sensor, a laser lighthouse base station, and a sensing handle. The laser lighthouse base stations are arranged at both ends of the diagonal of the user interaction device. Each laser lighthouse base station is about 2m high. A laser sensor array for positioning is provided in the laser lighthouse base station. The photosensitive sensors are arranged on the ground and the sensing handle of the communication room, so that the laser sensor array and the photosensitive sensor constitute a laser lighthouse laser positioning system.

[0022] As a preferred technical solution, the user interaction device further includes an ambient light sensor, which is arranged in the communication room and connected to the user processor;

[0023] The ambient light sensor is used to sense the light conditions around the user in the communication room. The user processor uses the ambient light sensor to obtain the user's ambient light information, and transmits the user's ambient light information to the cloud server through the cluster processing device via the first communication module. The cloud server forwards the user's ambient light information to the cluster processing device connected to the communication party. The communication party is one or any multiple communication users who have established communication interaction with the current user. The user processor of the communication party adjusts the display brightness of the environmental scene model according to the user's ambient light information.

[0024] As a preferred technical solution, the cluster processing device and the user interaction device are in a one-to-many relationship, and the data of each user interaction device in an area is solved and distributed by a cluster processing device deployed in the area.

[0025] As a preferred technical solution, the head-mounted VR interactive device also includes a mirror frame, and the micro-electrode sheet, display device, second image acquisition device and second communication module are all arranged on the mirror frame. The mirror frame is provided with multiple foam areas, and the foam areas are used to install the micro-electrode sheet.

[0026] In order to achieve the above second purpose, the present invention adopts the following technical solutions:

[0027] A VR real-time communication interaction method combining 5G, expression tracking and beautification, comprising the following steps:

[0028] Model building steps: Create a general body model and a facial model, and match the facial model with the general body model to obtain a user simulation model;

[0029] Scene building steps: select the target scene type and generate the environment scene model;

[0030] Data collection steps: collecting environmental images including user limbs, user facial images, and EOG signals;

[0031] Model information updating step: separating the environment image containing the user's limbs to obtain a user posture image, tracking the user's head, hands and limb movements according to the user posture image to obtain posture tracking information, tracking the user's expression according to the user's facial image to obtain eyeball features, combining the eyeball features and EOG signals as expression features and inputting them into the expression recognition model for prediction to obtain the expression recognition type;

[0032] The expression recognition model is obtained by machine learning through multiple sets of training data, each set of training data includes expression features and label information identifying the expression recognition type corresponding to the expression features, and the expression recognition type is an identifier of multiple expression types;

[0033] Image adjustment step: the image of the communication party is beautified by rendering the user simulation model according to the preset beautification type selected by the user, and the beautification processing includes face thinning, skin smoothing, and whitening.

[0034] User synchronization step: match the communication party that has established a connection with the current user, send the environment scene model, user simulation model, posture tracking information, and expression recognition type to the communication party, and the communication party updates the user simulation model based on the posture tracking information and expression recognition type to update the posture and expression of the user simulation model.

[0035] As a preferred technical solution, the expression recognition model is obtained by machine learning through multiple sets of training data, and the specific process includes:

[0036] Pupil diameter and fixation frequency are selected as eye features, and the effect of illumination on pupil diameter change is reduced by constructing an illumination reflection model.

[0037] The EOG signal is filtered with a 0.05-50 Hz bandpass filter to obtain the EOG signal with power spectrum changes at different scanning frequencies;

[0038] Through different external video and sound tests, the eye features and EOG signals of anger, pain, disgust, sadness, fear, surprise, happiness and joy are collected, and the corresponding expression types are used as label information to form the emotion training set data;

[0039] The SVM classifier is used to find the optimal feature value of the two feature data corresponding to different emotions to complete the classification of expression recognition types and obtain the expression recognition model.

[0040] As a preferred technical solution, the head tracking, hand tracking and limb movement tracking of the user according to the user posture image to obtain posture tracking information is specifically to use a segmented 3D posture estimation method to complete motion capture and posture estimation, and the 3D posture estimation method includes the following steps:

[0041] Use the depth camera to obtain the depth image information and plane information of the current camera shooting area;

[0042] Use the VGG network to extract the plane information and obtain the eigenvalue;

[0043] Use the target tracking algorithm based on deep learning to complete the tracking of single or multiple people: input the extracted feature value, use the CNN network to predict the 2D joint position of the target rectangle in real time to obtain the corresponding Heatmaps——H {j,t} ;

[0044] Based on Heatmaps——H {j,t}And the depth image information is used to extract the connection relationship of each joint;

[0045] Heatmaps——H {j,t} The connection relationship of each joint is used as input, and the pre-trained RCNN network is used to map the 2D image pixels to the 3D surface model points to obtain the corresponding 3D joint positions;

[0046] Integrate 3D joint positions to get location-maps——X {j,t} , Y {j,t} , Z {j,t} ; Where j represents the joint point and t represents the time;

[0047] Based on Heatmaps and locaiton maps, 2D key points and 3D pose estimation are extracted and combined to obtain the final motion capture and pose estimation data.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] (1) The present invention can quickly and accurately establish 3D models of both parties by setting a first image acquisition device, a second image acquisition device, and a microelectrode sheet in a user interaction device in a communication room to collect data on the user's body data. Then, by collecting facial information, the user's face can be modeled and imported into the face of the body model, so that the whole person can be restored more accurately. Subsequently, the first communication module and the second communication module are used for wireless transmission to import the models of both parties into a self-selected scene, so that the two communicating parties have the feeling of face-to-face conversation. In general, the present invention allows the two communicating parties to be immersed in the scene, solving the sense of alienation brought to people by existing communication methods.

[0050] (2) The VR real-time communication interaction system combining 5G, expression tracking and beauty in the present invention connects the cluster processing device with multiple adjacent communication rooms and calculates a large amount of data in real time. The cluster processing device is specifically deployed in edge computer rooms in various places. Compared with the traditional cloud computing method of distributing calculated data, the graphics rendering on the edge cloud will calculate and feedback the delay-sensitive device head tracking, controller tracking, hand tracking and motion tracking information accordingly, thereby achieving lower latency; the data processing work in the interactive system is completed through the cloud, reducing the cost of setting up hardware such as processors.

[0051] (3) Based on the high speed, large capacity and low latency of 5G technology, the present invention adopts a technical solution of applying 5G interaction technology to a VR real-time communication interaction system that combines 5G, expression tracking and beauty. It provides comprehensive support for the instant interaction between the VR interaction system and the terminal device, and solves the problem of the VR system's need for powerful computing and network support for the prior information (user's facial and body data, etc.) collected in advance during the instant interaction and the feedback data generated during the experiment. Compared with the previous mobile communication networks, the 5G network can carry an exponential amount of data and has good performance in accelerating data transmission rate, reducing congestion delay, and improving security and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a structural diagram of a VR real-time communication interaction system combining 5G, expression tracking and beautification in Example 1 of the present invention;

[0053] Figure 2 is a schematic diagram of the structure of a user interaction device in Embodiment 1 of the present invention;

[0054] Figure 3 This is a flowchart of the steps of the VR real-time communication interaction method combining 5G, expression tracking and beautification in Example 2 of the present invention;

[0055] Figure 4 This is a schematic diagram of scene update in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0056] In the description of the present disclosure, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present disclosure.

[0057] In addition, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. Similarly, words such as "one", "an", or "the" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Words such as "include" or "comprise" and the like mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connected" or "connected" and the like are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0058] In the description of the present disclosure, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be internal communication between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present disclosure can be understood according to specific circumstances. In addition, the technical features involved in the different embodiments of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] Example

[0061] Example 1

[0062] This embodiment takes the actual application scenario of a VR real-time communication interaction system that combines 5G, expression tracking and beautification and is concentrated in a telephone booth on the street as an example. Compared with ordinary telephone booths, this embodiment can meet the "face-to-face" experience and customization needs of users' communication interactions. Small and convenient telephone booths have low costs, good adaptability, and strong promotion. It has a certain degree of guiding significance for the practical application of the VR real-time communication interaction system that combines 5G, expression tracking and beautification.

[0063] like Figure 1 As shown, this embodiment provides a VR real-time communication interaction system that combines 5G, expression tracking and beauty. This embodiment takes VR glasses as a head-mounted VR interaction device as an example for explanation: the system is provided with a cloud server, multiple cluster processing devices, multiple communication rooms and multiple user interaction devices. The cloud server is respectively connected to the multiple cluster processing devices, and each cluster processing device is respectively connected to the adjacent multiple user interaction devices, thereby forming multiple regional processing stations, and each user interaction device is arranged in a corresponding communication room.

[0064] In this embodiment, there is a one-to-many relationship between the cluster processing device and the user interaction device, that is, the data of each user interaction device in an area is solved and distributed by a cluster processing device deployed in the area.

[0065] like Figure 2As shown, the user interaction device includes a first image acquisition device, a user processor, VR glasses and a first communication module. The user processor is connected to the first image acquisition device and the first communication module respectively. The first communication module is also connected to the cluster processing device. The user processor transmits data with the cluster processing device through the first communication module.

[0066] VR glasses are provided with a mirror frame, a display device, a second image acquisition device, a microelectrode sheet and a second communication module. The microelectrode sheet, the display device, the second image acquisition device and the second communication module are all arranged on the mirror frame. The microelectrode sheet, the display device and the second image acquisition device are respectively connected to the second communication module, and the second communication module is also wirelessly connected to the first communication module. The mirror frame is provided with multiple foam areas, and the foam areas are used to install microelectrode sheets. The microelectrode sheets are evenly distributed in the foam areas and form a circle around the foam areas. The microelectrode sheets are used to collect EOG signals. In actual application, the number of microelectrode sheets is set to 8; the second image acquisition device adopts an eye tracker, specifically an infrared camera; the first communication module and the second communication module both adopt 5G communication modules, use 5G instant messaging technology for wireless transmission, and adopt a base station deployment method integrating wireless access network and backhaul to achieve more flexible and low-cost base station deployment, and use 5G links to achieve real-time backhaul of signals to reduce latency and achieve higher real-time performance.

[0067] In this embodiment, the first image acquisition device is used to acquire an environmental image including the user's limbs, and the second image acquisition device is used to acquire an image of the user's face, wherein the second image acquisition device is disposed on the VR glasses. In actual application, the first image acquisition device uses a multi-eye camera.

[0068] The user processor is used to create a universal body model and a facial model, and send the universal body model and the facial model to the cluster processing device through the first communication module.

[0069] In this embodiment, the user processor is also used to identify the user's posture and expression to obtain posture tracking information and expression recognition type. Specifically, the environment image containing the user's limbs is separated to obtain the user posture image, the user's head tracking, hand tracking and limb movement tracking are performed according to the user posture image to obtain posture tracking information, the EOG signal collected by the microelectrode sheet is received, the user's expression is tracked according to the user's facial image to obtain the eyeball feature, the eyeball feature and the EOG signal are combined to obtain the expression recognition type, and the posture tracking information and the expression recognition type are sent to the cluster processing device through the first communication module, and the posture information includes the head posture information, the hand posture information and the limb posture information.

[0070] In this embodiment, the cluster processing device is an edge computing processing device connected to multiple adjacent communication rooms, and calculates a large amount of data in real time, and is specifically deployed in edge computer rooms in various places. The cluster processing device is used to receive a general body model, a facial model, receive posture tracking information and expression recognition type, match the facial model with the general body model to obtain a user simulation model, and forward the user simulation model to the communication party with which the communication connection has been established through the cloud server.

[0071] In this embodiment, the display device of the VR glasses is used to display the environmental scene model and the user simulation model generated based on virtual reality to the user.

[0072] In this embodiment, the cloud server stores multiple preset environmental scene types. The cloud server generates an environmental scene model according to the preset environmental scene type selected by the user through the user processor, matches the communication party connected to the current user, and sends the environmental scene model, user simulation model, posture tracking information, and expression recognition type to the cluster processing device connected to the communication party. The communication party is one or more communication users who have established communication interaction with the current user. The cluster processing device connected to the communication party updates the posture and expression of the user simulation model according to the posture tracking information and expression recognition type. The user processor of the communication party receives the updated user simulation model and displays it through the display device of the VR glasses. In addition, the cluster processing device is also used to beautify the user simulation model by rendering the image of the communication party according to the preset beautification type selected by the user. The beautification processing includes face thinning, skin grinding, and whitening. In actual application, the user processor receives the preset environmental scene type input by the user and forwards it to the cloud server through the cluster processing device.

[0073] In this embodiment, the user interaction device further includes a laser sensor and an ambient light sensor, which are arranged in the communication room, and are connected to the user processor respectively.

[0074] The laser sensor is used to locate the position of the user in the communication room. The user processor obtains the user position through the laser sensor, and transmits the user position to the cloud server through the cluster processing device through the first communication module. The cloud server forwards the user position to the cluster processing device connected to the communication party. The communication party is one or any multiple communication users who have established communication interaction with the current user. The cluster processing device connected to the communication party updates the relative position of the user simulation model in the environmental scene model according to the user position, and sends the position update information to the user processor of the communication party for display through the display device.

[0075] The ambient light sensor is used to sense the light conditions around the user in the communication room. The user processor uses the ambient light sensor to obtain the user's ambient light information, and transmits the user's ambient light information to the cloud server through the cluster processing device via the first communication module. The cloud server forwards the user's ambient light information to the cluster processing device connected to the communication party. The communication party is one or any multiple communication users who have established communication interaction with the current user. The user processor of the communication party adjusts the display brightness of the environmental scene model according to the user's ambient light information, thereby achieving energy saving for the VR glasses while improving the user's visual comfort.

[0076] In actual application, a laser lighthouse base station is set at both ends of the diagonal of the user interaction device. Each laser lighthouse base station is about 2m high. A laser sensor array for positioning is installed in the laser lighthouse base station. The laser sensor array and the photosensitive sensors distributed on the head-mounted VR interaction device, the ground and the sensing handle constitute a complete laser lighthouse laser positioning system; the first image acquisition device is specifically a depth camera, and the depth cameras are set to 8 and evenly distributed above the surrounding of the user interaction device; two ambient light sensors are set, respectively installed on the walls on both sides of the user interaction device. The sensing handle is specifically a game handle.

[0077] In addition, those skilled in the art should understand that the communication parties that establish a communication interaction are one or any multiple user interaction devices that communicate through the cloud server.

[0078] It should be noted that those skilled in the art can replace the VR glasses with other head-mounted VR interactive devices according to actual conditions, and accordingly set a microelectrode sheet, a display device, a second image acquisition device and a second communication module on the interactive device, and this embodiment does not limit this.

[0079] In this embodiment, from the perspective of the VR device, it can be divided into three parts. The first part is display technology. Based on the knowledge related to computer graphics and operating systems, the device engine renders the VR screen through low-latency rendering pipelines, foveation rendering, light field display, etc., and then displays it on the screen after computer processing processes such as anti-distortion, synthesis, and position prediction. Among them, in order to improve the efficiency of rendering, it is necessary to focus on introducing foveation rendering technology. The actions of each user operation can be fed back to the VR space in real time, and each user participating in the interaction can observe the actions of others in real time. When the user's vision reaches a certain position in the three-dimensional space, the rays emitted by the main camera on the VR glasses hit the objects involved in the user's vision, and the VR system can respond in real time, activate and display the hit objects. In addition, related computer graphics technologies, such as rendering technology, can also be used to beautify the facial and body images according to the preset beautification type selected by the user, so as to meet the user's imaging requirements as much as possible; the second part is positioning technology. In view of the high real-time nature of the VR real-time communication and interaction system that combines 5G, expression tracking and beauty, using an external laser positioning method such as HTC vive, after the transmitter scans the space, it calculates the time difference between the helmet and the sensing handle receiving the laser signal, and then derives the spatial coordinates of the device. Among them, tactile feedback is provided with the help of external auxiliary props such as sensing handles, and the specific grasping force is obtained by setting pressure sensors on the external auxiliary props. Specifically, through the separate six-degree-of-freedom spatial tracking of the two hands, namely three rotational degrees of freedom and three translational degrees of freedom, more precise and complex interactive actions are achieved by using the buttons and vibration feedback on the sensing handles, thereby realizing the full-dimensional interaction between each user and the virtual environment; the third part is recognition technology. In order to quickly and accurately establish the 3D model of the interacting parties, the VR real-time communication interaction system that combines 5G, expression tracking and beauty collects body data, facial data and other prior information in advance, and establishes a universal character model for the relevant character group, and performs gesture recognition, body posture recognition, expression recognition, eye movement tracking, etc. by wearing gloves or using sensing handles.

[0080] It should be noted that those skilled in the art should understand that the VR real-time communication interaction system combining 5G, expression tracking and beautification can not only be used in application scenarios of instant calls such as telephone booths, but also in application scenarios of entertainment activities, which is not limited in this embodiment.

[0081] Example 2

[0082] Combination Figure 3 and Figure 4 As shown, this embodiment provides a VR real-time communication interaction method combining 5G, expression tracking and beautification, and the method includes the following steps:

[0083] Model building steps: Create a general body model and a facial model, and match the facial model with the general body model to obtain a user simulation model;

[0084] Scene building steps: select the target scene type and generate the environment scene model;

[0085] Data collection steps: collecting environmental images including user limbs, user facial images, and EOG signals;

[0086] Model information updating step: separating the environment image containing the user's limbs to obtain a user posture image, tracking the user's head, hands and limb movements according to the user posture image to obtain posture tracking information, tracking the user's expression according to the user's facial image to obtain eyeball features, combining the eyeball features and EOG signals as expression features and inputting them into the expression recognition model for prediction to obtain the expression recognition type;

[0087] The facial expression recognition model is obtained by machine learning through multiple sets of training data. Each set of training data includes facial expression features and label information identifying the facial expression recognition type corresponding to the facial expression features. The facial expression recognition type is an identifier of multiple facial expression types, which specifically include anger, pain, disgust, sadness, fear, surprise, happiness and pleasure.

[0088] User synchronization step: match the communication party that has established a connection with the current user, send the environment scene model, user simulation model, posture tracking information, and expression recognition type to the communication party, and the communication party updates the user simulation model based on the posture tracking information and expression recognition type to update the posture and expression of the user simulation model.

[0089] Image adjustment step: The image of the communicating party is beautified by rendering the user simulation model according to the preset beautification type selected by the user, so as to further improve the user's interactive experience. The beautification processing includes face thinning, skin smoothing, and whitening.

[0090] In this embodiment, in order to further improve the realism of the user interaction experience, the user synchronization step also includes: using a laser sensor and a lighthouse laser positioning method to obtain the user's position, and updating the relative position of the user simulation model in the environmental scene model according to the user's position, thereby satisfying the synchronization effect of the user walking in a small range within the spatial range.

[0091] In this embodiment, the VR real-time communication interaction method combining 5G, expression tracking and beautification also includes an ambient light adjustment step. The ambient light adjustment step is specifically as follows: the user processor uses the ambient light sensor to obtain the user's ambient light information, and adjusts the display brightness of the environmental scene model according to the user's ambient light information, thereby achieving energy saving for the VR glasses while improving the user's visual comfort.

[0092] In this embodiment, the model building step specifically includes the following steps:

[0093] Generic body model is obtained based on user generation of matching user classification labels, where user classification labels include age, gender, race, and height;

[0094] Extracting frontal face image information and depth feature points from the user's facial image captured by the second image capture device, establishing a universal face model based on the frontal face image information, matching the depth feature points with the universal face model and performing smoothing processing to obtain a facial model;

[0095] The facial model is loaded into the universal body model, and the facial model is matched and combined with the universal body model to obtain a user simulation model.

[0096] In this embodiment, the construction process of the face modeling system of the face model is as follows:

[0097] Use a high-resolution camera to take a number of high-resolution photos of faces with different facial features from the side and front, including age, skin color, mouth shape, nose shape, etc.;

[0098] Using manual marking methods to mark facial feature points in the above photos, these features include but are not limited to facial contour points and facial features position points;

[0099] Introduce several labeled faces into a unified topological structure to obtain the original training set;

[0100] The original training set is imported into a GAN-based neural network for training, and a set of mapping functions from two-dimensional face photos to three-dimensional face models is obtained. The training process is specifically as follows: using the 3MSS algorithm to obtain a three-dimensional facial image with simple texture based on the input high-resolution image; using the super-resolution model to extract depth information to further improve the input image resolution; using the depth information to update the 3D facial image to obtain a 3D facial model with precise texture, and finally obtaining the mapping relationship between the input image and the 3D facial model with precise texture.

[0101] In actual application, the scene building steps adopt a pre-constructed method, using professional modeling tools such as PDMS and SP3D, and combined with deep learning algorithms to achieve a balance between data volume and accuracy. The three-dimensional factory design system PDMS establishes a component library covering the required component size information and a callable grade library that has a certain association with the component library to form an accurate and comprehensive independent database. By setting the equipment name, type, and placement point parameterization to establish equipment and attached ladders, platforms, etc., according to the relevant commands of the pipeline on the PID, the pipeline construction adjustment, additional pipeline support, and various information of the injection pipeline are carried out to achieve the establishment of the model and the output of the plane map. In the interactive design process, real-time collision checks are carried out, and errors in the generation of pipeline branches are discovered and corrected in a timely manner under the premise of data consistency. Make full use of the advantages of network resource sharing and interactive communication, and match the high real-time nature of the VR real-time communication interaction system that combines 5G, expression tracking and beauty. Support secondary development according to the needs of the VR environment to achieve a dynamic model information system with all-round overall optimization and reasonable layout. SP3D has automation functions and rule-driven technology. Through collaborative design of various professions and unified management of data, it significantly improves production efficiency, design and engineering quality. After the relevant parameters of model construction are summarized into a table pipeline table for processing, they are imported into the pipeline system at one time, and the processing tasks are planned in batches to efficiently establish the pipeline system. Through the comprehensive collision inspection on the server side and the real-time collision inspection during the work process, the inspection of the VR real-time communication interaction system combined with 5G, expression tracking and beauty, the positioning of the collision point and the real-time collision inspection during the work process are realized respectively.

[0102] When constructing a scene, the following steps are included:

[0103] a. Select several typical objects as materials for the model library;

[0104] b. Shoot a high-precision image of the material in step a to obtain parameters such as the texture and shape of the real scene;

[0105] c. Obtain accurate data on the size of each material by measuring it on site;

[0106] d. Use 3DMAX, SP3D, and PDMS to model the above typical objects and build a model library;

[0107] e. Obtain high-precision images of other possible texture shapes of typical objects in step a by taking pictures;

[0108] f. Use the pictures in e as training sets to train a convolutional neural network (CNN) as a style transfer network;

[0109] g. Use the convolutional neural network trained in f to perform style transfer on the model in the model library in d to obtain a model expansion library with different textures.

[0110] e. Select appropriate models from the material library and import them into Unity to complete the design and construction of the 3D scene.

[0111] Based on the model obtained by combining facial information and body information, the rendering technology of computer graphics is used to achieve the beautification effect of the face. The face thinning function is achieved by offsetting the pixel position: the coordinates before deformation are calculated based on the inverse change of the coordinates after deformation, and the RGB pixel value of the coordinate is obtained by interpolation and used as the pixel value of the corresponding coordinate after deformation. The image is subjected to skin color detection, and various filtering algorithms are used to achieve the effect of skin smoothing. For example, the weight information composed of Gaussian components and gradient components is blurred and smooth but can retain the edge of the bilateral filtering; the fast guided filtering based on mean convolution, etc. The portrait whitening effect is achieved by increasing the R, G, B, and A values ​​in the image to a certain extent. Users can adjust the degree of various beautification according to their own needs.

[0112] In this embodiment, the head tracking, hand tracking and limb movement tracking of the user are performed according to the user posture image to obtain the posture tracking information. Specifically, a segmented 3D posture estimation method is used to complete the motion capture and posture estimation. The 3D posture estimation method includes the following steps:

[0113] Use the depth camera to obtain the depth image information and plane information of the current camera shooting area;

[0114] Use the VGG network to extract the plane information and obtain the eigenvalue;

[0115] Use the target tracking algorithm based on deep learning to complete the tracking of single or multiple people: input the extracted feature value, use the CNN network to predict the 2D joint position of the target rectangle in real time to obtain the corresponding Heatmaps——H {j,t} ;

[0116] Based on Heatmaps——H {j,t} And the depth image information is used to extract the connection relationship of each joint;

[0117] Heatmaps——H {j,t} The connection relationship of each joint is used as input, and the pre-trained RCNN network is used to map the 2D image pixels to the 3D surface model points to obtain the corresponding 3D joint positions;

[0118] Integrate 3D joint positions to get location-maps——X {j,t} , Y {j,t} , Z {j,t} ; Where j represents the joint point and t represents the time;

[0119] Based on heatmaps and locaiton maps, 2D key points and 3D posture estimation are extracted and combined to obtain the final motion capture and posture estimation data, thereby completing 3D human posture estimation and real-time motion capture.

[0120] In actual application, the data of motion capture and posture estimation will be fed back to simulation software such as CATIA, DELMIA, JACK in real time. By calculating the object relationship between the current task model and the background model, the model update and the implementation of interactive actions will be completed.

[0121] In actual application, the expression recognition model is obtained by machine learning through multiple sets of training data. The specific process includes:

[0122] Pupil diameter and fixation frequency are selected as eye features, and the effect of illumination on pupil diameter change is reduced by constructing an illumination reflection model.

[0123] The EOG signal is filtered with a 0.05-50 Hz bandpass filter to obtain the EOG signal with power spectrum changes at different scanning frequencies;

[0124] Through different external video and sound tests, the eye features and EOG signals of anger, pain, disgust, sadness, fear, surprise, happiness and joy are collected, and the corresponding expression types are used as label information to form the emotion training set data;

[0125] The optimal characteristic value of the two characteristic data corresponding to different emotions is found through the SVM classifier to complete the classification of eight basic emotions and obtain the expression recognition model.

[0126] In the subsequent use of the expression recognition model, it is only necessary to make a judgment based on the above classification results and the data received by the sensor to obtain the expression with the highest probability, so as to realize real-time update of the character's facial expression.

[0127] It should be noted that those skilled in the art should understand that the VR real-time communication interaction method combining 5G, expression tracking and beautification can not only be used in application scenarios of instant calls such as telephone booths, but also in application scenarios of entertainment activities, which is not limited in this embodiment.

[0128] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A VR real-time communication and interaction system combining 5G, expression tracking and beautification, comprising a cloud server, a plurality of cluster processing devices, a plurality of communication rooms and a plurality of user interaction devices, wherein the cloud server is respectively connected to the plurality of cluster processing devices, each cluster processing device is respectively connected to a plurality of adjacent user interaction devices, thereby forming a plurality of regional processing stations, and each user interaction device is arranged in a corresponding communication room, characterized in that: The user interaction device includes a first image acquisition device, a user processor, a head-mounted VR interaction device and a first communication module. The user processor is connected to the first image acquisition device and the first communication module respectively. The first communication module is also connected to the cluster processing device. The user processor transmits data with the cluster processing device through the first communication module. The user interaction device further comprises a laser sensor, which is arranged in the communication room and connected to the user processor; The laser sensor is used to locate the position of the user in the communication room, the user processor obtains the user position through the laser sensor, transmits the user position to the cloud server through the cluster processing device through the first communication module, and the cloud server forwards the user position to the cluster processing device connected to the communication party, which is one or more communication users with whom the current user has established communication interaction, and the cluster processing device connected to the communication party updates the relative position of the user simulation model in the environmental scene model according to the user position, and sends the position update information to the user processor of the communication party; The user interaction device further comprises an ambient light sensor, which is arranged in the communication room and connected to the user processor; The ambient light sensor is used to sense the light conditions around the user in the communication room. The user processor uses the ambient light sensor to obtain the user's ambient light information, and transmits the user's ambient light information to the cloud server through the cluster processing device via the first communication module. The cloud server forwards the user's ambient light information to the cluster processing device connected to the communication party, which is one or more communication users with whom the current user has established communication interaction. The user processor of the communication party adjusts the display brightness of the environmental scene model according to the user's ambient light information. The head-mounted VR interactive device is provided with a display device, a second image acquisition device, a micro-electrode sheet and a second communication module, wherein the micro-electrode sheet, the display device and the second image acquisition device are respectively connected to the second communication module, and the second communication module is also wirelessly connected to the first communication module; The microelectrode sheet is used to collect EOG signals; The first image acquisition device is used to acquire an environment image including a user's limbs, and the second image acquisition device is used to acquire a user's facial image, and the second image acquisition device is arranged on a head-mounted VR interaction device; The user processor is used to create a general body model and a facial model, identify the user's posture and expression to obtain posture tracking information and expression recognition type, the posture information includes head posture information, hand posture information and limb posture information; The cloud server is used to forward data and pre-build the environment scene model and user simulation model; The display device is used to display the environment scene model and the user simulation model; The cluster processing device connected to the communication party updates the posture and expression of the user simulation model according to the posture tracking information and the expression recognition type.

2. The VR real-time communication interactive system combining 5G, expression tracking and beautification according to claim 1 is characterized in that: The cloud server is used to match a communication party that has established a connection with the current user, where the communication party is one or more communication users that have established communication interaction with the current user; The user processor receives a preset environment scene type input by a user, and forwards it to a cloud server through a cluster processing device. The cloud server stores a plurality of preset environment scene types. The cloud server generates an environment scene model according to the preset environment scene type selected by the user through the user processor, matches the communication party connected to the current user, and sends the environment scene model, user simulation model, posture tracking information, and expression recognition type to the cluster processing device connected to the communication party.

3. The VR real-time communication interaction system combining 5G, expression tracking and beautification according to claim 1 is characterized in that: The user interaction device also includes a photosensitive sensor, a laser lighthouse base station, and a sensing handle. The laser lighthouse base station is arranged at both ends of the diagonal of the user interaction device. A laser sensor array for positioning is provided in the laser lighthouse base station. The photosensitive sensor is arranged on the ground and the sensing handle of the communication room, so that the laser sensor array and the photosensitive sensor constitute a laser lighthouse laser positioning system.

4. The VR real-time communication interaction system combining 5G, expression tracking and beautification according to claim 1 is characterized in that: The cluster processing device and the user interaction device are in a one-to-many relationship, and the data of each user interaction device in an area is solved and distributed by a cluster processing device deployed in the area.

5. The VR real-time communication interaction system combining 5G, expression tracking and beautification according to claim 1 is characterized in that: The head-mounted VR interactive device also includes a mirror frame, and the micro-electrode sheet, the display device, the second image acquisition device and the second communication module are all arranged on the mirror frame. The mirror frame is provided with a plurality of foam areas, and the foam areas are used to install the micro-electrode sheet.

6. A VR real-time communication interaction method combining 5G, expression tracking and beautification, characterized in that: A VR real-time communication interaction system combining 5G, expression tracking and beautification as described in any one of claims 1 to 5 is provided, and the method comprises the following steps: Model building steps: create a general body model and a facial model, match the facial model with the general body model to obtain a user simulation model; Scene building steps: select the target scene type and generate the environment scene model; Data collection steps: collecting environmental images including user limbs, user facial images, and EOG signals; Model information updating step: separating the environment image containing the user's limbs to obtain a user posture image, tracking the user's head, hands and limb movements according to the user posture image to obtain posture tracking information, tracking the user's expression according to the user's facial image to obtain eyeball features, combining the eyeball features and EOG signals as expression features and inputting them into the expression recognition model for prediction to obtain the expression recognition type; The expression recognition model is obtained by machine learning through multiple sets of training data, each set of training data includes expression features and label information identifying the expression recognition type corresponding to the expression features, and the expression recognition type is an identifier of multiple expression types; User synchronization step: match the communication party that has established a connection with the current user, send the environment scene model, user simulation model, posture tracking information, and expression recognition type to the communication party, and the communication party updates the posture and expression of the user simulation model according to the posture tracking information and expression recognition type; Image adjustment step: Beautify the image of the communication party by rendering the user simulation model according to the preset beautification type selected by the user, and the beautification processing includes face thinning, skin smoothing, and whitening.

7. The VR real-time communication interaction method combining 5G, expression tracking and beautification according to claim 6 is characterized in that: The expression recognition model is obtained by machine learning through multiple sets of training data, and the specific process includes: Pupil diameter and fixation frequency are selected as eye features, and the effect of illumination on pupil diameter change is reduced by constructing an illumination reflection model. The EOG signal is filtered with a 0.05-50 Hz bandpass filter to obtain the EOG signal with power spectrum changes at different scanning frequencies; Through different external video and sound tests, the eye features and EOG signals of anger, pain, disgust, sadness, fear, surprise, happiness and joy are collected, and the corresponding expression types are used as label information to form the emotion training set data; The SVM classifier is used to find the optimal feature value of the two feature data corresponding to different emotions to complete the classification of expression recognition types and obtain the expression recognition model.

8. The VR real-time communication interaction method combining 5G, expression tracking and beautification according to claim 6 is characterized in that: The step of tracking the user's head, hand and limb movements according to the user's posture image to obtain posture tracking information specifically includes using a segmented 3D posture estimation method to complete motion capture and posture estimation. The 3D posture estimation method includes the following steps: Use the depth camera to obtain the depth image information and plane information of the current camera shooting area; Use the VGG network to extract the plane information and obtain the eigenvalue; Use the target tracking algorithm based on deep learning to complete the tracking of single or multiple people: input the extracted feature value, use the CNN network to predict the 2D joint position of the target rectangle in real time to obtain the corresponding Heatmaps——H {j,t} ; Based on Heatmaps——H {j,t} And the depth image information is used to extract the connection relationship of each joint; Heatmaps——H {j,t} The connection relationship of each joint is used as input, and the pre-trained RCNN network is used to map the 2D image pixels to the 3D surface model points to obtain the corresponding 3D joint positions; Integrate 3D joint positions to get location-maps——X {j,t} , Y {j,t} , Z {j,t} ; Where j represents the joint point and t represents the time; Based on Heatmaps and locaiton maps, 2D key points and 3D pose estimation are extracted and combined to obtain the final motion capture and pose estimation data.

Images