Communication method and apparatus, wearable device, and storage medium

Abstract

The present disclosure relates to a communication method and device, a wearable device and a storage medium. The communication method is applied to a wearable device and includes: obtaining first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera; wherein the first feature data represents data associated with the eyes of a wearer; the second feature data represents data associated with the face and body of the wearer; based on the first feature data and the second feature data, an image of the wearer is reconstructed to obtain a reconstructed image; the reconstructed image is used to represent a virtual image of the wearer; and the reconstructed image is sent to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image. The embodiment of the present disclosure can improve the wearing comfort of the wearable device and reduce the anxiety of using electricity.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computer vision, and particularly relates to a communication method and device, a wearable device, and a storage medium. BACKGROUND

[0002] At present, remote communication technology based on wearable devices such as AR glasses or VR glasses is a new application. In the process of remote communication by the wearable device, if the remote communication is realized by using the eye movement camera for shooting the eye and the camera for shooting the face built in the wearable device, since the camera for shooting the face needs to be additionally introduced into the wearable device, the power consumption and weight of the wearable device are increased, the wearing comfort is reduced, and the power anxiety is increased; if the remote communication is realized by using an external camera alone, since the eye of the wearer is blocked by the glasses, the eye recognition accuracy is reduced or cannot be recognized. SUMMARY

[0003] To overcome the problems in the related art, the present disclosure provides a communication method and device, a wearable device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a communication method is provided, at least comprising:

[0005] obtaining first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera; wherein the first feature data represents data associated with the eye of the wearer; and the second feature data represents data associated with the face and body of the wearer;

[0006] reconstructing an image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image; the reconstructed image is used to represent a virtual image of the wearer;

[0007] sending the reconstructed image to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0008] In some embodiments, the first feature data includes eye feature data; the second feature data includes face feature data and body feature data; and the reconstructing an image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image includes:

[0009] performing feature fusion on the eye feature data and the face feature data to obtain face fusion data;

[0010] obtaining model driving data based on the face fusion data and the body feature data;

[0011] obtaining the reconstructed image based on the model-driven data and voice data collected by the wearable device.

[0012] In some embodiments, the feature fusion of the eye feature data and the face feature data obtains face fusion data, including:

[0013] aligning the face feature data with standard face data to obtain face alignment data;

[0014] obtaining an eye correction scale based on initial eye data in the face alignment data and the eye feature data;

[0015] correcting the eye feature data based on the eye correction scale to obtain eye correction data;

[0016] fusing the eye correction data into the face alignment data to obtain the face fusion data.

[0017] In some embodiments, the aligning of the face feature data with standard face data to obtain face alignment data includes:

[0018] obtaining an overall optimization error of face alignment based on a plurality of face key points of the face feature data and a plurality of standard key points of the standard face data;

[0019] performing nonlinear optimization on the overall optimization error to obtain the face alignment data.

[0020] In some embodiments, the determination of an eye correction scale based on initial eye data in the face alignment data and the eye feature data includes:

[0021] obtaining a relative extrinsic parameter of the eye movement camera and the external camera;

[0022] obtaining an actual distance between two eye key points based on the relative extrinsic parameter, center coordinates of the two eye key points contained in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points;

[0023] obtaining the eye correction scale based on a ratio between an initial distance between the two eye key points in the initial eye data and the actual distance.

[0024] In some embodiments, the obtaining of an actual distance between two eye key points based on the relative extrinsic parameter, center coordinates of the two eye key points contained in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points includes:

[0025] obtain three-dimensional coordinates of the two eye key points based on the center coordinates of the two eye key points and the depth distance;

[0026] obtain an actual distance between the two eye key points based on the three-dimensional coordinates of the two eye key points and the relative extrinsic parameter.

[0027] In some embodiments, the model-driven data includes expression weights and coordinates of a plurality of body key points; and the obtaining the model-driven data based on the face fusion data and the body feature data includes:

[0028] obtain expression weights corresponding to a target expression based on the target expression represented by the face fusion data, a plurality of reference expressions, and weights of the plurality of reference expressions;

[0029] map the plurality of body key points of the body feature data to a three-dimensional space to obtain coordinates of the plurality of body key points.

[0030] In some embodiments, the obtaining the reconstructed image based on the model-driven data and voice data collected by the wearable device includes:

[0031] obtain a virtual driving model;

[0032] drive the virtual driving model based on the model-driven data and the voice data collected by the wearable device to obtain the reconstructed image.

[0033] In some embodiments, the obtaining first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera includes:

[0034] send a clock synchronization signal to a camera device where the external camera is located or receive the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize clocks of the eye movement camera and the external camera;

[0035] collect the first feature data and the second feature data based on the eye movement camera and the external camera after clock synchronization.

[0036] According to a second aspect of the embodiments of the present disclosure, a communication apparatus is provided, applied to a wearable device, and at least includes:

[0037] The feature acquisition module is configured to acquire first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera; wherein the first feature data represents data associated with the eye of the wearer; and the second feature data represents data associated with the face and body of the wearer.

[0038] The reconstructed image module is configured to perform image reconstruction on the wearer based on the first feature data and the second feature data to obtain a reconstructed image.

[0039] The communication module is configured to send the reconstructed image to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0040] According to a third aspect of an embodiment of the present disclosure, a wearable device is provided, comprising at least:

[0041] a processor;

[0042] a memory for storing processor-executable instructions;

[0043] The processor is configured to perform the communication method as described in the first aspect above.

[0044] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, comprising:

[0045] When the instructions in the storage medium are executed by the processor of the wearable device, the wearable device can perform the communication method as described in the first aspect above.

[0046] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0047] The embodiments of the present disclosure obtain a reconstructed image based on the first feature data and the second feature data collected by the eye movement camera and the external camera of the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image. In this way, the embodiments of the present disclosure do not need to additionally set a camera for collecting the face and body in the wearable device, which can reduce the power consumption and weight increase of the wearable device caused by the additional camera, and improve the wearing comfort of the wearable device and reduce the anxiety of using electricity.

[0048] Moreover, the embodiment of the present disclosure is to reconstruct the image of the wearer by combining the eye movement camera and the external camera, instead of reconstructing the image of the wearer by the external camera alone, so as to reduce the inaccuracy of the external collected eye data caused by the occlusion of the eye of the wearer by the wearable device, improve the eye recognition accuracy in the image reconstruction process, and further reconstruct a better virtual image of the wearer, and improve the immersive experience of the communication of the wearable device.

[0049] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, together with the description.

[0051] Figure 1 is a flowchart of a communication method according to an example embodiment.

[0052] Figure 2 is a block diagram of an image reconstruction process according to an example embodiment.

[0053] Figure 3 is a block diagram of a communication device according to an example embodiment.

[0054] Figure 4 is a block diagram of a wearable device according to an example embodiment. DETAILED DESCRIPTION

[0055] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0056] The embodiment of the present disclosure provides a communication method. Figure 1 is a flowchart of a communication method according to an example embodiment. The communication method is applied to a scenario of remote communication based on a wearable device, so as to make the remote communication more immersive, and realize more natural and intuitive remote communication. For example, in the process that the wearer wears the wearable device and communicates with the communication device through the wearable device, the communication based on the reconstructed image can display the virtual image of the wearer in the field of view through virtual reality technology, and further realize the face-to-face remote communication effect. For example, Figure 1As shown, the communication method performed by the wearable device comprises the following steps:

[0057] In step S101, first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera are acquired; wherein the first feature data represents data associated with the wearer's eyes; and the second feature data represents data associated with the wearer's face and body.

[0058] In step S102, based on the first feature data and the second feature data, an image of the wearer is reconstructed to obtain a reconstructed image.

[0059] In step S103, the reconstructed image is sent to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0060] In the embodiments of the present disclosure, the wearable device can communicate remotely with the communication device, and the wearable device can include an Augmented Reality (AR) glasses device or a Virtual Reality (VR) glasses device; the communication device can include a smartphone, a notebook, a tablet computer, an AR glasses device or a VR glasses device, and the embodiments of the present disclosure do not limit this.

[0061] Taking the wearable device as an AR glasses device and the communication device as a smartphone as an example, the AR glasses device can obtain a virtual image of the wearer representing the AR glasses device by performing the communication method, and during the communication between the AR glasses device and the smartphone, the virtual image of the wearer can be sent to the communication device, thereby realizing more natural and intuitive remote communication between the AR glasses device and the smartphone.

[0062] In step S101, the eye movement camera is a built-in camera of the wearable device, and the first feature data collected by the eye movement camera represents data associated with the wearer's eyes. For example, the first feature data can include feature data of eye key points; wherein the feature data of the eye key points includes eye key point size data and position data; the eye key points include eye key points such as eyeballs, upper and lower eyelids, or eyebrows, and the embodiments of the present disclosure do not limit this.

[0063] The external camera is not a built-in camera of the wearable device, and the second feature data collected by the external camera represents data associated with the wearer's face and body. For example, the second feature data can include feature data of face key points and feature data of body key points; wherein the face key points include key points such as mouth, nose, and cheek; and the body key points include key points such as arms, legs, fingers, and head, and the embodiments of the present disclosure do not limit this.

[0064] It should be noted that the external camera can be a built-in camera of a camera device, and the camera device and the wearable device belong to two independent devices. The camera device can include a smart phone, a notebook, a tablet computer, a smart watch, or a monitoring camera device, and the like, and the embodiments of the present disclosure do not limit this.

[0065] In the embodiments of the present disclosure, the first feature data and the second feature data can be obtained by key point detection. For the first feature data, the key point detection method can include a convolutional neural network (CNN) or a recurrent neural network (RNN) detection method; and for the second feature data, the key point detection method can include a deep learning method, such as OpenPose and the like.

[0066] Here, since the eye movement camera and the external camera belong to cameras on different devices, clock synchronization needs to be performed on the eye movement camera and the external camera before image reconstruction of the wearer. In some embodiments, the first feature data collected by the eye movement camera of the wearable device and the second feature data collected by the external camera other than the eye movement camera include:

[0067] sending a clock synchronization signal to a camera device where the external camera is located or receiving the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize the clock of the eye movement camera and the external camera;

[0068] Based on the eye movement camera and the external camera after clock synchronization, the first feature data and the second feature data are obtained.

[0069] That is, the embodiments of the present disclosure synchronize and calibrate the clock of the eye movement camera and the external camera, so that the first feature data and the second feature data collected by the two cameras can ensure data synchronization, and in the process of driving the preset virtual driving model based on the first feature data and the second feature data, the consistency of driving can be ensured, and a better reconstructed image can be obtained.

[0070] In the embodiments of the present disclosure, the clock synchronization signal can be sent at a preset time interval. The preset time can be set in the range of 150 milliseconds to 250 milliseconds, and the embodiments of the present disclosure do not limit this.

[0071] It should be noted that, in the case of the wearable device as the data synchronization center, the wearable device can send a clock synchronization signal to the camera device, so that the external camera in the camera device can be clock-synchronized with the eye movement camera, and thus the data synchronization of the first feature data collected by the eye movement camera and the second feature data collected by the external camera is realized.

[0072] In the case of the camera device as the data synchronization center, the camera device can send a clock synchronization signal to the wearable device, so that the eye movement camera of the wearable device can be clock-synchronized with the external camera, and thus the data synchronization of the first feature data collected by the eye movement camera and the second feature data collected by the external camera is also realized.

[0073] In the embodiments of the present disclosure, the wearable device and the camera device can be connected under the same communication network, so that the clock synchronization based on the same communication network can make the data transmission of clock synchronization faster and the data transmission delay lower.

[0074] In step S102, the wearer is image-reconstructed based on the first feature data and the second feature data, and a reconstructed image obtained by the image reconstruction represents a virtual image of the wearer. In remote communication, the virtual image is displayed in the field of view, so as to realize immersive communication experience between the user of the communication device and the wearer of the wearable device. The virtual image of the wearer can be a two-dimensional image or a three-dimensional image, which is not limited in the embodiments of the present disclosure.

[0075] In the embodiments of the present disclosure, the first feature data represents data associated with the eye of the wearer, and the corresponding first feature data can reconstruct the eye of the wearer; and the second feature data represents data associated with the face and body of the wearer, and the corresponding second feature data can reconstruct the face and body of the wearer. In this way, based on the reconstructed eye of the wearer, the reconstructed face and body of the wearer, the virtual image of the wearer can be obtained.

[0076] It should be noted that the reconstructed image obtained by the image reconstruction of the wearer can be an image sequence, and the virtual image of the wearer constructed based on the image sequence can be linked, which can present the animation effect of the virtual image of the wearer. The virtual image can correspond to an animal model digital person, a cartoon digital person, or a hyper-realistic digital person, etc. in a digital person, which is not limited in the embodiments of the present disclosure.

[0077] In step S103, the reconstructed image is sent to the communication device in communication with the wearable device, which can include that the wearable device sends the reconstructed image to an intermediate server device, the server device relays the reconstructed image to the communication device, and the communication device can display the virtual image of the wearer to achieve immersive remote communication between the wearable device and the communication device.

[0078] Exemplarily, in a video conference or remote call scenario, the wearable device can send the reconstructed image to the communication device, and the virtual image of the wearer is displayed on the display screen of the communication device, so that the user of the communication device can see the virtual image of the wearer in the field of view. Of course, the feature data of the user of the communication device can also be collected and sent to the wearable device after obtaining the virtual image of the user of the communication device based on the feature data of the user of the communication device, so that the wearer of the wearable device can also see the virtual image of the user of the communication device. In this way, the immersive conference experience or remote call experience between the wearer of the wearable device and the user of the communication device can be realized through the above-mentioned manner.

[0079] It should be noted that the reconstructed image can also be used for self-display and sent to other display devices (such as a television device) for display, and the embodiments of the present disclosure do not limit this.

[0080] In the embodiments of the present disclosure, the first feature data and the second feature data obtained by the eye movement camera and the external camera of the wearable device are used to obtain the reconstructed image, so that the wearable device and the communication device can communicate based on the reconstructed image. In this way, the embodiments of the present disclosure do not need to additionally set a camera for collecting faces and bodies in the wearable device, which can reduce the power consumption and weight increase of the wearable device caused by the additional camera, improve the wearing comfort of the wearable device, and reduce the anxiety of using electricity. And, the embodiments of the present disclosure reconstruct the image of the wearer by combining the eye movement camera and the external camera, rather than using the external camera alone to reconstruct the image of the wearer, which can reduce the inaccuracy of the external collected eye data caused by the occlusion of the eye of the wearer by the wearable device, improve the eye recognition accuracy in the image reconstruction process, and further reconstruct a better virtual image of the wearer, thereby improving the immersive experience of the communication of the wearable device.

[0081] In some embodiments, the image reconstruction of the wearer based on the first feature data and the second feature data to obtain a reconstructed image comprises:

[0082] performing feature fusion on the eye feature data and the face feature data to obtain face fusion data;

[0083] obtaining model driving data based on the face fusion data and the body feature data;

[0084] obtaining the reconstructed image based on the model driving data and the voice data collected by the wearable device.

[0085] In the embodiments of the present disclosure, the first feature data includes eye feature data, and the second feature data includes face feature data and body feature data. In the process of collecting the second feature data by the external camera, the face feature data collected by the external camera is not accurate due to the occlusion of the wearer's eyes by the wearable device. Therefore, the eye feature data collected by the eye movement camera and the face feature data are fused.

[0086] It should be noted that, in the process of fusing the eye feature data and the face feature data, the eye feature data can be input into the first layer model level fusion to obtain first fusion data, and then the first fusion data and the face feature data are input into the second layer model level fusion to obtain second fusion data, and finally the second fusion data is input into the full connection layer to obtain the face fusion data. The face fusion data can also be obtained by first aligning the face feature data with the standard face data, and then correcting the face alignment data obtained by the alignment.

[0087] In the embodiments of the present disclosure, after the face fusion data is obtained by feature fusion, model driving data can be obtained based on the face fusion data and the body feature data. The model driving data is used to drive the virtual driving model, so that the virtual image of the wearer after reconstruction can present an animation effect in the communication process.

[0088] It should be noted that, in the communication process, it is usually a voice call, which can be a social voice call or a conference voice call, and the embodiments of the present disclosure do not limit this.

[0089] Here, the embodiments of the present disclosure need to collect voice data after obtaining the model driving parameters, and obtain the reconstructed image based on the model driving data and the voice data. That is, in the voice call process, the communication device can not only see the virtual image of the wearer but also synchronously hear the virtual image outputting voice data. In this way, the immersive experience of the communication of the wearable device is improved.

[0090] In some embodiments, the fusing the eye feature data and the face feature data to obtain face fusion data comprises:

[0091] aligning the face feature data with standard face data to obtain face alignment data;

[0092] obtaining eye correction scale based on the eye initial data in the face alignment data and the eye feature data;

[0093] correcting the eye feature data based on the eye correction scale to obtain eye correction data;

[0094] Fusing the eye correction data into the face alignment data to obtain the face fusion data.

[0095] In the embodiments of the present disclosure, the face of the wearer presents different shapes under different expressions, and thus the face feature data needs to be aligned. The face alignment can include alignment based on a face model or alignment based on deep learning of a convolutional neural network. The face model can include an active shape model (ASM) or an active appearance model (AAM), and the deep learning of the convolutional neural network can include a DeepID convolutional neural network or a DeepFace convolutional neural network.

[0096] In some embodiments, the aligning the face feature data with standard face data to obtain face alignment data includes:

[0097] obtaining an overall optimization error of face alignment based on a plurality of face key points of the face feature data and a plurality of standard key points of the standard face data;

[0098] performing nonlinear optimization on the overall optimization error to obtain the face alignment data.

[0099] In the embodiments of the present disclosure, the overall optimization error of face alignment can be obtained based on a plurality of face key points of the face feature data and a plurality of standard key points of the standard face data, which can include: obtaining data error, smooth error and key point error of face alignment based on the plurality of face key points and the plurality of standard key points; and obtaining the overall optimization error based on the data error, the smooth error and the key point error.

[0100] Here, the data error E d (X) can be obtained based on the plurality of face key points and the plurality of standard key points by formula (1), the smooth error E s (X) can be obtained by formula (2), and the key point error E l (X) can be obtained by formula (3).

[0101]

[0102] wherein X represents the face key points; v i represents a vertex of the i th standard key point; V represents a set of vertices; w i represents the i th weight; τ represents a point closest to the vertex of the standard key point in the plurality of face key points; dist() represents a distance function for calculating a distance; X i represents a corresponding transformation matrix of v i .

[0103]

[0104] wherein i and j are two labels of the transformed facial landmarks; X i is represented as a transformation matrix corresponding to v i is represented as a transformation matrix corresponding to v j is represented as a transformation matrix corresponding to v j is represented as a transformation matrix corresponding to v G is represented as a 4*4 diagonal matrix.

[0105]

[0106] wherein l is represented as a facial landmark, and L is represented as a set of facial landmarks and transformation matrices.

[0107] It should be noted that in the face alignment process, a plurality of facial landmarks can be selected to match a plurality of standard landmarks, and the number of the selected landmarks can be set in a range of 15 to 25, and the embodiments of the present disclosure do not limit this.

[0108] In the embodiments of the present disclosure, the overall optimization error is obtained based on the data error, the smoothing error and the landmark error, which can include: obtaining a first product of the smoothing error and a first coefficient; obtaining a second product of the landmark error and a second coefficient; and obtaining the overall optimization error based on a sum of the data error, the first product and the second product.

[0109] Here, the overall optimization error E can be obtained by formula (4).

[0110] E = E d (X) + a * E s (X) + b * E l (X) (4)

[0111] wherein a represents the first coefficient, and b represents the second coefficient.

[0112] In the embodiments of the present disclosure, after obtaining the overall optimization error, the overall optimization error can be subjected to nonlinear optimization to obtain the face alignment data. The nonlinear optimization method can include a nonlinear least square method or a Gauss-Newton method, and the embodiments of the present disclosure do not limit this.

[0113] It should be noted that after obtaining the face alignment data, since the eye initial data in the face alignment data and the eye feature data belong to different dimensional data, a correction scale needs to be calculated based on the eye initial data and the eye feature data.

[0114] In some embodiments, the determining the eye correction scale based on the eye initial data and the eye feature data in the face alignment data includes:

[0115] obtaining a relative extrinsic parameter of the eye movement camera and the external camera;

[0116] obtaining an actual distance between the two eye key points based on the relative extrinsic parameter, center coordinates of the two eye key points contained in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points;

[0117] obtaining the eye correction scale based on a ratio between an initial distance between the two eye key points in the eye initial data and the actual distance.

[0118] In the embodiments of the present disclosure, since the poses of the eye movement camera and the external camera in the real space are different, the relative extrinsic parameter between the two cameras needs to be determined, and then the mapping relationship between the feature data obtained by different cameras can be determined, so that the image reconstruction can be realized by jointly using the eye movement camera and the external camera. The relative extrinsic parameter of the eye movement camera and the external camera refers to the conversion relationship of the camera coordinate systems of the eye movement camera and the external camera.

[0119] In some embodiments, obtaining the relative extrinsic parameter of the eye movement camera and the external camera can include: determining the intrinsic parameter and the extrinsic parameter of the eye movement camera, and the intrinsic parameter and the extrinsic parameter of the external camera through a reference object such as a calibration board or a calibration rod; determining the mapping relationship between the eye movement camera and the external camera by comparing the image of the reference object obtained by the eye movement camera and the image of the reference object obtained by the external camera; and obtaining the relative extrinsic parameter of the eye movement camera and the external camera based on the mapping relationship, the intrinsic parameter and the extrinsic parameter of the eye movement camera, and the intrinsic parameter and the extrinsic parameter of the external camera.

[0120] In other embodiments, obtaining the relative extrinsic parameter of the eye movement camera and the external camera can include: determining a co-view image in the images obtained by the eye movement camera and the external camera; determining the coordinates of the co-view image in the world coordinate system; then determining the initial value extrinsic parameter of the eye movement camera and the external camera; establishing the mapping relationship of the co-view image in the pixel coordinate system or the image coordinate system of different cameras; and determining the relative extrinsic parameter of the eye movement camera and the external camera based on the mapping relationship and the initial value extrinsic parameter.

[0121] In the embodiments of the present disclosure, after obtaining the relative extrinsic parameter, the actual distance between the two eye key points can be obtained based on the relative extrinsic parameter and the center coordinates of the two eye key points. The two eye key points can include two eyeball key points, two eyelid key points, or two eyebrow key points, and the embodiments of the present disclosure do not limit this.

[0122] In some embodiments, the obtaining of the actual distance between the two eye key points based on the relative extrinsic parameter, the center coordinates of the two eye key points contained in the eye feature data, and the preset depth distance of the eye movement camera relative to the eye key points includes:

[0123] Based on the center coordinates of the two eye key points and the depth distance, three-dimensional coordinates of the two eye key points are obtained; based on the three-dimensional coordinates of the two eye key points and the relative extrinsic parameters, an actual distance between the two eye key points is obtained.

[0124] In the embodiments of the present disclosure, the two eye key points include a left eye key point and a right eye key point; the three-dimensional coordinates of the left eye key point can be obtained based on the product of the center coordinates of the left eye key point, the depth distance and a third coefficient; the three-dimensional coordinates of the right eye key point can be obtained based on the product of the center coordinates of the right eye key point, the depth distance and the third coefficient.

[0125] Here, the three-dimensional coordinates of the left eye key point P 3Dl can be obtained by formula (5). 3Dr

[0126] P 3Dl = K -1 *d*P uvl (5)

[0127] P 3Dr = K -1 *d*P uvr (6)

[0128] Wherein, K represents the third coefficient; d represents the depth distance; P uvl represents the center coordinates of the left eye key point; P uvr represents the center coordinates of the right eye key point.

[0129] In the embodiments of the present disclosure, based on the three-dimensional coordinates of the two eye key points and the relative extrinsic parameters, the actual distance between the two eye key points can include: obtaining a third product between the relative extrinsic parameters and the three-dimensional coordinates of the right eye key point; based on the difference between the three-dimensional coordinates of the left eye key point and the third product, the actual distance is obtained.

[0130] Here, the actual distance dist can be obtained by formula (7). Wherein, T lr represents the relative extrinsic parameters.

[0131] dist = P 3Dl -T lr *P 3Dr (7)

[0132] In the embodiments of the present disclosure, after the actual distance is obtained, the eye correction scale can be obtained by formula (7).

[0133] S = Fddist (8) ​

[0134] wherein S represents an eye correction scale; and Fd represents an initial distance.

[0135] In the embodiments of the present disclosure, the eye correction scale is used to correct the eye feature data, and after obtaining the eye correction data, the eye correction data is fused into the face alignment data to obtain the face fusion data, which can include: replacing the eye correction data into the face alignment data to obtain the face fusion data.

[0136] In this way, in the embodiments of the present disclosure, the face alignment data is obtained through alignment first, then the eye correction data is obtained based on the face alignment data obtained through alignment, and finally the face fusion data is obtained based on the fusion of the eye correction data, so that the accuracy of fusion can be improved through alignment and correction.

[0137] In some embodiments, the model-driven data includes an expression weight and coordinates of a plurality of body key points; and the model-driven data is obtained based on the face fusion data and the body feature data, including:

[0138] The expression weight corresponding to the target expression is obtained based on the target expression represented by the face fusion data, a plurality of reference expressions and weights of the plurality of reference expressions.

[0139] The plurality of body key points of the body feature data are mapped into a three-dimensional space to obtain coordinates of the plurality of body key points.

[0140] In the embodiments of the present disclosure, the expression weight corresponding to the target expression is obtained based on the target expression represented by the face fusion data, a plurality of reference expressions and weights of the plurality of reference expressions, which can include: obtaining a plurality of reference expressions and weights corresponding to the plurality of reference expressions to obtain an expression vector; obtaining a difference value based on a difference between the expression vector and the target expression, and obtaining the expression weight based on a norm processing of the difference value.

[0141] Here, the expression weight f(w) can be obtained through formula (9).

[0142]

[0143] wherein A represents a plurality of reference expressions; [w0...w n-1 ] T represent weights of the plurality of reference expressions; and B represents a target expression.

[0144] In the embodiments of the present disclosure, the model-driven data for the face is an expression weight, and the model-driven data for the body is a plurality of coordinates of a plurality of body key points obtained by mapping the plurality of body key points to a three-dimensional space. Here, the embodiments of the present disclosure can first establish a spatial coordinate system, and then obtain the coordinates of the plurality of body key points through the positions of the plurality of body key points and a spatial transformation matrix. In this way, the embodiments of the present disclosure can obtain the model-driven data, and further drive the virtual driving model, so that the virtual image of the wearer after reconstruction can present an animation effect in the communication process.

[0145] In some embodiments, the obtaining the reconstructed image based on the model-driven data and the voice data collected by the wearable device comprises:

[0146] obtaining a virtual driving model;

[0147] driving the virtual driving model based on the model-driven data and the voice data collected by the wearable device to obtain the reconstructed image.

[0148] In the embodiments of the present disclosure, the virtual driving model is a model that can be driven by a simulated driving parameter to present an animation effect. The obtaining the virtual driving model can include: obtaining a three-dimensional digital human model; calculating a task expression or artificially modeling for the face of the three-dimensional digital human model; and then binding a standard skeleton point for the body of the three-dimensional digital human model, and performing skinning and weight calculation on each skeleton point of the bound model, thereby obtaining the virtual driving model.

[0149] The driving the virtual driving model based on the model-driven data and the voice data collected by the wearable device to obtain the reconstructed image can include: driving the virtual driving model based on the model-driven data and the voice data collected by the wearable device, and then rendering the driven model through a rendering engine to obtain the reconstructed image. The rendering engine can include an Unreal Engine (UE) or a Unity engine, and the embodiments of the present disclosure are not limited thereto.

[0150] It should be noted that driving the virtual driving model with the model-driven data and the language data together can realize that the virtual image of the wearer can also output voice data synchronously when the wearer is seen, thereby improving the immersive experience of the communication of the wearable device.

[0151] In order to better understand the communication method in one or more embodiments of the present application, the embodiments of the present disclosure further illustrate as follows:

[0152] As Figure 2As shown, the embodiment of the present disclosure can first acquire a three-dimensional digital human model, and model binding is performed on the three-dimensional digital human to obtain a virtual driving model; then the clock synchronization is performed on the eye movement camera and the external camera, and the first feature data and the second feature data are collected; then the eye feature data included in the first feature data and the facial feature data included in the second feature data are fused to obtain facial fusion data; then the model driving data is obtained based on the facial fusion data and the body feature data; then the virtual driving model is driven based on the model driving data and the voice data collected by the wearable device; and finally, the rendered image is obtained by rendering the driven model through a rendering engine.

[0153] In this way, through the clock synchronization of the eye movement camera and the external camera, the feature fusion of the eye feature data and the facial feature data, the model driving data calculation, the driving of the virtual driving model, and finally the rendering output, the virtual image of the wearer can be obtained by image reconstruction of the wearer through the eye movement camera and the external camera. It can be seen that the embodiment of the present disclosure does not need to additionally set a camera for collecting the face and the body in the wearable device, which can reduce the power consumption and weight increase of the wearable device caused by the additional camera, and improve the wearing comfort and reduce the power anxiety of the wearable device.

[0154] Moreover, the embodiment of the present disclosure can also combine the eye movement camera and the external camera to perform image reconstruction on the wearer, instead of using the external camera alone to perform image reconstruction on the wearer, which can reduce the inaccuracy of the external collected eye data caused by the occlusion of the eye of the wearer by the wearable device, improve the eye recognition accuracy in the image reconstruction process, and further reconstruct a better virtual image of the wearer, thereby improving the immersive experience of the communication of the wearable device.

[0155] The embodiment of the present disclosure also provides a communication device applied in a wearable device. Figure 3 FIG. 1 is a block diagram of a communication device according to an exemplary embodiment. As shown in the figure, the communication device 1000 includes: Figure 3

[0156] The feature acquisition module 1001 is configured to acquire first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera; wherein the first feature data represents data associated with the eye of the wearer; and the second feature data represents data associated with the face and body of the wearer;

[0157] The reconstructed image module 1002 is configured to perform image reconstruction on the wearer based on the first feature data and the second feature data to obtain a reconstructed image.

[0158] ​The communication module 1003 is configured to send the reconstructed image to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0159] In some embodiments, the first feature data includes eye feature data; the second feature data includes face feature data and body feature data; and the reconstructed image module 1002 includes:

[0160] The fusion module is configured to perform feature fusion on the eye feature data and the face feature data to obtain face fusion data.

[0161] The driving calculation module is configured to obtain model driving data based on the face fusion data and the body feature data.

[0162] The data reconstruction module is configured to obtain the reconstructed image based on the model driving data and voice data collected by the wearable device.

[0163] In some embodiments, the fusion module is further configured to align the face feature data with standard face data to obtain face alignment data; obtain an eye correction scale based on eye initial data in the face alignment data and the eye feature data; correct the eye feature data based on the eye correction scale to obtain eye correction data; and fuse the eye correction data into the face alignment data to obtain the face fusion data.

[0164] In some embodiments, the fusion module is further configured to obtain an overall optimization error of face alignment based on a plurality of face key points of the face feature data and a plurality of standard key points of the standard face data; and perform nonlinear optimization on the overall optimization error to obtain the face alignment data.

[0165] In some embodiments, the fusion module is further configured to obtain a relative external parameter of the eye movement camera and the external camera; obtain an actual distance between two eye key points based on the relative external parameter, center coordinates of the two eye key points contained in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points; and obtain the eye correction scale based on a ratio between an initial distance between the two eye key points in the eye initial data and the actual distance.

[0166] In some embodiments, the fusion module is further configured to obtain three-dimensional coordinates of the two eye key points based on the center coordinates of the two eye key points and the depth distance; and obtain the actual distance between the two eye key points based on the three-dimensional coordinates of the two eye key points and the relative external parameter.

[0167] In some embodiments, the model-driven data comprises expression weights and coordinates of a plurality of body key points; the driving calculation module is configured to obtain expression weights corresponding to a target expression based on the target expression represented by the facial fusion data, a plurality of reference expressions and weights of the plurality of reference expressions; and map the plurality of body key points of the body feature data to a three-dimensional space to obtain coordinates of the plurality of body key points.

[0168] In some embodiments, the data reconstruction module is further configured to obtain a virtual driving model; and drive the virtual driving model based on the model-driven data and voice data collected by the wearable device to obtain the reconstructed image.

[0169] In some embodiments, the feature acquisition module 1001 is further configured to send a clock synchronization signal to a camera device where the external camera is located or receive the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize the eye movement camera and the external camera in time; and the first feature data and the second feature data are obtained based on the eye movement camera and the external camera after time synchronization.

[0170] The specific manners in which the various modules in the above-described embodiments perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0171] Figure 4 is a block diagram of a wearable device according to an exemplary embodiment. For example, the wearable device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0172] Referring to Figure 4 , the wearable device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0173] The processing component 802 usually controls overall operations of the wearable device 800, such as operations associated with displaying, phone calling, data communication, camera operation, and recording operation. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0174] Memory 804 is configured to store various types of data to support operations of the device 800. Examples of such data include instructions for any application or methods operating on the device 800, contact data, phonebook data, messages, pictures, videos, and so on. The memory 804 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks.

[0175] Power supply component 806 supplies electrical power for the various components of the device 800. Power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing electrical power for the device 800.

[0176] Multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a back camera. The front and / or back camera can receive external multimedia data when the device 800 is in an operation mode such as a photographing mode or a video mode. Each of the front and back camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0177] Audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) configured to receive external audio signals when the device 800 is in an operation mode such as a call mode, a recording mode and a voice recognition mode. The received audio signals can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0178] I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0179] The sensor component 814 includes one or more sensors to provide status assessment of various aspects of the wearable device 800. For example, the sensor component 814 can detect an on / off status of the device 800, relative positioning of components, such as a display and a keypad of the wearable device 800, a change of location of the wearable device 800 or a component of the wearable device 800, presence or absence of user contact with the wearable device 800, orientation or acceleration / deceleration / g-force and temperature changes of the wearable device 800. The sensor component 814 can include proximity sensor configured to detect presence of nearby objects without any physical contact. The sensor component 814 can further include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0180] The communication component 816 is configured to facilitate wired or wireless communication between the wearable device 800 and other devices. The wearable device 800 can access a wireless network based on a communication standard, such as WiFi, 3G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0181] In an exemplary embodiment, the wearable device 800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic elements to perform the above-described methods.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the wearable device 800 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0183] A kind of non-transitory computer readable storage medium, when the instruction in the storage medium is executed by the processor of wearable device, it enables wearable device to execute communication method, the method includes: obtaining the first feature data that the eye movement camera of the wearable device is collected and the second feature data that the external camera other than the eye movement camera is collected;Wherein, the first feature data is characterized as the data associated with the eye of wearer;The second feature data is characterized as the data associated with the face and body of the wearer;

[0184] Based on the first feature data and the second feature data, the wearer is reconstructed to image to obtain reconstructed image;The reconstructed image is used to represent the virtual image of the wearer;

[0185] The reconstructed image is sent to the communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0186] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. This application is intended to cover any variations, uses or adaptations of the disclosure that are deemed to fall within the general principles of the disclosure and include examples of the known art or customarily used in the art. The specification and examples are to be regarded as illustrative only and the true scope and spirit of the disclosure is indicated by the following claims.

[0187] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A communication method characterized by comprising: Applied to a wearable device, comprising: obtaining first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera; wherein the first feature data represents data associated with the wearer's eyes; the second feature data represents data associated with the wearer's face and body; based on the first feature data and the second feature data, the wearer is reconstructed to obtain a reconstructed image; the reconstructed image is used to represent the virtual image of the wearer; send the reconstructed image to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image; the first feature data includes eye feature data; the second feature data includes face feature data and body feature data; the first feature data and the second feature data are based on the first feature data and the second feature data, the wearer is reconstructed to obtain a reconstructed image, comprising: the eye feature data and the face feature data are fused to obtain face fusion data; based on the face fusion data and the body feature data, model driven data is obtained; based on the model driven data and the voice data collected by the wearable device, the reconstructed image is obtained; the eye feature data and the face feature data are fused to obtain face fusion data, comprising: align the face feature data with the standard face data to obtain face alignment data; based on the initial eye data in the face alignment data and the eye feature data, an eye correction scale is obtained; based on the eye correction scale, the eye feature data is corrected to obtain eye correction data; the eye correction data is fused into the face alignment data to obtain the face fusion data; based on the initial eye data in the face alignment data and the eye feature data, the eye correction scale is determined, comprising: obtain the relative external parameters of the eye movement camera and the external camera; based on the relative external parameters, the center coordinates of two eye key points contained in the eye feature data, and the preset depth distance of the eye movement camera relative to the eye key points, the actual distance between the two eye key points is obtained; based on the ratio between the initial distance between the two eye key points in the eye initial data and the actual distance, the eye correction scale is obtained.

2. The method of claim 1, wherein, the face feature data is aligned with the standard face data to obtain face alignment data, comprising: based on the plurality of face key points of the face feature data and the plurality of standard key points of the standard face data, the overall optimization error of face alignment is obtained; nonlinear optimization is performed on the overall optimization error to obtain the face alignment data.

3. The method of claim 1, wherein, based on the relative external parameters, the center coordinates of two eye key points contained in the eye feature data, and the preset depth distance of the eye movement camera relative to the eye key points, the actual distance between the two eye key points is obtained, comprising: Based on the center coordinates of the two eye key points and the depth distance, three-dimensional coordinates of the two eye key points are obtained. Based on the three-dimensional coordinates of the two eye key points and the relative extrinsic parameters, an actual distance between the two eye key points is obtained.

4. The method of claim 1, wherein, The model driving data includes expression weights and coordinates of a plurality of body key points; and the model driving data is obtained based on the face fusion data and the body feature data, including: Based on a target expression represented by the face fusion data, a plurality of preset reference expressions, and weights of the plurality of reference expressions, an expression weight corresponding to the target expression is obtained. The plurality of body key points of the body feature data are mapped to a three-dimensional space to obtain coordinates of the plurality of body key points.

5. The method of claim 1, wherein, The reconstructed image is obtained based on the model driving data and voice data collected by the wearable device. A virtual driving model is obtained. The virtual driving model is driven based on the model driving data and the voice data collected by the wearable device to obtain the reconstructed image.

6. The method according to any one of claims 1 to 5, characterized in that, The first feature data collected by the eye movement camera of the wearable device and the second feature data collected by an external camera other than the eye movement camera are obtained, including: A clock synchronization signal is sent to a camera device where the external camera is located or the clock synchronization signal sent by the camera device is received; the clock synchronization signal is used to synchronize the clocks of the eye movement camera and the external camera. The first feature data and the second feature data are obtained based on the eye movement camera and the external camera after clock synchronization.

7. A communication device, characterized by Applied to a wearable device, including: A feature acquisition module configured to obtain first feature data collected by an eye movement camera of the wearable device and second feature data collected by an external camera other than the eye movement camera; wherein the first feature data represents data associated with the eyes of a wearer; and the second feature data represents data associated with the face and body of the wearer. A reconstructed image module configured to perform image reconstruction on the wearer based on the first feature data and the second feature data to obtain a reconstructed image. A communication module configured to send the reconstructed image to a communication device in communication with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image. The first feature data includes eye feature data; the second feature data includes face feature data and body feature data; and the image reconstruction on the wearer based on the first feature data and the second feature data to obtain a reconstructed image includes: Feature fusion is performed on the eye feature data and the face feature data to obtain face fusion data. Model driving data is obtained based on the face fusion data and the body feature data. The reconstructed image is obtained based on the model driving data and voice data collected by the wearable device. The feature fusion on the eye feature data and the face feature data to obtain face fusion data includes: aligning the face feature data with standard face data to obtain face alignment data; determining an eye correction scale based on the eye initial data in the face alignment data and the eye feature data; correcting the eye feature data based on the eye correction scale to obtain eye correction data; fusing the eye correction data into the face alignment data to obtain the face fusion data; the determining an eye correction scale based on the eye initial data in the face alignment data and the eye feature data, comprises: obtaining a relative external parameter of the eye movement camera and the external camera; obtaining an actual distance between two eye key points based on the relative external parameter, center coordinates of the two eye key points contained in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points; obtaining the eye correction scale based on a ratio between an initial distance between the two eye key points in the eye initial data and the actual distance.

8. A wearable device, comprising: comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the communication method as claimed in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a wearable device, enabling the wearable device to perform the communication method as claimed in any one of claims 1 to 6.

Images