Communication method and device, wearable equipment and storage medium

Abstract

The invention relates to a communication method and device, wearable equipment and a storage medium. The communication method is applied to the wearable device, and comprises the following steps: acquiring first feature data acquired by an eye movement camera of the wearable device and second feature data acquired by an external camera except the eye movement camera; wherein the first feature data is represented as data associated with the eyes of the wearer; the second feature data is represented as data associated with the face and the body of the wearer; based on the first feature data and the second feature data, performing image reconstruction on the wearer to obtain a reconstructed image; the reconstructed image is used for representing a virtual image of the wearer; and sending the reconstructed image to a communication device communicating with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image. According to the embodiment of the invention, the wearing comfort of the wearable equipment can be improved, and the electric anxiety can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a communication method and apparatus, a wearable device, and a storage medium. Background Art

[0002] At present, remote communication technology based on wearable devices such as AR glasses or VR glasses is an emerging application. During the remote communication process of wearable devices, if the built-in eye movement camera and facial camera in the wearable device are used to achieve remote communication, the wearable device needs to introduce an additional facial camera, which will increase the power consumption and weight of the wearable device, making the wearing comfort less and the power consumption anxiety more; if an external camera is used alone to achieve remote communication, the wearer's eyes will be blocked by the glasses, which will cause the eye recognition accuracy to decrease or even fail to recognize. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a communication method and apparatus, a wearable device and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a communication method is provided, comprising at least:

[0005] Acquire first feature data collected by an eye-tracking camera of the wearable device and second feature data collected by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the wearer's eyes; and the second feature data is characterized as data associated with the wearer's face and body;

[0006] Based on the first feature data and the second feature data, reconstructing the image of the wearer to obtain a reconstructed image; the reconstructed image is used to represent the virtual image of the wearer;

[0007] The reconstructed image is sent to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0008] In some embodiments, the first feature data includes: eye feature data; the second feature data includes facial feature data and body feature data; and reconstructing the image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image includes:

[0009] Performing feature fusion on the eye feature data and the facial feature data to obtain facial fusion data;

[0010] Obtaining model driving data based on the facial fusion data and the body feature data;

[0011] The reconstructed image is obtained based on the model driving data and the voice data collected by the wearable device.

[0012] In some embodiments, the step of fusing the eye feature data and the facial feature data to obtain facial fusion data includes:

[0013] Aligning the facial feature data with standard facial data to obtain facial alignment data;

[0014] Obtaining an eye correction scale based on the initial eye data and the eye feature data in the face alignment data;

[0015] Correcting the eye feature data based on the eye correction scale to obtain eye correction data;

[0016] The eye correction data is fused into the facial alignment data to obtain the facial fusion data.

[0017] In some embodiments, aligning the facial feature data with standard facial data to obtain facial alignment data includes:

[0018] Based on the multiple facial key points of the facial feature data and the multiple standard key points of the standard facial data, an overall optimization error of facial alignment is obtained;

[0019] The overall optimization error is subjected to nonlinear optimization to obtain the facial alignment data.

[0020] In some embodiments, determining the eye correction scale based on the initial eye data and the eye feature data in the facial alignment data includes:

[0021] Obtaining relative external parameters of the eye-tracking camera and the external camera;

[0022] Obtaining an actual distance between the two eye key points based on the relative external parameter, the center coordinates of the two eye key points included in the eye feature data, and a preset depth distance of the eye-movement camera relative to the eye key points;

[0023] The eye correction scale is obtained based on the ratio of the initial distance between the two eye key points in the initial eye data and the actual distance.

[0024] In some embodiments, obtaining the actual distance between the two eye key points based on the relative external reference, the center coordinates of the two eye key points included in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points includes:

[0025] Based on the center coordinates of the two eye key points and the depth distance, obtaining the three-dimensional coordinates of the two eye key points;

[0026] Based on the three-dimensional coordinates of the two eye key points and the relative external parameter, the actual distance between the two eye key points is obtained.

[0027] In some embodiments, the model driving data includes expression weights and coordinates of a plurality of body key points; and obtaining the model driving data based on the facial fusion data and the body feature data includes:

[0028] Based on a target expression represented by the facial fusion data, a plurality of preset reference expressions, and weights of the plurality of reference expressions, obtaining an expression weight corresponding to the target expression;

[0029] The multiple body key points of the body feature data are mapped into a three-dimensional space to obtain the coordinates of the multiple body key points.

[0030] In some embodiments, obtaining the reconstructed image based on the model driving data and the voice data collected by the wearable device includes:

[0031] Get the virtual drive model;

[0032] Based on the model driving data and the voice data collected by the wearable device, the virtual driving model is driven to obtain the reconstructed image.

[0033] In some embodiments, the acquiring of first feature data acquired by an eye-tracking camera of the wearable device and second feature data acquired by an external camera other than the eye-tracking camera includes:

[0034] Sending a clock synchronization signal to a camera device where the external camera is located or receiving the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize the clocks of the eye-tracking camera and the external camera;

[0035] The first feature data and the second feature data are acquired based on the clock-synchronized eye-tracking camera and the external camera.

[0036] According to a second aspect of an embodiment of the present disclosure, a communication device is provided, which is applied to a wearable device and at least includes:

[0037] A feature acquisition module, configured to acquire first feature data acquired by an eye-tracking camera of the wearable device and second feature data acquired by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the wearer's eyes; and the second feature data is characterized as data associated with the wearer's face and body;

[0038] an image reconstruction module, configured to reconstruct the image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image;

[0039] The communication module is configured to send the reconstructed image to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0040] According to a third aspect of an embodiment of the present disclosure, there is provided a wearable device, comprising at least:

[0041] processor;

[0042] a memory for storing processor-executable instructions;

[0043] The processor is configured to execute the communication method as described in the first aspect above.

[0044] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0045] When the instructions in the storage medium are executed by the processor of the wearable device, the wearable device is enabled to execute the communication method in the first aspect as described above.

[0046] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0047] The disclosed embodiment obtains the first feature data and the second feature data obtained by the eye-tracking camera of the wearable device and the external camera to obtain a reconstructed image, so that the wearable device and the communication device can communicate based on the reconstructed image. In this way, the disclosed embodiment does not need to set up an additional camera for collecting face and body images in the wearable device, which can reduce the power consumption and weight increase caused by the additional camera of the wearable device, improve the wearing comfort of the wearable device and reduce the anxiety of power consumption.

[0048] Furthermore, the disclosed embodiment combines an eye-tracking camera and an external camera to reconstruct the image of the wearer, rather than using an external camera alone to reconstruct the image of the wearer. This can reduce the inaccuracy of externally collected eye data caused by the wearer's eyes being blocked by the wearable device, improve the eye recognition accuracy during the image reconstruction process, and thus reconstruct a better virtual image of the wearer, thereby improving the immersive experience of wearable device communication.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] Figure 1 The figure is a flowchart of a communication method according to an exemplary embodiment.

[0052] Figure 2 The figure is a block diagram of an image reconstruction process according to an exemplary embodiment.

[0053] Figure 3 It is a block diagram of a communication device according to an exemplary embodiment.

[0054] Figure 4 The present invention is a block diagram of a wearable device according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0056] An embodiment of the present disclosure provides a communication method. Figure 1 This is a flow chart of a communication method according to an exemplary embodiment. The communication method is applied to a scenario of remote communication based on a wearable device, which can make remote communication more immersive and achieve more natural and intuitive remote communication. For example, when a wearer wears the wearable device and communicates with a communication device through the wearable device, communication based on reconstructed images can display the wearer's virtual image in the field of view through virtual reality technology, thereby achieving a face-to-face remote communication effect. Figure 1As shown, the wearable device executes the communication method including the following steps:

[0057] S101, obtaining first feature data collected by an eye-tracking camera of the wearable device and second feature data collected by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the wearer's eyes; and the second feature data is characterized as data associated with the wearer's face and body;

[0058] S102, reconstructing the image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image;

[0059] S103: Send the reconstructed image to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0060] In the embodiments of the present disclosure, the wearable device can communicate remotely with the communication device, and the wearable device may include an augmented reality (AR) glasses device or a virtual reality (VR) glasses device; the communication device may include a smart phone, a notebook, a tablet computer, an AR glasses device or a VR glasses device, which is not limited in the embodiments of the present disclosure.

[0061] Taking the wearable device as an AR glasses device and the communication device as a smart phone as an example, the AR glasses device can obtain a virtual image representing the wearer of the AR glasses device by executing the communication method. During the communication process between the AR glasses device and the smart phone, the virtual image of the wearer can be sent to the communication device, thereby realizing more natural and intuitive remote communication between the AR glasses device and the smart phone.

[0062] In step S101, the eye-tracking camera is a built-in camera of the wearable device, and the first feature data collected by the eye-tracking camera is characterized as data associated with the wearer's eyes. Exemplarily, the first feature data may include feature data of key points of the eye; wherein the feature data of the key points of the eye include: size data and position data of the key points of the eye; the key points of the eye include key points such as eyeballs, upper and lower eyelids, or eyebrows, and the embodiments of the present disclosure are not limited to this.

[0063] The external camera is not a built-in camera of the wearable device, and the second feature data collected by the external camera is characterized as data related to the face and body of the wearer. Exemplarily, the second feature data may include feature data of key points of the face and feature data of key points of the body; wherein the key points of the face include key points such as the mouth, nose, and cheeks; the key points of the body include key points such as the arms, legs, fingers, and head, and the embodiments of the present disclosure do not limit this.

[0064] It should be noted that the external camera may be a built-in camera of a camera device, and the camera device and the wearable device are two independent devices. The camera device may include a smart phone, a notebook, a tablet computer, a smart watch, or a surveillance camera device, etc., which is not limited in the embodiments of the present disclosure.

[0065] In the disclosed embodiment, the first feature data and the second feature data may be obtained through key point detection. For the first feature data, the key point detection method may include detection methods such as convolutional neural networks (CNN) or recurrent neural networks (RNN); for the second feature data, the key point detection method may include deep learning methods, such as OpenPose.

[0066] Here, since the eye-tracking camera and the external camera are cameras on different devices, the eye-tracking camera and the external camera need to be clock synchronized before reconstructing the image of the wearer. In some embodiments, the step of obtaining the first feature data collected by the eye-tracking camera of the wearable device and the second feature data collected by the external camera other than the eye-tracking camera includes:

[0067] Sending a clock synchronization signal to a camera device where the external camera is located or receiving the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize the clocks of the eye-tracking camera and the external camera;

[0068] The first feature data and the second feature data are acquired based on the clock-synchronized eye-tracking camera and the external camera.

[0069] That is to say, the embodiment of the present disclosure performs clock synchronization calibration on the eye-tracking camera and the external camera, so that the first feature data and the second feature data collected by the two cameras can ensure data synchronization, and then in the process of driving the preset virtual driving model based on the first feature data and the second feature data, the driving consistency can be guaranteed, thereby obtaining a better reconstructed image.

[0070] In the embodiment of the present disclosure, the clock synchronization signal can be sent at a preset time interval. The preset time can be set in the range of 150 milliseconds to 250 milliseconds, and the embodiment of the present disclosure does not limit this.

[0071] It should be noted that when the wearable device serves as the data synchronization center, the wearable device can send a clock synchronization signal to the camera device, so that the external camera in the camera device can be synchronized with the eye-tracking camera, thereby achieving data synchronization between the first feature data collected by the eye-tracking camera and the second feature data collected by the external camera.

[0072] When the camera device serves as the data synchronization center, the camera device can send a clock synchronization signal to the wearable device, so that the eye-tracking camera of the wearable device can synchronize the clock with the external camera, thereby achieving data synchronization between the first feature data collected by the eye-tracking camera and the second feature data collected by the external camera.

[0073] In the disclosed embodiment, the wearable device and the camera device can be connected to the same communication network, so that clock synchronization based on the same communication network can make the clock synchronized data transmission faster and the data transmission delay lower.

[0074] In step S102, an image of the wearer is reconstructed based on the first feature data and the second feature data. The reconstructed image obtained by the image reconstruction is characterized as a virtual image of the wearer. In remote communication, the virtual image is displayed in the field of view, so that an immersive communication experience can be achieved between the user of the communication device and the wearer of the wearable device. The virtual image of the wearer can be a two-dimensional image or a three-dimensional image, which is not limited in the embodiments of the present disclosure.

[0075] In the disclosed embodiment, the first feature data is characterized as data associated with the wearer's eyes, and the wearer's eyes can be reconstructed based on the first feature data; the second feature data is characterized as data associated with the wearer's face and body, and the wearer's face and body can be reconstructed based on the second feature data. In this way, based on the reconstruction of the wearer's eyes and the reconstruction of the wearer's face and body, the wearer's virtual image can be obtained.

[0076] It should be noted that the reconstructed image obtained by reconstructing the image of the wearer can be an image sequence, and the virtual image of the wearer constructed based on the image sequence can be linked, which can present the animation effect of the virtual image of the wearer. Among them, the virtual image can correspond to the animal model digital human, cartoon digital human or hyper-realistic digital human in the digital human, etc., and the embodiments of the present disclosure are not limited to this.

[0077] In step S103, sending the reconstructed image to a communication device that communicates with the wearable device may include: the wearable device sends the reconstructed image to an intermediate server device, the server device transfers the reconstructed image to the communication device, and the communication device can display the virtual image of the wearer to achieve immersive remote communication between the wearable device and the communication device.

[0078] For example, in a video conference or remote call scenario, the wearable device can send the reconstructed image to the communication device, and the communication device can display the wearer's virtual image on the display screen, so that the user of the communication device can see the wearer's virtual image in the field of view. Of course, the characteristic data of the user of the communication device can also be collected and sent to the wearable device after the virtual image of the communication device user is obtained based on the characteristic data of the communication device user, so that the wearer of the wearable device can also see the virtual image of the communication device user. In this way, an immersive conference experience or remote call experience between the wearer of the wearable device and the user of the communication device can be achieved through the above method.

[0079] It should be noted that the reconstructed image can also be used for self-display and sent to other display devices (such as television devices) for display, and the embodiments of the present disclosure are not limited to this.

[0080] In the disclosed embodiment, the reconstructed image is obtained by obtaining the first feature data and the second feature data through the eye-tracking camera of the wearable device and the external camera, so that the wearable device and the communication device can communicate based on the reconstructed image. In this way, the disclosed embodiment does not need to set up an additional camera for collecting face and body images in the wearable device, which can reduce the increase in power consumption and weight of the wearable device due to the additional camera, improve the wearing comfort of the wearable device and reduce the anxiety of electricity consumption. In addition, the disclosed embodiment combines the eye-tracking camera and the external camera to reconstruct the image of the wearer, rather than using the external camera alone to reconstruct the image of the wearer, which can reduce the inaccuracy of the eye data collected externally due to the wearer's eyes being blocked by the wearable device, and can improve the eye recognition accuracy during the image reconstruction process, thereby being able to reconstruct a better virtual image of the wearer and improve the immersive experience of wearable device communication.

[0081] In some embodiments, reconstructing the image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image includes:

[0082] Performing feature fusion on the eye feature data and the facial feature data to obtain facial fusion data;

[0083] Obtaining model driving data based on the facial fusion data and the body feature data;

[0084] The reconstructed image is obtained based on the model driving data and the voice data collected by the wearable device.

[0085] In the disclosed embodiment, the first feature data includes: eye feature data; the second feature data includes facial feature data and body feature data. In the process of acquiring the second feature data by the external camera, the facial feature data acquired by the external camera is not very accurate because the wearer's eyes are blocked by the wearable device. Therefore, the eye feature data and facial feature data acquired by the eye-tracking camera are feature fused.

[0086] It should be noted that, in the process of feature fusion of eye feature data and facial feature data, the eye feature data can be input into the first layer of model-level fusion to obtain first fusion data, and then the first fusion data and the facial feature data are input into the second layer of model-level fusion to obtain second fusion data, and finally the second fusion data is input into the fully connected layer to obtain the facial fusion data; or the facial feature data can be first aligned with the standard facial data, and then the facial alignment data obtained by the alignment is corrected to obtain the facial fusion data.

[0087] In the disclosed embodiment, after facial fusion data is obtained through feature fusion, model driving data can be obtained based on the facial fusion data and the body feature data. The model driving data is used to drive the virtual driving model so that the reconstructed virtual image of the wearer can present animation effects during the communication process.

[0088] It should be noted that the communication process is usually a voice call, which can be a social voice call or a conference voice call, and the embodiments of the present disclosure do not limit this.

[0089] Here, after obtaining the model driving parameters, the embodiment of the present disclosure also needs to collect voice data, and obtain a reconstructed image based on the model driving data and the voice data. That is to say, during a voice call, the communication device of the embodiment of the present disclosure can not only see the wearer's virtual image but also synchronously hear the virtual image outputting voice data, thus improving the immersive experience of wearable device communication.

[0090] In some embodiments, the step of fusing the eye feature data and the facial feature data to obtain facial fusion data includes:

[0091] Aligning the facial feature data with standard facial data to obtain facial alignment data;

[0092] Obtaining an eye correction scale based on the initial eye data and the eye feature data in the face alignment data;

[0093] Correcting the eye feature data based on the eye correction scale to obtain eye correction data;

[0094] The eye correction data is fused into the facial alignment data to obtain the facial fusion data.

[0095] In the disclosed embodiment, the wearer's face presents different shapes under different expressions, so it is necessary to align the facial feature data. The facial alignment method may include alignment based on a face model or alignment based on deep learning of a convolutional neural network. The face model may include an active shape model (ActiveShapeModel, ASM) or an active appearance model (active appearance model, AAM); the deep learning based on a convolutional neural network may include a DeepID convolutional neural network or a DeepFace convolutional neural network.

[0096] In some embodiments, aligning the facial feature data with standard facial data to obtain facial alignment data includes:

[0097] Based on the multiple facial key points of the facial feature data and the multiple standard key points of the standard facial data, an overall optimization error of facial alignment is obtained;

[0098] The overall optimization error is subjected to nonlinear optimization to obtain the facial alignment data.

[0099] In the disclosed embodiment, obtaining an overall optimization error of facial alignment based on multiple facial key points of facial feature data and multiple standard key points of standard facial data may include: obtaining data error, smoothing error and key point error of facial alignment based on multiple facial key points and multiple standard key points; and then obtaining an overall optimization error based on the data error, smoothing error and key point error.

[0100] Here, the embodiment of the present disclosure is based on multiple facial key points and multiple standard key points, and the data error E can be obtained by formula (1): d (X), the smoothing error E is obtained by formula (2) s (X), and the key point error E is obtained by formula (3) l (X).

[0101]

[0102] Among them, X represents the facial key points; v i is represented by the vertex of the i-th standard key point; V is represented by the set of vertices; w i is represented as the i-th weight; τ is represented as the point closest to the vertex of the standard key point among multiple facial key points; dist() is represented as the distance function used to calculate the distance; X i Expressed as v i The corresponding transformation matrix.

[0103]

[0104] Among them, i and j are two labels for the converted facial key points; X i Expressed as v i The corresponding transformation matrix; X j Expressed as v j The corresponding transformation matrix; G is represented as a 4*4 diagonal matrix.

[0105]

[0106] Among them, l represents the facial key points, and L represents the set of facial key points and transformation matrices.

[0107] It should be noted that, during the facial alignment process, multiple facial key points may be selected for matching with multiple standard key points, and the number of selected key points may be set within the range of 15 to 25, which is not limited in the embodiments of the present disclosure.

[0108] In the embodiment of the present disclosure, obtaining the overall optimization error based on the data error, smoothing error and key point error may include: obtaining a first product of the smoothing error and a first coefficient; obtaining a second product of the key point error and a second coefficient; and obtaining the overall optimization error based on the sum of the data error, the first product and the second product.

[0109] Here, the overall optimization error E can be obtained by formula (4).

[0110] E=E d (X)+α*E s (X)+β*E l (X)(4)

[0111] Here, α is represented as the first coefficient, and β is represented as the second coefficient.

[0112] In the embodiment of the present disclosure, after the overall optimization error is obtained, the overall optimization error may be subjected to nonlinear optimization to obtain the facial alignment data. The nonlinear optimization method may include a nonlinear least squares method or a Gauss-Newton method, which is not limited in the embodiment of the present disclosure.

[0113] It should be noted that after obtaining the facial alignment data, since the initial eye data and the eye feature data in the facial alignment data belong to data of different dimensions, it is necessary to calculate a correction scale based on the initial eye data and the eye feature data.

[0114] In some embodiments, determining the eye correction scale based on the initial eye data and the eye feature data in the facial alignment data includes:

[0115] Obtaining relative external parameters of the eye-tracking camera and the external camera;

[0116] Obtaining an actual distance between the two eye key points based on the relative external parameter, the center coordinates of the two eye key points included in the eye feature data, and a preset depth distance of the eye-movement camera relative to the eye key points;

[0117] The eye correction scale is obtained based on the ratio of the initial distance between the two eye key points in the initial eye data and the actual distance.

[0118] In the disclosed embodiment, since the eye-tracking camera and the external camera have different positions in the real space, it is necessary to determine the relative external parameters between the two cameras, and then determine the mapping relationship between the feature data collected by different cameras, so as to realize the image reconstruction by combining the eye-tracking camera and the external camera. The relative external parameters of the eye-tracking camera and the external camera refer to the conversion relationship between the camera coordinate systems of the eye-tracking camera and the external camera.

[0119] In some embodiments, obtaining the relative external parameters of the eye-movement camera and the external camera may include: determining the internal and external parameters of the eye-movement camera, as well as the internal and external parameters of the external camera, through reference objects such as calibration plates and calibration rods; determining the mapping relationship between the eye-movement camera and the external camera by comparing the image of the reference object acquired by the eye-movement camera with the image of the reference object acquired by the external camera; and obtaining the relative external parameters of the eye-movement camera and the external camera through the mapping relationship, the internal and external parameters of the eye-movement camera, and the internal and external parameters of the external camera.

[0120] In other embodiments, obtaining the relative extrinsic parameters of the eye-tracking camera and the external camera may include: determining a co-viewed image in the images acquired by the eye-tracking camera and the external camera; determining the coordinates of the co-viewed image in the world coordinate system; and then determining the initial extrinsic parameters of the eye-tracking camera and the external camera; establishing a mapping relationship between the co-viewed image in the pixel coordinate system or the image coordinate system of different cameras; and determining the relative extrinsic parameters of the eye-tracking camera and the external camera based on the mapping relationship and the initial extrinsic parameters.

[0121] In the embodiment of the present disclosure, after obtaining the relative external reference, the actual distance between the two eye key points can be obtained based on the relative external reference and the center coordinates of the two eye key points. The two eye key points may include two eyeball key points, two eyelid key points or two eyebrow key points, which is not limited in the embodiment of the present disclosure.

[0122] In some embodiments, obtaining the actual distance between the two eye key points based on the relative external reference, the center coordinates of the two eye key points included in the eye feature data, and a preset depth distance of the eye movement camera relative to the eye key points includes:

[0123] Based on the center coordinates of the two eye key points and the depth distance, the three-dimensional coordinates of the two eye key points are obtained; based on the three-dimensional coordinates of the two eye key points and the relative external parameters, the actual distance between the two eye key points is obtained.

[0124] In the disclosed embodiment, the two eye key points include a left eye key point and a right eye key point; the three-dimensional coordinates of the left eye key point can be obtained based on the product of the center coordinates of the left eye key point, the depth distance and the third coefficient; the three-dimensional coordinates of the right eye key point can be obtained based on the product of the center coordinates of the right eye key point, the depth distance and the third coefficient.

[0125] Here, the three-dimensional coordinates P of the left eye key point can be obtained by formula (5): 3Dl The three-dimensional coordinates P of the right eye key point can be obtained by formula (6): 3Dr .

[0126] P 3Dl =K -1 *d*P uvl (5)

[0127] P 3Dr =K -1 *d*P uvr (6)

[0128] Where K represents the third coefficient; d represents the depth distance; P uvl Expressed as the center coordinates of the left eye key point; P uvr Expressed as the center coordinates of the right eye keypoint.

[0129] In the embodiment of the present disclosure, obtaining the actual distance between the two eye key points based on the three-dimensional coordinates of the two eye key points and the relative external parameter may include: obtaining the third product between the relative external parameter and the three-dimensional coordinate value of the right eye key point; and obtaining the actual distance based on the difference between the three-dimensional coordinate of the left eye key point and the third product.

[0130] Here, the actual distance dist can be obtained by formula (7). lr Expressed as a relative external parameter.

[0131] dist=P 3Dl -T lr *P 3Dr (7)

[0132] In the disclosed embodiment, after the actual distance is obtained, the eye correction scale can be obtained by formula (7).

[0133] S=Fddist(8)

[0134] Among them, S represents the eye correction scale; Fd represents the initial distance.

[0135] In the disclosed embodiment, the eye correction scale is used to correct the eye feature data. After obtaining the eye correction data, the eye correction data is fused into the facial alignment data to obtain the facial fusion data, which may include: replacing the eye correction data into the facial alignment data to obtain the facial fusion data.

[0136] In this way, the embodiment of the present disclosure first obtains facial alignment data through alignment, then obtains eye correction data based on the facial alignment data obtained through alignment, and finally obtains facial fusion data based on the fusion of the eye correction data. In this way, the accuracy of fusion can be improved through alignment and correction.

[0137] In some embodiments, the model driving data includes expression weights and coordinates of a plurality of body key points; and obtaining the model driving data based on the facial fusion data and the body feature data includes:

[0138] Based on a target expression represented by the facial fusion data, a plurality of preset reference expressions, and weights of the plurality of reference expressions, obtaining an expression weight corresponding to the target expression;

[0139] The multiple body key points of the body feature data are mapped into a three-dimensional space to obtain the coordinates of the multiple body key points.

[0140] In the disclosed embodiment, based on the target expression represented by the facial fusion data, a plurality of preset baseline expressions and the weights of the plurality of baseline expressions, an expression weight corresponding to the target expression is obtained, which may include: obtaining the product of the plurality of baseline expressions and the weights corresponding to the plurality of baseline expressions to obtain an expression vector; obtaining a difference based on the difference between the expression vector and the target expression, and obtaining the expression weight based on the norm processing of the difference.

[0141] Here, the expression weight f(w) can be obtained by formula (9).

[0142]

[0143] Where A represents multiple base expressions; [w0...w n-1 ] T It is represented by the weights of multiple benchmark expressions; B is represented by the target expression.

[0144] In the disclosed embodiment, the model driving data for the face is the expression weight, and the model driving data for the body is the coordinates of multiple body key points obtained by mapping multiple body key points to three-dimensional space. Here, the disclosed embodiment can first establish a spatial coordinate system, and then obtain the coordinates of multiple body key points through the positions of multiple body key points and the spatial transformation matrix. In this way, the disclosed embodiment can obtain the model driving data, and then can realize the driving of the virtual driving model, so that the reconstructed virtual image of the wearer can present an animation effect during the communication process.

[0145] In some embodiments, obtaining the reconstructed image based on the model driving data and the voice data collected by the wearable device includes:

[0146] Get the virtual drive model;

[0147] Based on the model driving data and the voice data collected by the wearable device, the virtual driving model is driven to obtain the reconstructed image.

[0148] In the disclosed embodiment, the virtual driving model is a model that can be driven by simulated driving parameters to present animation effects. The acquisition of the virtual driving model may include: acquiring a three-dimensional digital human model; calculating the face of the three-dimensional digital human model using task expressions or artificial modeling; binding the body of the three-dimensional digital human model using standard bone points, and performing skinning and weight calculation on each bone point of the bound model, thereby obtaining the virtual driving model.

[0149] The above-mentioned driving the virtual driving model based on the model driving data and the voice data collected by the wearable device to obtain the reconstructed image may include first driving the virtual driving model based on the model driving data and the voice data collected by the wearable device, and then rendering the driven model through a rendering engine to obtain the reconstructed image. The rendering engine may include Unreal Engine (UE) and Unity engine, which is not limited in the embodiments of the present disclosure.

[0150] It should be noted that by driving the virtual driving model together with model driving data and language data, it is possible to see the wearer's virtual image and simultaneously hear the wearer's virtual image outputting voice data, thereby improving the immersive experience of wearable device communication.

[0151] In order to better understand the communication method in one or more of the above embodiments of the present application, the embodiments of the present disclosure are further illustrated as follows:

[0152] like Figure 2As shown, the disclosed embodiment can first obtain a three-dimensional digital human model, and perform model binding on the three-dimensional digital human to obtain a virtual driving model; then perform clock synchronization on the eye-tracking camera and the external camera and collect the first feature data and the second feature data; then perform feature fusion on the eye feature data included in the first feature data and the facial feature data included in the second feature data to obtain facial fusion data; then obtain model driving data based on the facial fusion data and the body feature data; then drive the virtual driving model based on the model driving data and the voice data collected by the wearable device; finally, render the driven model through a rendering engine to obtain a reconstructed image.

[0153] In this way, through the clock synchronization of the eye-tracking camera and the external camera, the feature fusion of the eye feature data and the facial feature data, the model-driven data calculation, the driving of the virtual driving model and the final rendering output, it is possible to reconstruct the image of the wearer through the eye-tracking camera and the external camera to obtain the wearer's virtual image. It can be seen that the embodiment of the present disclosure does not require an additional camera for collecting face and body images in the wearable device, which can reduce the power consumption and weight increase of the wearable device caused by the additional camera, improve the wearing comfort of the wearable device and reduce the anxiety of power consumption.

[0154] Furthermore, the disclosed embodiments can also combine an eye-tracking camera and an external camera to reconstruct the image of the wearer, rather than using an external camera alone to reconstruct the image of the wearer. This can reduce the inaccuracy of externally collected eye data caused by the wearer's eyes being blocked by the wearable device, improve the eye recognition accuracy during the image reconstruction process, and thus reconstruct a better virtual image of the wearer, thereby improving the immersive experience of wearable device communication.

[0155] The disclosed embodiment also provides a communication device for use in a wearable device. Figure 3 FIG. 1 is a block diagram of a communication device according to an exemplary embodiment. Figure 3 As shown, the communication device 1000 includes:

[0156] The feature acquisition module 1001 is configured to acquire first feature data acquired by the eye-tracking camera of the wearable device and second feature data acquired by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the wearer's eyes; and the second feature data is characterized as data associated with the wearer's face and body;

[0157] An image reconstruction module 1002 is configured to reconstruct the image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image;

[0158] The communication module 1003 is configured to send the reconstructed image to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0159] In some embodiments, the first feature data includes: eye feature data; the second feature data includes facial feature data and body feature data; the image reconstruction module 1002 includes:

[0160] A fusion module, configured to perform feature fusion on the eye feature data and the facial feature data to obtain facial fusion data;

[0161] A driving calculation module, configured to obtain model driving data based on the facial fusion data and the body feature data;

[0162] The data reconstruction module is configured to obtain the reconstructed image based on the model driving data and the voice data collected by the wearable device.

[0163] In some embodiments, the fusion module is further configured to align the facial feature data with the standard facial data to obtain facial alignment data; obtain an eye correction scale based on the initial eye data and the eye feature data in the facial alignment data; correct the eye feature data based on the eye correction scale to obtain eye correction data; and fuse the eye correction data into the facial alignment data to obtain the facial fusion data.

[0164] In some embodiments, the fusion module is further configured to obtain an overall optimization error of facial alignment based on multiple facial key points of the facial feature data and multiple standard key points of the standard facial data; and perform nonlinear optimization on the overall optimization error to obtain the facial alignment data.

[0165] In some embodiments, the fusion module is further configured to obtain relative external parameters of the eye-movement camera and the external camera; obtain the actual distance between the two eye key points based on the relative external parameters, the center coordinates of the two eye key points contained in the eye feature data and the preset depth distance of the eye-movement camera relative to the eye key points; obtain the eye correction scale based on the ratio between the initial distance between the two eye key points in the eye initial data and the actual distance.

[0166] In some embodiments, the fusion module is further configured to obtain the three-dimensional coordinates of the two eye key points based on the center coordinates of the two eye key points and the depth distance; and obtain the actual distance between the two eye key points based on the three-dimensional coordinates of the two eye key points and the relative external parameters.

[0167] In some embodiments, the model driving data includes expression weights and coordinates of multiple body key points; the driving calculation module is configured to obtain expression weights corresponding to the target expression based on the target expression represented by the facial fusion data, multiple preset baseline expressions and weights of the multiple baseline expressions; and map the multiple body key points of the body feature data into three-dimensional space to obtain coordinates of multiple body key points.

[0168] In some embodiments, the data reconstruction module is further configured to obtain a virtual driving model; drive the virtual driving model based on the model driving data and the voice data collected by the wearable device to obtain the reconstructed image.

[0169] In some embodiments, the feature acquisition module 1001 is also configured to send a clock synchronization signal to the camera device where the external camera is located or receive the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize the clocks of the eye-tracking camera and the external camera; and based on the clock-synchronized eye-tracking camera and the external camera, the first feature data and the second feature data are acquired.

[0170] The specific manner in which each module in the device of the above embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0171] Figure 4 800 is a block diagram of a wearable device according to an exemplary embodiment. For example, the wearable device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0172] Reference Figure 4 The wearable device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0173] The processing component 802 generally controls the overall operation of the wearable device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0174] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the wearable device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0175] The power supply component 806 provides power to various components of the wearable device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the wearable device 800.

[0176] The multimedia component 808 includes a screen that provides an output interface between the wearable device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0177] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the wearable device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0178] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0179] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the wearable device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of the components, such as the display and keypad of the wearable device 800, and the sensor assembly 814 can also detect the position change of the wearable device 800 or a component of the wearable device 800, the presence or absence of contact between the user and the wearable device 800, the orientation or acceleration / deceleration of the wearable device 800, and the temperature change of the wearable device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0180] The communication component 816 is configured to facilitate wired or wireless communication between the wearable device 800 and other devices. The wearable device 800 can access a wireless network based on a communication standard, such as WiFi, 3G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0181] In an exemplary embodiment, the wearable device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0182] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of the wearable device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0183] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a wearable device, enables the wearable device to perform a communication method, the method comprising: obtaining first feature data collected by an eye-tracking camera of the wearable device and second feature data collected by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the eyes of the wearer; and the second feature data is characterized as data associated with the face and body of the wearer;

[0184] Based on the first feature data and the second feature data, reconstruct the image of the wearer to obtain a reconstructed image; the reconstructed image is used to represent the virtual image of the wearer;

[0185] The reconstructed image is sent to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

[0186] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0187] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A communication method, characterized in that: Applied in wearable devices, including: Acquire first feature data collected by an eye-tracking camera of the wearable device and second feature data collected by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the wearer's eyes; and the second feature data is characterized as data associated with the wearer's face and body; Based on the first feature data and the second feature data, reconstruct the image of the wearer to obtain a reconstructed image; the reconstructed image is used to represent the virtual image of the wearer; The reconstructed image is sent to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

2. The method according to claim 1, characterized in that The first feature data includes: eye feature data; the second feature data includes facial feature data and body feature data; the image reconstruction of the wearer based on the first feature data and the second feature data to obtain a reconstructed image includes: Performing feature fusion on the eye feature data and the facial feature data to obtain facial fusion data; Obtaining model driving data based on the facial fusion data and the body feature data; The reconstructed image is obtained based on the model driving data and the voice data collected by the wearable device.

3. The method according to claim 2, characterized in that The step of fusing the eye feature data and the facial feature data to obtain facial fusion data includes: Aligning the facial feature data with standard facial data to obtain facial alignment data; Obtaining an eye correction scale based on the initial eye data and the eye feature data in the face alignment data; Correcting the eye feature data based on the eye correction scale to obtain eye correction data; The eye correction data is fused into the facial alignment data to obtain the facial fusion data.

4. The method according to claim 3, characterized in that The step of aligning the facial feature data with standard facial data to obtain facial alignment data comprises: Based on the multiple facial key points of the facial feature data and the multiple standard key points of the standard facial data, an overall optimization error of facial alignment is obtained; The overall optimization error is subjected to nonlinear optimization to obtain the facial alignment data.

5. The method according to claim 3, characterized in that: The step of determining the eye correction scale based on the initial eye data and the eye feature data in the face alignment data comprises: Obtaining relative external parameters of the eye-tracking camera and the external camera; Obtaining an actual distance between the two eye key points based on the relative external parameter, the center coordinates of the two eye key points included in the eye feature data, and a preset depth distance of the eye-movement camera relative to the eye key points; The eye correction scale is obtained based on the ratio of the initial distance between the two eye key points in the initial eye data and the actual distance.

6. The method according to claim 5, characterized in that The obtaining the actual distance between the two eye key points based on the relative external parameter, the center coordinates of the two eye key points included in the eye feature data, and the depth distance of the preset eye-movement camera relative to the eye key points comprises: Based on the center coordinates of the two eye key points and the depth distance, obtaining the three-dimensional coordinates of the two eye key points; Based on the three-dimensional coordinates of the two eye key points and the relative external parameter, the actual distance between the two eye key points is obtained.

7. The method according to claim 2, characterized in that: The model driving data includes expression weights and coordinates of a plurality of body key points; the model driving data is obtained based on the facial fusion data and the body feature data, including: Based on a target expression represented by the facial fusion data, a plurality of preset reference expressions, and weights of the plurality of reference expressions, obtaining an expression weight corresponding to the target expression; The multiple body key points of the body feature data are mapped into a three-dimensional space to obtain the coordinates of the multiple body key points.

8. The method according to claim 2, characterized in that: The step of obtaining the reconstructed image based on the model driving data and the voice data collected by the wearable device includes: Get the virtual drive model; Based on the model driving data and the voice data collected by the wearable device, the virtual driving model is driven to obtain the reconstructed image.

9. The method according to any one of claims 1 to 8, characterized in that: The acquiring of the first feature data acquired by the eye-tracking camera of the wearable device and the second feature data acquired by an external camera other than the eye-tracking camera includes: Sending a clock synchronization signal to a camera device where the external camera is located or receiving the clock synchronization signal sent by the camera device; the clock synchronization signal is used to synchronize the clocks of the eye-tracking camera and the external camera; The first feature data and the second feature data are acquired based on the clock-synchronized eye-tracking camera and the external camera.

10. A communication device, characterized in that: Applied in wearable devices, including: A feature acquisition module, configured to acquire first feature data acquired by an eye-tracking camera of the wearable device and second feature data acquired by an external camera other than the eye-tracking camera; wherein the first feature data is characterized as data associated with the wearer's eyes; and the second feature data is characterized as data associated with the wearer's face and body; an image reconstruction module, configured to reconstruct the image of the wearer based on the first feature data and the second feature data to obtain a reconstructed image; The communication module is configured to send the reconstructed image to a communication device that communicates with the wearable device, so that the wearable device and the communication device can communicate based on the reconstructed image.

11. A wearable device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the communication method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a wearable device, enables the wearable device to execute the communication method as described in any one of claims 1 to 9.

Images