Interactive display method, wearable display device, storage medium and program product

Abstract

The invention provides an interactive display method, wearing display equipment, a storage medium and a program product, and the method comprises the steps: receiving a first image shot by second equipment after first wearing display equipment is connected with the second equipment, and the first image is an image of a first wearer wearing the first wearing display equipment; the first wearing display device processes the first image to obtain a second image, and the second image is a face area image corresponding to a shielding area of the first wearing display device in the first image; the first wearing display device obtains an eyeball image, shot by an eyeball tracking camera of the first wearing display device in real time, of the first wearer, and eyeballs in the second image are replaced with eyeballs in the eyeball image to obtain a to-be-displayed image; and the first wearing display device displays the to-be-displayed image on an outer screen of the first wearing display device, or sends the to-be-displayed image to the second device so as to display the to-be-displayed image on an inner screen of the second device.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of display technologies, and in particular, to an interactive display method, a wearable display device, a storage medium, and a program product. Background Art

[0002] Currently, the mixed reality (MR) headsets with a perspective function on the market cleverly blend the boundaries between the real and virtual worlds. One of their core features is their unique dual-screen design, where an inner screen and an outer screen work together. The inner screen, as the main visual window for the wearer, not only presents a carefully constructed virtual scene but also seamlessly integrates the real-world scene outside through perspective technology, providing users with an unprecedented immersive experience. This design enables users to simultaneously perceive dual information of virtual and real worlds, greatly enriching the level and depth of interaction. The outer screen, on the other hand, plays an important role as a bridge connecting the wearer with the outside world. It mainly undertakes the task of displaying the wearer's eye images. The original intention of this design is to maintain natural eye contact during the wearer's communication with others, thereby reducing the potential sense of social isolation brought by wearing the MR device. However, the content currently displayed on the outer screen is mostly limited to the eye images captured by the built-in eye-tracking camera of the MR device. Although this realizes the real-time transmission of eye contact, it also restricts the external observation of the overall facial situation of the wearer. When observing, external personnel can often only see the wearer's eyes and cannot glimpse the facial information partially or completely blocked by the MR device, which undoubtedly weakens the authenticity and richness of communication to a certain extent. Summary of the Invention

[0003] Embodiments of the present invention provide an interactive display method, a wearable display device, a storage medium, and a program product, which are used to solve the problem that the wearable display device (such as an MR device) cannot display the complete facial information of the wearer, affecting the naturalness of communication.

[0004] To solve the above technical problems, the present invention is implemented as follows:

[0005] In a first aspect, embodiments of the present invention provide an interactive display method, including:

[0006] After the first wearable display device establishes a connection with the second device, it receives a first image captured by the second device, where the first image is an image of a first wearer wearing the first wearable display device;

[0007] The first wearable display device processes the first image to obtain a second image, where the second image is a facial region image corresponding to the occlusion region of the first wearable display device in the first image;

[0008] The first wearable display device obtains an eye image of the first wearer captured in real time by an eye-tracking camera of the first wearable display device, and replaces the eyes in the second image with the eyes in the eye image to obtain an image to be displayed;

[0009] The first wearable display device displays the image to be displayed on an outer screen of the first wearable display device, or sends the image to be displayed to the second device for display on an inner screen of the second device.

[0010] Optionally, the first wearable display device processes the first image to obtain a second image, including:

[0011] Face feature point detection step: The first wearable display device performs face detection on the first image to obtain face feature points;

[0012] Image generation step: The first wearable display device captures a third image at a preset angle of a three-dimensional face model of the first wearer to generate a third image;

[0013] Judgment step: The first wearable display device determines whether the face feature points of the currently captured third image match those of the first image;

[0014] Adjustment step: If they do not match, adjust the preset angle, capture a new third image of the three-dimensional face model, and return to the judgment step;

[0015] Image acquisition step: If they match, obtain a second image according to the currently captured third image and the first image, where the second image is a face area image in the third image corresponding to the occlusion area of the first wearable display device in the first image.

[0016] Optionally, before the first wearable display device processes the first image to obtain a second image, it further includes:

[0017] The first wearable display device obtains multiple face depth images of the first wearer captured by a front structured light camera of the first wearable display device, calculates three-dimensional point cloud and texture data of the face of the first wearer according to the multiple face depth images, and creates a three-dimensional face model of the first wearer according to the three-dimensional point cloud and texture data of the face of the first wearer;

[0018] Or

[0019] The first wearable display device receives the three-dimensional face model of the first wearer sent by a third device and saves it.

[0020] Optionally, the first wearable display device processes the first image to obtain a second image, including:

[0021] The first wearable display device inputs the first image into a neural network to obtain a fourth image, which is a facial image of the first wearer without wearing the first wearable display device. The neural network is trained by multiple groups of control images, and each group of the control images includes a facial image of the first wearer wearing the first wearable display device and a facial image of the first wearer not wearing the first wearable display device;

[0022] The first wearable display device obtains a second image according to the first image and the fourth image. The second image is a facial area image in the fourth image corresponding to the occlusion area of the first wearable display device in the first image.

[0023] Optionally, it further includes:

[0024] The first wearable display device detects whether there is anyone else approaching. If it detects that there is someone else approaching, it sends a broadcast message for requesting to establish a connection.

[0025] Optionally, the method further includes:

[0026] The first wearable display device captures a fifth image of the second wearer wearing the second device and sends the fifth image of the second wearer to the second device.

[0027] Optionally, after the first wearable display device captures a fifth image of the second wearer wearing the second device and sends the fifth image of the second wearer to the second device, it further includes:

[0028] The first wearable display device detects whether the relative pose between it and the second device has changed. When it does not detect that the relative pose between it and the second device has changed, it stops capturing. When it detects that the relative pose between it and the second device has changed, it continues to capture the fifth image of the second wearer wearing the second device and sends the currently captured fifth image of the second wearer to the second device.

[0029] In a second aspect, an embodiment of the present invention provides a wearable display device, including:

[0030] A receiving module, configured to receive the first image captured by the second device after establishing a connection with the second device. The first image is an image of the first wearer wearing the first wearable display device;

[0031] A first acquisition module, configured to process the first image to obtain a second image, where the second image is a facial area image corresponding to the occlusion area of the first wearable display device in the first image;

[0032] A second acquisition module, configured to intentionally acquire an eye image of the first wearer captured in real time by an eye tracking camera of the first wearable display device, and replace the eyes in the second image with the eyes in the eye image to obtain an image to be displayed;

[0033] A display module, configured to display the image to be displayed on an outer screen of the first wearable display device, or send the image to be displayed to the second device for display on an inner screen of the second device.

[0034] In a third aspect, an embodiment of the present invention provides a wearable display device, including: a processor, a memory, and a program stored on the memory and executable on the processor, where when the program is executed by the processor, the steps of the interactive display method described in the first aspect above are implemented.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the interactive display method described in the first aspect above are implemented.

[0036] In a fifth aspect, an embodiment of the present invention provides a computer program product, including computer instructions, and when the computer instructions are executed by a processor, the steps of the interactive display method described in the first aspect above are implemented.

[0037] In an embodiment of the present invention, the picture displayed on the outer screen of the wearable display device, or the picture displayed on the inner screen of the interactive device (the second device), includes both the picture of the user's real eye dynamics and the picture of the user's face occluded by the wearable display device, which can increase the naturalness of the user's communication with surrounding people and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0039] Figure 1 is a flowchart of the interactive display method according to an embodiment of the present invention;

[0040] Figure 2Schematic diagram of interaction between the first wearable display device and the second device according to an embodiment of the present invention;

[0041] Figure 3 Schematic diagram of the first image and the second image according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of matching between the first image and the third image of the three-dimensional face model according to an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of facial scanning using the first wearable display device according to an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of the structured light camera according to an embodiment of the present invention;

[0045] Figure 7 Schematic flow diagram of the interactive display method of Solution 1 of the present invention;

[0046] Figure 8 Schematic diagram of obtaining a facial image without wearing a wearable display device according to Solution 2 of the present invention;

[0047] Figure 9 One of the schematic structural diagrams of the wearable display device according to an embodiment of the present invention;

[0048] Figure 10 Another schematic structural diagram of the wearable display device according to an embodiment of the present invention. Detailed implementation manners

[0049] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0050] Please refer to Figure 1 , an embodiment of the present invention provides an interactive display method, including:

[0051] Step S1: After the first wearable display device establishes a connection with the second device, receive the first image captured by the second device, where the first image is an image of the first wearer wearing the first wearable display device;

[0052] In an embodiment of the present invention, the first wearable display device may be an MR device, an augmented reality (AR) device, or the like.

[0053] The second device may be a wearable display device, such as an MR device, an AR device, etc., or other devices with camera and transmission functions such as a mobile phone or a handheld camera.

[0054] Please refer to Figure 2 , Figure 2 which is an interaction schematic diagram of the first wearable display device and the second device according to an embodiment of the present invention. Figure 2 In this figure, 10 is the first wearable display device, and 20 is the second device. In this embodiment, the second device is an AR device. The second device can capture an image of the first wearer wearing the first wearable display device and transmit it to the first wearable display device 10.

[0055] In an embodiment of the present invention, the first wearable display device and the second device can establish a connection and transmit data through a wireless connection method such as Wifi or Bluetooth.

[0056] Step S2: The first wearable display device processes the first image to obtain a second image, where the second image is a facial area image corresponding to the occluded area of the first wearable display device in the first image.

[0057] Please refer to Figure 3 , Figure 3 where P1 is the first image, the first image is an image captured by the second device, and a part of the wearer's facial area in the first image is occluded by the first wearable display device. P2 is the second image, and the second image includes the facial area of the wearer occluded by the first wearable display device.

[0058] Step S3: The first wearable display device obtains the eye image of the first wearer captured in real time by the eye tracking camera of the first wearable display device, and replaces the eyes in the second image with the eyes in the eye image to obtain an image to be displayed.

[0059] Step S4: The first wearable display device displays the image to be displayed on the outer screen of the first wearable display device, or sends the image to be displayed to the second device for display on the inner screen of the second device.

[0060] In an embodiment of the present invention, the first wearable display device sends the image to be displayed to the second device, and the second device can synthesize the first image (the image of the first wearer wearing the first wearable display device) captured by itself with the image to be displayed (using the image to be displayed to replace the area occluded by the first wearable display device in the first image) to obtain a synthesized image and display it on the inner screen.

[0061] In an embodiment of the present invention, the picture displayed on the outer screen of the wearable display device, or the picture displayed on the inner screen of the interactive device (the second device), includes both the picture of the user's real eye dynamics and the picture of the user's face blocked by the wearable display device, which can increase the naturalness of the user's communication with surrounding people and improve the user experience.

[0062] In some embodiments, optionally, it further includes: the first wearable display device detects whether there is anyone else approaching. If it detects that there is someone else approaching, it sends a broadcast message, and the broadcast message is used to request to establish a connection.

[0063] In an embodiment of the present invention, the first wearable display device may include a depth camera, which captures a real-time image of the front environment through the depth camera and detects whether there is anyone else approaching through the front environment image. For example, if it detects that there is someone directly in front, with the face facing the first wearer and the distance not exceeding a preset distance, it is considered that someone is approaching. Of course, it is not excluded to use other methods to detect whether there is someone approaching, such as using a human body sensor, etc.

[0064] After the first wearable display device establishes a connection with the second device, the second device starts the camera to capture an image of the first wearer wearing the first wearable display device and transmits it to the first wearable display device. Optionally, the second device may also send the position of the first wearer in the coordinate system of the second device to the first wearable display device for subsequent image processing.

[0065] In some embodiments, optionally, the first wearable display device may also receive a broadcast signal sent by the second device and establish a connection with the second device.

[0066] The following details the method of how the first wearable display device processes the first image to obtain the second image in step S2 above.

[0067] Solution 1:

[0068] In some embodiments, optionally, the first wearable display device processes the first image to obtain the second image, including:

[0069] Face feature point detection step: The first wearable display device performs face detection on the first image to obtain face feature points; specifically, it can first perform face detection on the first image, extract the face region, and perform feature point detection on the extracted face region to obtain face feature points. As Figure 3 shown, since the eyes and nose parts are blocked, the detected face feature points can be the face contour and the mouth.

[0070] Image generation step: The first wearable display device captures a third image of a preset angle of the three-dimensional face model of the first wearer to generate the third image. Specifically, the virtual camera can be used to capture the three-dimensional face model of the first wearer in an initial pose to obtain the third image. Among them, the initial preset angle can be set in advance or determined randomly.

[0071] Judgment step: The first wearable display device determines whether the face feature points of the currently captured third image match the face feature points of the first image. Optionally, face detection can be performed on the third image first to obtain face feature points, and the face feature points in the first image are feature-matched with the face feature points in the third image. For example, the facial contour and mouth in the first image are feature-matched with the facial contour and mouth in the third image.

[0072] Adjustment step: If they do not match, adjust the preset angle, capture a new third image of the three-dimensional face model, and return to the judgment step. That is, adjust the preset angle, capture the image of the three-dimensional face model of the first wearer again to obtain a new third image, and continue to determine whether the face feature points of the currently captured third image match the face feature points of the first image. That is, continuously adjust the pose of the virtual camera until the face feature points in the first image match the face feature points in the third image. In the embodiment of the present invention, the so-called matching may be that the matching degree is greater than a preset ratio. For example, if it is greater than 95%, it is considered a match. The adjustment of the preset angle can be adjusted according to a preset step size. For example, adjust by 1 degree at a time, etc.

[0073] Image acquisition step: If they match, obtain a second image according to the currently captured third image and the first image. The second image is an image of the facial area corresponding to the occlusion area of the first wearable display device in the first image in the third image. That is, the second image is an image of the intercepted area from the third image, and the intercepted area is the occlusion area of the first wearable display device in the first image. Please refer to Figure 4 , P1 is the first image, and P3 is the third image. Figure 4 P3 in

[0074] Since the position of the eyeballs in the second image is not the real-time position of the eyeballs of the first wearer, it is necessary to replace the eyeballs in the second image with the eyeballs in the eyeball image of the first wearer captured by the eyeball tracking camera in real time to obtain the final display image to be output, and use the outer screen of the first wearable display device to display it. In this way, the person wearing the second device can see the complete face of the first wearer, and the experience is more realistic.

[0075] In some embodiments, optionally, before the first wearable display device processes the first image to obtain a second image, the method further includes: the first wearable display device obtains multiple facial depth images of the first wearer captured by a front structured light camera of the first wearable display device, calculates the three-dimensional point cloud and texture data of the face of the first wearer according to the multiple facial depth images, and creates a three-dimensional facial model of the first wearer according to the three-dimensional point cloud and texture data of the face of the first wearer.

[0076] That is, in the embodiments of the present invention, the first wearable display device needs to include a front structured light camera. As Figure 5 shown, when the first wearer needs to create a three-dimensional facial model, the structured light camera can be used to align with the wearer's face for facial scanning, for example, scanning 180 degrees (or a larger range) to obtain multiple facial depth images of the first wearer. As Figure 6 shown, the structured light camera in the embodiments of the present invention may include: a structured light projection device (such as a laser), a camera, and an image acquisition and processing system. The scanning process is that the structured light projection device emits light onto the object to be measured (such as a human face), the camera captures the three-dimensional light pattern (light strip) formed on the object to be measured, obtains a depth image, and the captured depth image is processed by the image acquisition and processing system to obtain the three-dimensional point cloud and texture data of the object to be measured. Among them, when the relative positions of the camera and the structured light projection device are fixed, the degree of light distortion projected onto the object to be measured depends on the depth of the object surface, so a depth image can be obtained, and thus the three-dimensional point cloud and texture data of the human face can be calculated. In the embodiments of the present invention, the position of the camera at the initial acquisition moment can be set as the three-dimensional coordinate origin Os to provide a data source for subsequent frame reconstruction.

[0077] In the embodiments of the present invention, optionally, the first wearable display device may further include a voice prompt system for, during the facial scanning process, prompting the user to perform operations such as turning the head and nodding in a voice to obtain more comprehensive three-dimensional facial information.

[0078] In the above embodiments, the three-dimensional facial model of the wearer can be conveniently obtained through the first wearable display device.

[0079] In some other embodiments, optionally, before the first wearable display device processes the first image to obtain a second image, the method further includes: the first wearable display device receives the three-dimensional facial model of the first wearer sent by a third device and saves it. In this embodiment, the first wearable display device may not have a front structured light camera, thereby reducing costs.

[0080] Please refer to Figure 7 , Figure 7It is a schematic flowchart of the interactive display method of the first solution of the present invention. In this embodiment, a three-dimensional face model of the first wearer is obtained and saved in advance by using a first wearable display device. When the first wearer uses the first wearable display device, it is detected in real time whether there is a person wearing a second device approaching. If it is detected that someone is approaching, a connection is established with the second device, and a first image of the first wearer captured by the second device is obtained. The facial feature points in the first image are detected and feature-matched with the facial feature points in the third image of the three-dimensional face model captured by the virtual camera to obtain a third image matching the first image, and a second image is obtained based on the matched third image and the first image. The second image is a facial area image in the third image corresponding to the occlusion area of the first wearable display device in the first image. An eye image captured by an eye tracking camera is obtained in real time, and the eyes in the eye image are used to replace the eyes in the second image (i.e., image synthesis) to obtain an image to be displayed, and the outer screen is used for display.

[0081] Solution Two:

[0082] In some embodiments, optionally, please refer to Figure 8 , the processing of the first image by the first wearable display device to obtain the second image includes:

[0083] The first wearable display device inputs the first image into a neural network to obtain a fourth image, which is a facial image of the first wearer without wearing the first wearable display device. The neural network is trained by multiple groups of control images, and each group of the control images includes a facial image of the first wearer wearing the first wearable display device and a facial image of the first wearer not wearing the first wearable display device;

[0084] The first wearable display device obtains a second image based on the first image and the fourth image. The second image is a facial area image in the fourth image corresponding to the occlusion area of the first wearable display device in the first image.

[0085] That is, the first image and the fourth image are compared to obtain the facial area in the fourth image corresponding to the occlusion area of the first wearable display device in the first image, and the facial area is intercepted to obtain the second image.

[0086] In an embodiment of the present invention, the neural network may be, for example, a GAN network, etc., which is pre-trained and saved in the first wearable display device. A large number of facial images of users (including facial images of users wearing the first wearable display device and facial images of users not wearing the first wearable display device) can be used to train the neural network, so that the neural network learns to accurately infer and reconstruct the facial image of a user not wearing the first wearable display device from the facial image of the user wearing the first wearable display device. That is, its input is a facial image of a user wearing the first wearable display device, and the output is a facial image of the user not wearing the first wearable display device.

[0087] In an embodiment of the present invention, when training the neural network, multiple groups of control groups can be divided into a training set and a test set to make the performance of the trained neural network better.

[0088] It should be noted that if the second device is also an MR device or an AR device, in an embodiment of the present invention, after the first wearable display device establishes a connection with the second device, it can also capture a fifth image of the second wearer wearing the second device (i.e., an image of the second wearer with the face blocked by the second device) and send it to the second device, so that the second device executes the above interactive display method, and the blocked facial information of the second wearer can also be displayed on the outer screen of the second device.

[0089] That is, the interactive display method in an embodiment of the present invention may further include: the first wearable display device captures a fifth image of the second wearer wearing the second device and sends the fifth image of the second wearer to the second device.

[0090] In an embodiment of the present invention, the first wearable display device can capture the fifth image of the second wearer in real time and send it to the second device in real time. However, if images are continuously captured and sent, the performance requirements for the wearable display device are relatively high. Therefore, optionally, images may not be captured and sent in real time. Instead, after capturing an image once, if the relative pose between the first wearable display device and the second device does not change, the capture can be stopped, and when the relative pose between the first wearable display device and the second device changes, the capture and sending of images can be continued to save resources.

[0091] That is, the first wearable display device captures a fifth image of the second wearer wearing the second device, and sends the fifth image of the second wearer to the second device. After that, it further includes: the first wearable display device detects whether the relative pose with the second device has changed. When it detects that the relative pose with the second device has not changed, it stops capturing. When it detects that the relative pose with the second device has changed, it continues to capture the fifth image of the second wearer wearing the second device, and sends the currently captured fifth image of the second wearer to the second device.

[0092] Of course, the same applies to the second device side. After capturing an image of the first wearer once, it can detect whether the relative pose with the first wearable display device has changed. If it has not changed, it stops capturing and sending images. At this time, the first wearable device can stop updating the second image. Of course, it should be noted that the image to be displayed also needs to be updated according to the eye images captured in real time. When receiving the newly obtained first image sent by the second device again, it restarts the step of obtaining the second image and updates the second image, which will help save the resources of the first wearable device.

[0093] It should be noted that when the second device receives the fifth image, it can process the fifth image in the same way as the first image to obtain a sixth image. The sixth image is a facial area image corresponding to the occluded area of the second device in the fifth image. And the second wearable device can also obtain the eye image of the second wearer captured by the eye tracking camera of the second device, replace the eyes in the sixth image with the eyes in the eye image to obtain the image to be displayed, and display it on the outer screen of the second device, or send the image to be displayed to the first wearable display device for display on the inner screen of the first wearable display device.

[0094] Similarly, the first wearable display device can synthesize the fifth image captured by itself (the image of the second wearer wearing the second device) with the received image to be displayed (using the image to be displayed to replace the area occluded by the second device in the fifth image) to obtain a synthesized image, and display it on the inner screen.

[0095] Please refer to Figure 9 , an embodiment of the present invention further provides a wearable display device 100, including:

[0096] A receiving module 101, configured to receive the first image captured by the second device after establishing a connection with the second device. The first image is an image of the first wearer wearing the first wearable display device;

[0097] The first acquisition module 102 is configured to process the first image to obtain a second image, where the second image is a facial region image corresponding to the occlusion region of the first wearable display device in the first image;

[0098] The second acquisition module 103 is configured to deliberately acquire the eye image of the first wearer captured by the eye tracking camera of the first wearable display device, and replace the eyes in the second image with the eyes in the eye image to obtain an image to be displayed;

[0099] The display module 104 is configured to display the image to be displayed on the outer screen of the first wearable display device, or send the image to be displayed to the second device for display on the inner screen of the second device.

[0100] In the embodiments of the present invention, the picture displayed on the outer screen of the wearable display device, or the picture displayed on the inner screen of the interaction device (second device), includes both the picture of the user's real eye dynamics and the picture of the user's face occluded by the wearable display device, which can increase the naturalness of the user's communication with the surrounding people and improve the user experience.

[0101] In some embodiments, optionally, the first acquisition module 102 is configured to perform the following steps:

[0102] Facial feature point detection step: The first wearable display device performs face detection on the first image to obtain facial feature points;

[0103] Image generation step: The first wearable display device captures a third image of a preset angle of the three-dimensional facial model of the first wearer and generates a third image;

[0104] Judgment step: The first wearable display device determines whether the currently captured third image matches the facial feature points of the first image;

[0105] Adjustment step: If not matching, adjust the preset angle, capture a new third image of the three-dimensional facial model, and return to the judgment step;

[0106] Image acquisition step: If matching, obtain a second image according to the currently captured third image and the first image, where the second image is the facial region image in the third image corresponding to the occlusion region of the first wearable display device in the first image.

[0107] In some embodiments, optionally, the wearable display device 100 further includes:

[0108] A 3D model creation module, configured to obtain multiple facial depth images of a first wearer captured by a front structured light camera of the first wearable display device, calculate 3D point cloud and texture data of the face of the first wearer based on the multiple facial depth images, and create a 3D facial model of the first wearer according to the 3D point cloud and texture data of the face of the first wearer;

[0109] Or

[0110] A receiving module, configured to receive the 3D facial model of the first wearer sent by a third device and save it.

[0111] In some embodiments, optionally, the first obtaining module 102 is configured to input the first image into a neural network to obtain a fourth image, where the fourth image is a facial image of the first wearer without wearing the first wearable display device, and the neural network is trained by multiple groups of control images, where each group of the control images includes a facial image of the first wearer wearing the first wearable display device and a facial image of the first wearer not wearing the first wearable display device; obtain a second image according to the first image and the fourth image, where the second image is a facial area image in the fourth image corresponding to the occlusion area of the first wearable display device in the first image.

[0112] In some embodiments, optionally, the wearable display device 100 further includes:

[0113] A connection module, configured to detect whether there is another person approaching, and if it is detected that there is another person approaching, send a broadcast message, where the broadcast message is used to request to establish a connection.

[0114] In some embodiments, optionally, the wearable display device 100 further includes:

[0115] A shooting and sending module, configured to shoot a fifth image of a second wearer wearing the second device and send the fifth image of the second wearer to the second device.

[0116] In some embodiments, optionally, the wearable display device 100 further includes:

[0117] A processing module, configured to detect whether the relative pose with the second device has changed. When it is not detected that the relative pose with the second device has changed, stop shooting. When it is detected that the relative pose with the second device has changed, continue to shoot the fifth image of the second wearer wearing the second device and send the currently shot fifth image of the second wearer to the second device.

[0118] Please refer to Figure 10, an embodiment of the present invention further provides a wearable display device 200, including a processor 201, a memory 202, and a computer program stored on the memory 202 and executable on the processor 201. When the computer program is executed by the processor 201, it implements each process of the above-mentioned interactive display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0119] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each process of the above-mentioned interactive display method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0120] An embodiment of the present application further provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement each process of the above Figure 1 shown method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0121] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0122] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0123] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit of the present invention and the scope protected by the claims, and all of them fall within the protection scope of the present invention.

Claims

1. An interactive display method, characterized in that: include: After the first wearable display device establishes a connection with the second device, receiving a first image captured by the second device, where the first image is an image of a first wearer wearing the first wearable display device; The first wearable display device processes the first image to obtain a second image, where the second image is a facial region image corresponding to the occluded region of the first wearable display device in the first image; The first wearable display device obtains an eyeball image of the first wearer captured in real time by an eyeball tracking camera of the first wearable display device, and replaces the eyeball in the second image with the eyeball in the eyeball image to obtain an image to be displayed; The first wearable display device displays the image to be displayed on an external screen of the first wearable display device, or sends the image to be displayed to the second device to be displayed on an internal screen of the second device.

2. The method according to claim 1, characterized in that The first wearable display device processes the first image to obtain a second image, including: Face feature point detection step: the first wearable display device performs face detection on the first image to obtain face feature points; Image generating step: the first wearable display device captures a third image of a three-dimensional facial model of the first wearer at a preset angle; Determination step: the first wearable display device determines whether the facial feature points of the currently captured third image match those of the first image; Adjustment step: if there is no match, adjust the preset angle, capture a new third image of the three-dimensional face model, and return to the determination step; Image acquisition step: If a match is found, a second image is acquired based on the currently captured third image and the first image, wherein the second image is a facial area image in the third image corresponding to an occluded area of ​​the first wearable display device in the first image.

3. The method according to claim 2, characterized in that The first wearable display device processes the first image to obtain the second image, and the process further includes: The first wearable display device obtains a plurality of facial depth images of the first wearer taken by a front structured light camera of the first wearable display device, calculates a three-dimensional point cloud and texture data of the face of the first wearer according to the plurality of facial depth images, and creates a three-dimensional facial model of the first wearer according to the three-dimensional point cloud and texture data of the face of the first wearer; or The first wearable display device receives and saves the three-dimensional face model of the first wearer sent by the third device.

4. The method according to claim 1, characterized in that: The first wearable display device processes the first image to obtain a second image, including: The first wearable display device inputs the first image into a neural network to obtain a fourth image, wherein the fourth image is a facial image of the first wearer who is not wearing the first wearable display device, and the neural network is obtained by training multiple groups of control group images, wherein each group of the control group images includes a facial image of the first wearer who is wearing the first wearable display device and a facial image of the first wearer who is not wearing the first wearable display device; The first wearable display device obtains a second image based on the first image and the fourth image, where the second image is a face area image in the fourth image corresponding to the occluded area of ​​the first wearable display device in the first image.

5. The method according to claim 1, characterized in that Also includes: The first wearable display device detects whether there is another person approaching, and if it is detected that there is another person approaching, sends a broadcast message, where the broadcast message is used to request to establish a connection.

6. The method according to claim 1, characterized in that Also includes: The first wearable display device captures a fifth image of a second wearer wearing the second device, and sends the fifth image of the second wearer to the second device.

7. The method according to claim 6, characterized in that The first wearable display device captures a fifth image of a second wearer wearing the second device, and sends the fifth image of the second wearer to the second device, and then further includes: The first wearable display device detects whether the relative posture between it and the second device has changed. When no change in the relative posture between it and the second device is detected, shooting is stopped. When a change in the relative posture between it and the second device is detected, the fifth image of the second wearer wearing the second wearer is continued to be shot, and the fifth image of the second wearer currently shot is sent to the second device.

8. A wearable display device, characterized in that: include: a receiving module, configured to receive a first image captured by a second device after establishing a connection with the second device, wherein the first image is an image of a first wearer wearing the first wearable display device; A first obtaining module, configured to process the first image to obtain a second image, where the second image is a facial region image corresponding to an occluded region of the first wearable display device in the first image; a second obtaining module, which intentionally obtains an eyeball image of the first wearer captured in real time by an eyeball tracking camera of the first wearable display device, and replaces the eyeball in the second image with the eyeball in the eyeball image to obtain an image to be displayed; The display module is used to display the image to be displayed on the external screen of the first wearable display device, or to send the image to be displayed to the second device so as to be displayed on the internal screen of the second device.

9. A wearable display device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the interactive display method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the interactive display method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the interactive display method according to any one of claims 1 to 7.

Images