Video communication method and system based on three-dimensional display

Abstract

This article discloses a video communication method based on three-dimensional display, including: a first device obtains information of a first gaze point of a first user at a first moment and sends it to a second device; the second device captures a first image of the second user through m cameras, determines a first high / low definition area of ​​the first image according to the first gaze point, encodes the first high definition and low definition areas respectively and sends the encoded data to the first device; the first device decodes to obtain m second images, determines a second high definition area and a second low definition area of ​​the second image according to the offset of the current gaze point relative to the first gaze point; calculates and renders the second high definition area through a first neural network to obtain a first three-dimensional model, calculates and renders the second low definition area through a second neural network to obtain a second three-dimensional model, splices the two to obtain a third three-dimensional model, and determines and displays the target display position of the third three-dimensional model on the display screen according to the second gaze point. This article can realize real-time three-dimensional communication and save communication bandwidth and computing resources.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a video communication method and system based on three-dimensional display. Background Art

[0002] 3D display has been recognized as the ultimate dream of display technology development. After years of technological development, there are two major stereoscopic display technology systems on the market: glasses-type and naked-eye-type. Among them, naked-eye 3D display technology is also called automatic stereoscopic display technology, which is the future development trend of 3D display technology.

[0003] The display content of traditional naked-eye 3D display devices is all pre-made playback content sources. If naked-eye 3D display devices are to be used for real-time communication, a real 3D scene needs to be created. At this time, multiple cameras take photos synchronously at multiple angles, and calculate based on these photos to generate a 3D model, and then paste the image texture onto the model for display. Due to the limitations of bandwidth and graphics card capabilities, it is difficult to meet the real-time requirements, so naked-eye 3D display devices are difficult to use for real-time communication. Summary of the invention

[0004] The embodiment of the present application provides a video communication method based on three-dimensional display, including:

[0005] The first device obtains information about the first gaze point of the first user at the first moment and sends it to the second device; after receiving the information about the first gaze point, the second device uses m cameras to shoot a first image of the second user, determines a first high-definition area and a first low-definition area of ​​each first image according to the information about the first gaze point, encodes the first high-definition area and the first low-definition area respectively so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, and sends the data of the encoded m first images to the first device; wherein the area around the first gaze point is the first high-definition area, and the other areas except the first high-definition area are the first low-definition areas; m is greater than or equal to 2;

[0006] The first device decodes the encoded data of the m first images to obtain m second images, obtains information of the second gaze point of the first user at the second moment, determines an offset of the second gaze point relative to the first gaze point, and determines a second high-definition area and a second low-definition area of ​​the m second images according to the offset; wherein the area around the second gaze point is the second high-definition area, and the other areas except the second high-definition area are the second low-definition area;

[0007] The first device calculates and renders the second high-definition area of ​​the m second images through the first neural network to obtain a first three-dimensional model, calculates and renders the second low-definition area of ​​the m second images through the second neural network to obtain a second three-dimensional model, and splices the first three-dimensional model and the second three-dimensional model to obtain a third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network;

[0008] The first device determines a target display position of the third three-dimensional model on the display screen according to the information of the second gaze point, and displays the third three-dimensional model at the target display position;

[0009] Among them, the first device and the second device are three-dimensional display devices.

[0010] The embodiment of the present application provides a video communication system based on three-dimensional display, including:

[0011] A first device is configured to obtain information about a first gaze point of a first user at a first moment and send it to a second device, receive data of m encoded first images sent by the second device, decode the data of the m encoded first images to obtain m second images, and obtain information about a second gaze point of the first user at a second moment, determine an offset of the second gaze point relative to the first gaze point, and determine a second high-definition area and a second low-definition area of ​​the m second images according to the offset; wherein the area around the second gaze point is the second high-definition area, and the other areas except the second high-definition area are the second low-definition area; calculate and render the second high-definition area of ​​the m second images through a first neural network to obtain a first three-dimensional model, calculate and render the second low-definition area of ​​the m second images through a second neural network to obtain a second three-dimensional model, and splice the first three-dimensional model and the second three-dimensional model to obtain a third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network; determine a target display position of the third three-dimensional model on the display screen according to the information of the second gaze point, and display the third three-dimensional model at the target display position;

[0012] The second device is configured to, after receiving the information of the first gaze point, shoot a first image of the second user through m cameras, determine a first high-definition area and a first low-definition area of ​​each first image according to the information of the first gaze point, encode the first high-definition area and the first low-definition area respectively so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, and send the data of the m encoded first images to the first device; wherein the area around the first gaze point is the first high-definition area, and the other areas except the first high-definition area are the first low-definition areas; m is greater than or equal to 2;

[0013] Among them, the first device and the second device are three-dimensional display devices.

[0014] In the video communication method and system based on three-dimensional display provided by the embodiment of the present application, the first device sends the gaze point information of the first user to the second device, and the second device determines the high-definition area (the area around the gaze point) and the low-definition area of ​​the m first images of the second user taken by m cameras according to the gaze point information, and encodes the high-definition area and the low-definition area using different encoding methods so that the image resolution of the encoded high-definition area is higher than the image resolution of the low-definition area. The transmission bandwidth occupied by the encoded image can be saved by partition encoding, and since the area around the gaze point is encoded in high definition, the image clarity of the area concerned by the first user is not affected while saving bandwidth. The second device sends the data of the m encoded first images to the first device, and the first device decodes the data of the m encoded first images to obtain the m second images, and determines the offset of the current gaze point (second gaze point) of the first user relative to the previous gaze point (first gaze point), determines the high-definition area and the low-definition area of ​​the m second images according to the offset, calculates and renders the high-definition area of ​​the m second images by the first neural network to obtain the first three-dimensional model, calculates and renders the low-definition area of ​​the m second images by the second neural network to obtain the second three-dimensional model, and splices the first three-dimensional model and the second three-dimensional model to obtain the third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network. By using neural networks with different complexities to perform three-dimensional modeling on the high-definition area and the low-definition area respectively, computing resources can be saved, and since the area around the gaze point uses a neural network with higher complexity, the effect of three-dimensional modeling can be guaranteed. The first device determines the target display position of the third three-dimensional model on the display screen according to the information of the current gaze point, and displays the third three-dimensional model at the target display position. The video communication method and system of this article can realize real-time three-dimensional scene video communication and save communication bandwidth and computing resources.

[0015] Other aspects will be apparent upon reading and understanding the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0017] Figure 1 A flowchart of a video communication method based on three-dimensional display according to an embodiment of the present application;

[0018] Figure 2 A schematic diagram of the distribution of cameras on a display screen according to an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a second high definition area of ​​a second image according to an embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of smoothing (optimizing) the first curve (the second curve) according to an embodiment of the present application;

[0021] Figure 5 This is a structural diagram of a video communication system based on three-dimensional display according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present application. Although many possible feature combinations are shown in the drawings and discussed in the specific embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0023] The present application includes and contemplates combinations of features known to those of ordinary skill in the art. The embodiments and features disclosed in the present application may also be combined with any conventional features to form a unique invention scheme defined by the appended claims. Any feature of any embodiment may also be combined with features from other invention schemes to form another unique invention scheme defined by the appended claims. Therefore, it should be understood that any feature shown and / or discussed in the present application may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the appended claims and their equivalents, the embodiments are not subject to other restrictions. In addition, various modifications and changes may be made within the scope of protection of the appended claims.

[0024] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps of the specific order described. As will be understood by those of ordinary skill in the art, other sequences of steps are also possible. Therefore, the specific sequence of the steps set forth in the specification should not be interpreted as a limitation to the appended claims. In addition, the claims for the method and / or process should not be limited to the steps of performing them in the order written, and those skilled in the art can easily understand that these sequences can be changed and still remain within the spirit and scope of the embodiments of the present application.

[0025] like Figure 1 As shown, the embodiment of the present application provides a video communication method based on three-dimensional display, including:

[0026] Step S10, the first device obtains information about the first gaze point of the first user at the first moment and sends it to the second device; after receiving the information about the first gaze point, the second device captures the first image of the second user through m cameras, determines the first high-definition area and the first low-definition area of ​​each first image according to the information about the first gaze point, encodes the first high-definition area and the first low-definition area respectively so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, and sends the data of the encoded m first images to the first device; wherein the area around the first gaze point is the first high-definition area, and the other areas except the first high-definition area are the first low-definition areas; m is greater than or equal to 2;

[0027] Step S20, the first device decodes the encoded data of the m first images to obtain m second images, and obtains information of the second gaze point of the first user at the second moment, determines an offset of the second gaze point relative to the first gaze point, and determines a second high-definition area and a second low-definition area of ​​the m second images according to the offset; wherein the area around the second gaze point is the second high-definition area, and the other areas except the second high-definition area are the second low-definition area;

[0028] Step S30: the first device calculates and renders the second high-definition area of ​​the m second images through the first neural network to obtain a first three-dimensional model, calculates and renders the second low-definition area of ​​the m second images through the second neural network to obtain a second three-dimensional model, and splices the first three-dimensional model and the second three-dimensional model to obtain a third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network;

[0029] Step S40: the first device determines a target display position of the third three-dimensional model on the display screen according to the information of the second gaze point, and displays the third three-dimensional model at the target display position;

[0030] Among them, the first device and the second device are three-dimensional display devices.

[0031] In the video communication method based on three-dimensional display provided by the embodiment of the present application, the first device sends the gaze point information of the first user to the second device, and the second device determines the high-definition area (the area around the gaze point) and the low-definition area of ​​the m first images of the second user taken by m cameras according to the received gaze point information, and encodes the high-definition area and the low-definition area using different encoding methods so that the image resolution of the encoded high-definition area is higher than the image resolution of the low-definition area. The transmission bandwidth occupied by the encoded image can be saved by partition encoding, and because the area around the gaze point adopts high-definition encoding, the image clarity of the area concerned by the first user is not affected while saving bandwidth. The second device sends the data of the m encoded first images to the first device, and the first device decodes the data of the m encoded first images to obtain the m second images, and determines the offset of the current gaze point (second gaze point) of the first user relative to the previous gaze point (first gaze point), determines the high-definition area and the low-definition area of ​​the m second images according to the offset, calculates and renders the high-definition area of ​​the m second images through the first neural network to obtain the first three-dimensional model, calculates and renders the low-definition area of ​​the m second images through the second neural network to obtain the second three-dimensional model, and splices the first three-dimensional model and the second three-dimensional model to obtain the third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network. By using neural networks with different complexities to perform three-dimensional modeling on the high-definition area and the low-definition area respectively, computing resources can be saved, and since the area around the gaze point uses a neural network with higher complexity, the effect of three-dimensional modeling can be guaranteed. The first device determines the target display position of the third three-dimensional model on the display screen according to the information of the current gaze point, and displays the third three-dimensional model at the target display position. The video communication method of this article can realize real-time video communication of three-dimensional scenes and save communication bandwidth and computing resources.

[0032] In some exemplary embodiments, the first device obtains information about a first gaze point of a first user at a first moment, including:

[0033] The first device captures a facial image of the first user through a first camera at a first moment, performs facial feature point detection on the facial image, and if a face is detected, performs eye recognition in the face area, and marks the left eye area and the right eye area, recognizes the left pupil in the left eye area, and determines the relative position of the left pupil in the left eye area, recognizes the right pupil in the right eye area, and determines the relative position of the right pupil in the right eye area, determines the intersection position of the first user's binocular sight lines on the display screen of the first device according to the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and uses the intersection position as the first gaze point of the first user at the first moment.

[0034] In some exemplary embodiments, Figure 2 As shown, the first camera C1 is arranged in the middle of the upper frame of the display screen of the first device.

[0035] In some exemplary embodiments, video communication is performed between the first device and the second device via a remote network.

[0036] The remote network includes: a wireless communication network, a mobile communication network, a wired communication network, etc.

[0037] In some exemplary embodiments, the first device sends information about the first gaze point of the first user at the first moment to the second device, including: the first device sends information about the first gaze point of the first user at the first moment to the second device via a remote communication network.

[0038] In some exemplary embodiments, after the first device captures the facial image of the first user through the first camera, the method further includes:

[0039] The resolution of the facial image is reduced.

[0040] Reducing the resolution of the facial image can save computing resources and speed up computing time.

[0041] In some exemplary embodiments, Figure 2 As shown, the m cameras (camera C2-1, camera C2-2, camera C2-3, camera C2-4, camera C2-5, camera C2-6) are respectively arranged in the left half area and the right half area of ​​the upper frame of the display screen of the second device, the left half area and the right half area of ​​the lower frame, and the middle area of ​​the left frame and the middle area of ​​the right frame. That is, the six cameras are symmetrically distributed on the four frames of the display screen.

[0042] In some exemplary embodiments, respectively encoding the first high-definition area and the first low-definition area so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area includes:

[0043] Keeping the number of pixels in the first high definition area unchanged;

[0044] The number of pixels in the first low-definition area is compressed horizontally to 1 / N of the original number, or vertically to 1 / N of the original number; wherein N is greater than or equal to 2.

[0045] In some exemplary embodiments, compressing the number of pixels in the first low-definition area to 1 / N of the original number laterally comprises:

[0046] Starting from the first column of pixels in the first low-definition area, every N columns of pixels are compressed into a new column of pixels, and the pixel value of the new pixel is the average value or weighted average value of the pixel values ​​of the N columns of pixels.

[0047] In some exemplary embodiments, compressing the number of pixels in the first low-definition area vertically to 1 / N of the original number includes:

[0048] Starting from the first row of pixels in the first low-definition area, every N rows of pixels are compressed into a new row of pixels, and the pixel value of the new pixel is the average value or weighted average value of the pixel values ​​of the N rows of pixels.

[0049] In some exemplary embodiments, the gaze point is the intersection of the user's binocular sight lines on the display screen.

[0050] In some exemplary embodiments, the first device obtains information about a second gaze point of the first user at a second moment, including:

[0051] The first device captures a facial image of the first user through the first camera at the second moment, performs facial feature point detection on the facial image, and if a face is detected, performs eye recognition in the face area, and marks the left eye area and the right eye area, recognizes the left pupil in the left eye area, and determines the relative position of the left pupil in the left eye area, recognizes the right pupil in the right eye area, and determines the relative position of the right pupil in the right eye area, determines the intersection position of the first user's binocular sight lines on the display screen of the first device according to the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and uses the intersection position as the second gaze point of the first user at the second moment.

[0052] In some exemplary embodiments, the first device decodes the encoded data of the m first images to obtain the m second images, including:

[0053] For any encoded data of the first image, the first high-definition area and the first low-definition area of ​​the first image are decoded, and the low-definition area of ​​the first image is decompressed to obtain the second image.

[0054] In some exemplary embodiments, Figure 3 As shown, the first device determines the second high-definition area and the second low-definition area of ​​the m second images according to the offset, including:

[0055] For any second image, marking the same area on the second image as a high-definition reference area according to the position of the first high-definition area of ​​the first image used to generate the second image, and marking the same area on the second image as a low-definition reference area according to the position of the first low-definition area of ​​the first image used to generate the second image;

[0056] The high-definition reference area is translated according to the offset to obtain a second high-definition area, and the low-definition reference area is translated according to the offset to obtain a second low-definition area;

[0057] Keeping pixels in the area where the high-definition reference area and the second high-definition area overlap unchanged;

[0058] Processing pixels in the to-be-processed pixel area belonging to the second high-definition area in the low-definition reference area:

[0059] The following processing is performed on any row of target pixel rows from the 1st row to the cth row of pixels from top to bottom in the pixel area to be processed: the pixel values ​​of the ath column of pixels in the same row as the target pixel row and located in the high-definition reference area and the pixel values ​​of the ath column of pixels included in the target pixel row are plotted on the coordinate axis to generate a first curve, the first curve is smoothed, and the original pixel values ​​on the target pixel row are replaced by the new pixel values ​​corresponding to the target pixel row on the smoothed first curve; wherein a is the number of pixels included in the horizontal direction of the offset; and c is the bottommost row in the area where the high-definition reference area and the second high-definition area overlap;

[0060] The following processing is performed on any target pixel column from the 1st column to the dth column from left to right in the pixel area to be processed: the pixel values ​​of the b-row pixels in the same column as the target pixel column and the pixel values ​​of the b-row pixels included in the target pixel column are plotted on the coordinate axis to generate a second curve, the second curve is smoothed, and the original pixel values ​​on the target pixel column are replaced by the new pixel values ​​corresponding to the target pixel column on the smoothed second curve; wherein b is the number of pixels included in the offset in the longitudinal direction; and d is the rightmost column in the area where the high-definition reference area and the second high-definition area overlap.

[0061] In some exemplary embodiments, the smoothing process includes: Bessel cubic fitting.

[0062] Figure 4 This is a schematic diagram of smoothing (optimizing) the first curve (second curve). Before optimization, the right area on the first curve (second curve) corresponds to the original pixel value of the target pixel row (column), and the pixel value difference between different pixels is large. After optimization, the right area on the first curve (second curve) corresponds to the new pixel value of the target pixel row (column), and the pixel value difference between different pixels becomes smaller.

[0063] In some exemplary embodiments, the first neural network is a deep neural network, and the second neural network is a shallow neural network.

[0064] In some exemplary embodiments, the first device determines a target display position of the third three-dimensional model on the display screen according to information of the second gaze point, and displays the third three-dimensional model at the target display position, including:

[0065] The first device uses two left and right virtual cameras to shoot the third three-dimensional model according to the information of the second gaze point to obtain a left eye image and a right eye image, and combines the left eye image and the right eye image to generate a target image of the third three-dimensional model, wherein the left eye image is located to the left of the second gaze point, and the right eye image is located to the right of the second gaze point;

[0066] The target image is displayed on a display screen of the first device.

[0067] like Figure 5 As shown, an embodiment of the present application provides a video communication system based on three-dimensional display, including:

[0068] The first device 10 is configured to obtain information about a first gaze point of a first user at a first moment and send it to a second device, receive data of m encoded first images sent by the second device, decode the data of the m encoded first images to obtain m second images, and obtain information about a second gaze point of the first user at a second moment, determine an offset of the second gaze point relative to the first gaze point, and determine a second high-definition area and a second low-definition area of ​​the m second images according to the offset; wherein the area around the second gaze point is the second high-definition area, and the other areas except the second high-definition area are the second low-definition area; calculate and render the second high-definition area of ​​the m second images through a first neural network to obtain a first three-dimensional model, calculate and render the second low-definition area of ​​the m second images through a second neural network to obtain a second three-dimensional model, and splice the first three-dimensional model and the second three-dimensional model to obtain a third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network; determine a target display position of the third three-dimensional model on the display screen according to the information of the second gaze point, and display the third three-dimensional model at the target display position;

[0069] The second device 20 is configured to, after receiving the information of the first gaze point, shoot the first image of the second user through m cameras, determine the first high-definition area and the first low-definition area of ​​each first image according to the information of the first gaze point, encode the first high-definition area and the first low-definition area respectively so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, and send the data of the encoded m first images to the first device; wherein the area around the first gaze point is the first high-definition area, and the other areas except the first high-definition area are the first low-definition areas; m is greater than or equal to 2;

[0070] Among them, the first device and the second device are three-dimensional display devices.

[0071] In the video communication system based on three-dimensional display provided by the embodiment of the present application, the first device sends the gaze point information of the first user to the second device, and the second device determines the high-definition area (the area around the gaze point) and the low-definition area of ​​the m first images of the second user taken by m cameras according to the received gaze point information, and encodes the high-definition area and the low-definition area using different encoding methods so that the image resolution of the encoded high-definition area is higher than the image resolution of the low-definition area. The transmission bandwidth occupied by the encoded image can be saved by partition encoding, and since the area around the gaze point is encoded in high definition, the image clarity of the area concerned by the first user is not affected while saving bandwidth. The second device sends the data of the m encoded first images to the first device, and the first device decodes the data of the m encoded first images to obtain the m second images, and determines the offset of the current gaze point (second gaze point) of the first user relative to the previous gaze point (first gaze point), determines the high-definition area and the low-definition area of ​​the m second images according to the offset, calculates and renders the high-definition area of ​​the m second images through the first neural network to obtain the first three-dimensional model, calculates and renders the low-definition area of ​​the m second images through the second neural network to obtain the second three-dimensional model, and splices the first three-dimensional model and the second three-dimensional model to obtain the third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network. By using neural networks with different complexities to perform three-dimensional modeling on the high-definition area and the low-definition area respectively, computing resources can be saved, and since the area around the gaze point uses a neural network with higher complexity, the effect of three-dimensional modeling can be guaranteed. The first device determines the target display position of the third three-dimensional model on the display screen according to the information of the current gaze point, and displays the third three-dimensional model at the target display position. The video communication system of this article can realize real-time video communication of three-dimensional scenes and save communication bandwidth and computing resources.

[0072] In some exemplary embodiments, the first device is configured to obtain information about a first gaze point of a first user at a first moment in the following manner:

[0073] At a first moment, a facial image of the first user is captured by a first camera, and facial feature points are detected on the facial image; if a face is detected, eyes are recognized in the face area, and the left eye area and the right eye area are marked; the left pupil is recognized in the left eye area to determine the relative position of the left pupil in the left eye area; the right pupil is recognized in the right eye area to determine the relative position of the right pupil in the right eye area; the position of the intersection of the first user's binocular sight lines on the first device display screen is determined based on the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and the intersection position is used as the first gaze point of the first user at the first moment.

[0074] In some exemplary embodiments, the first camera C1 is disposed in the middle of an upper frame of a display screen of the first device.

[0075] In some exemplary embodiments, video communication is performed between the first device and the second device via a remote network.

[0076] In some exemplary embodiments, the first device is configured to send information about a first gaze point of a first user at a first moment to a second device in the following manner: the first device sends information about a first gaze point of a first user at a first moment to the second device via a remote communication network.

[0077] In some exemplary embodiments, the first device is further configured to reduce the resolution of the facial image of the first user after taking the facial image of the first user through the first camera.

[0078] Reducing the resolution of the facial image can save computing resources and speed up computing time.

[0079] In some exemplary embodiments, Figure 2 As shown, the m cameras (camera C2-1, camera C2-2, camera C2-3, camera C2-4, camera C2-5, camera C2-6) are respectively arranged in the left half area and the right half area of ​​the upper frame of the display screen of the second device, the left half area and the right half area of ​​the lower frame, and the middle area of ​​the left frame and the middle area of ​​the right frame. That is, the six cameras are symmetrically distributed on the four frames of the display screen.

[0080] In some exemplary embodiments, the first device is configured to encode the first high-definition area and the first low-definition area respectively in the following manner so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area:

[0081] Keeping the number of pixels in the first high definition area unchanged;

[0082] The number of pixels in the first low-definition area is compressed horizontally to 1 / N of the original number, or vertically to 1 / N of the original number; wherein N is greater than or equal to 2.

[0083] In some exemplary embodiments, compressing the number of pixels in the first low-definition area to 1 / N of the original number laterally comprises:

[0084] Starting from the first column of pixels in the first low-definition area, every N columns of pixels are compressed into a new column of pixels, and the pixel value of the new pixel is the average value or weighted average value of the pixel values ​​of the N columns of pixels.

[0085] In some exemplary embodiments, compressing the number of pixels in the first low-definition area vertically to 1 / N of the original number includes:

[0086] Starting from the first row of pixels in the first low-definition area, every N rows of pixels are compressed into a new row of pixels, and the pixel value of the new pixel is the average value or weighted average value of the pixel values ​​of the N rows of pixels.

[0087] In some exemplary embodiments, the first device is configured to obtain information about the second gaze point of the first user at the second moment in the following manner:

[0088] At the second moment, a facial image of the first user is captured by the first camera, and facial feature points are detected on the facial image; if a face is detected, eyes are recognized in the face area, and the left eye area and the right eye area are marked; the left pupil is recognized in the left eye area to determine the relative position of the left pupil in the left eye area; the right pupil is recognized in the right eye area to determine the relative position of the right pupil in the right eye area; the position of the intersection of the first user's binocular sight lines on the display screen of the first device is determined based on the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and the intersection position is used as the second gaze point of the first user at the second moment.

[0089] In some exemplary embodiments, the first device is configured to decode the encoded data of the m first images to obtain the m second images in the following manner:

[0090] For any encoded data of the first image, the first high-definition area and the first low-definition area of ​​the first image are decoded, and the low-definition area of ​​the first image is decompressed to obtain the second image.

[0091] In some exemplary embodiments, the first device is configured to determine the second high-definition area and the second low-definition area of ​​the m second images according to the offset in the following manner:

[0092] For any second image, marking the same area on the second image as a high-definition reference area according to the position of the first high-definition area of ​​the first image used to generate the second image, and marking the same area on the second image as a low-definition reference area according to the position of the first low-definition area of ​​the first image used to generate the second image;

[0093] The high-definition reference area is translated according to the offset to obtain a second high-definition area, and the low-definition reference area is translated according to the offset to obtain a second low-definition area;

[0094] Keeping pixels in the area where the high-definition reference area and the second high-definition area overlap unchanged;

[0095] Processing pixels in the to-be-processed pixel area belonging to the second high-definition area in the low-definition reference area:

[0096] The following processing is performed on any row of target pixel rows from the 1st row to the mth row of pixels from top to bottom in the pixel area to be processed: the pixel values ​​of the a-th column of pixels in the same row as the target pixel row and located in the high-definition reference area and the pixel values ​​of the a-th column of pixels included in the target pixel row are plotted on the coordinate axis to generate a first curve, the first curve is smoothed, and the original pixel values ​​on the target pixel row are replaced by the new pixel values ​​corresponding to the target pixel row on the smoothed first curve; wherein a is the number of pixels included in the horizontal direction of the offset; and m is the bottom row in the area where the high-definition reference area and the second high-definition area overlap;

[0097] The following processing is performed on any target pixel column from the 1st column to the nth column from left to right in the pixel area to be processed: the pixel values ​​of the b-row pixels in the same column as the target pixel column and the pixel values ​​of the b-row pixels included in the target pixel column are plotted on the coordinate axis to generate a second curve, the second curve is smoothed, and the original pixel values ​​on the target pixel column are replaced by the new pixel values ​​corresponding to the target pixel column on the smoothed second curve; wherein b is the number of pixels included in the offset in the longitudinal direction; and n is the rightmost column in the area where the high-definition reference area and the second high-definition area overlap.

[0098] In some exemplary embodiments, the smoothing process includes: Bessel cubic fitting.

[0099] In some exemplary embodiments, the first neural network is a deep neural network, and the second neural network is a shallow neural network.

[0100] In some exemplary embodiments, the first device is configured to determine a target display position of the third three-dimensional model on the display screen according to information of the second gaze point in the following manner, and display the third three-dimensional model at the target display position:

[0101] Using left and right virtual cameras to shoot the third three-dimensional model according to the information of the second gaze point to obtain a left eye image and a right eye image, and combining the left eye image and the right eye image to generate a target image of the third three-dimensional model, wherein the left eye image is located to the left of the second gaze point, and the right eye image is located to the right of the second gaze point;

[0102] The target image is displayed on a display screen of the first device.

[0103] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A video communication method based on three-dimensional display, comprising: The first device obtains information about a first gaze point of a first user at a first moment and sends the information to the second device; After receiving the information of the first gaze point, the second device uses m cameras to capture the first image of the second user, determines the first high-definition area and the first low-definition area of ​​each first image according to the information of the first gaze point, encodes the first high-definition area and the first low-definition area respectively so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, and sends the data of the m encoded first images to the first device; wherein the area around the first gaze point is the first high-definition area, and the other areas except the first high-definition area are the first low-definition areas; m is greater than or equal to 2; The first device decodes the encoded data of the m first images to obtain m second images, obtains information of the second gaze point of the first user at the second moment, determines an offset of the second gaze point relative to the first gaze point, and determines a second high-definition area and a second low-definition area of ​​the m second images according to the offset; wherein the area around the second gaze point is the second high-definition area, and the other areas except the second high-definition area are the second low-definition area; The first device calculates and renders the second high-definition area of ​​the m second images through the first neural network to obtain a first three-dimensional model, calculates and renders the second low-definition area of ​​the m second images through the second neural network to obtain a second three-dimensional model, and splices the first three-dimensional model and the second three-dimensional model to obtain a third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network; The first device determines a target display position of the third three-dimensional model on the display screen according to the information of the second gaze point, and displays the third three-dimensional model at the target display position; Among them, the first device and the second device are three-dimensional display devices.

2. The method according to claim 1, characterized in that: The first device obtains information about a first gaze point of a first user at a first moment, including: The first device captures a facial image of the first user through a first camera at a first moment, performs facial feature point detection on the facial image, and if a face is detected, performs eye recognition in the face area, and marks a left eye area and a right eye area, recognizes a left pupil in the left eye area, and determines a relative position of the left pupil in the left eye area, recognizes a right pupil in the right eye area, and determines a relative position of the right pupil in the right eye area, determines a position of an intersection of the first user's binocular sight lines on the display screen of the first device according to the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and uses the intersection position as a first gaze point of the first user at the first moment; The first device obtains information about a second gaze point of the first user at a second moment, including: The first device captures a facial image of the first user through the first camera at the second moment, performs facial feature point detection on the facial image, and if a face is detected, performs eye recognition in the face area, and marks the left eye area and the right eye area, recognizes the left pupil in the left eye area, and determines the relative position of the left pupil in the left eye area, recognizes the right pupil in the right eye area, and determines the relative position of the right pupil in the right eye area, determines the intersection position of the first user's binocular sight lines on the display screen of the first device according to the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and uses the intersection position as the second gaze point of the first user at the second moment.

3. The method according to claim 1, characterized in that: The first device and the second device perform video communication via a remote network.

4. The method according to claim 1, characterized in that: The first high-definition area and the first low-definition area are respectively encoded so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, comprising: Keeping the number of pixels in the first high definition area unchanged; The number of pixels in the first low-definition area is compressed horizontally to 1 / N of the original number, or vertically to 1 / N of the original number; wherein N is greater than or equal to 2.

5. The method according to claim 4, characterized in that: Compressing the number of pixels in the first low-definition area horizontally to 1 / N of the original number, including: Starting from the first column of pixels in the first low-definition area, every N columns of pixels are compressed into a new column of pixels, and the pixel value of the new pixel is the average value or weighted average value of the pixel values ​​of the N columns of pixels; Compressing the number of pixels in the first low-definition area vertically to 1 / N of the original number, including: Starting from the first row of pixels in the first low-definition area, every N rows of pixels are compressed into a new row of pixels, and the pixel value of the new pixel is the average value or weighted average value of the pixel values ​​of the N rows of pixels.

6. The method according to claim 4 or 5, characterized in that: The first device decodes the encoded data of the m first images to obtain the m second images, including: For any encoded data of the first image, the first high-definition area and the first low-definition area of ​​the first image are decoded, and the low-definition area of ​​the first image is decompressed to obtain the second image.

7. The method according to claim 1, characterized in that: The first device determines the second high-definition area and the second low-definition area of ​​the m second images according to the offset, including: For any second image, marking the same area on the second image as a high-definition reference area according to the position of the first high-definition area of ​​the first image used to generate the second image, and marking the same area on the second image as a low-definition reference area according to the position of the first low-definition area of ​​the first image used to generate the second image; According to the offset, the high-definition reference area is translated to obtain a second high-definition area, and according to the offset, the low-definition reference area is translated to obtain a second low-definition area; Keeping pixels in the area where the high-definition reference area and the second high-definition area overlap unchanged; Processing pixels in the to-be-processed pixel area belonging to the second high-definition area in the low-definition reference area: The following processing is performed on any row of target pixel rows from the 1st row to the cth row of pixels from top to bottom in the pixel area to be processed: the pixel values ​​of the ath column of pixels in the same row as the target pixel row and located in the high-definition reference area and the pixel values ​​of the ath column of pixels included in the target pixel row are plotted on the coordinate axis to generate a first curve, the first curve is smoothed, and the original pixel values ​​on the target pixel row are replaced by the new pixel values ​​corresponding to the target pixel row on the smoothed first curve; wherein a is the number of pixels included in the horizontal direction of the offset; and c is the bottommost row in the area where the high-definition reference area and the second high-definition area overlap; The following processing is performed on any target pixel column from the 1st column to the dth column from left to right in the pixel area to be processed: the pixel values ​​of the b-row pixels in the same column as the target pixel column and the pixel values ​​of the b-row pixels included in the target pixel column are plotted on the coordinate axis to generate a second curve, the second curve is smoothed, and the original pixel values ​​on the target pixel column are replaced by the new pixel values ​​corresponding to the target pixel column on the smoothed second curve; wherein b is the number of pixels included in the offset in the longitudinal direction; and d is the rightmost column in the area where the high-definition reference area and the second high-definition area overlap.

8. The method according to claim 1, characterized in that: The first device determines a target display position of a third three-dimensional model on a display screen according to information of the second gaze point, and displays the third three-dimensional model at the target display position, including: The first device uses two left and right virtual cameras to shoot the third three-dimensional model according to the information of the second gaze point to obtain a left eye image and a right eye image, and combines the left eye image and the right eye image to generate a target image of the third three-dimensional model, wherein the left eye image is located to the left of the second gaze point, and the right eye image is located to the right of the second gaze point; The target image is displayed on a display screen of the first device.

9. A video communication system based on three-dimensional display, comprising: A first device is configured to obtain information about a first gaze point of a first user at a first moment and send it to a second device, receive data of m encoded first images sent by the second device, decode the data of the m encoded first images to obtain m second images, and obtain information about a second gaze point of the first user at a second moment, determine an offset of the second gaze point relative to the first gaze point, and determine a second high-definition area and a second low-definition area of ​​the m second images according to the offset; wherein the area around the second gaze point is the second high-definition area, and the other areas except the second high-definition area are the second low-definition area; calculate and render the second high-definition area of ​​the m second images through a first neural network to obtain a first three-dimensional model, calculate and render the second low-definition area of ​​the m second images through a second neural network to obtain a second three-dimensional model, and splice the first three-dimensional model and the second three-dimensional model to obtain a third three-dimensional model; wherein the complexity of the first neural network is higher than the complexity of the second neural network; determine a target display position of the third three-dimensional model on the display screen according to the information of the second gaze point, and display the third three-dimensional model at the target display position; The second device is configured to, after receiving the information of the first gaze point, shoot a first image of the second user through m cameras, determine a first high-definition area and a first low-definition area of ​​each first image according to the information of the first gaze point, encode the first high-definition area and the first low-definition area respectively so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area, and send the data of the m encoded first images to the first device; wherein the area around the first gaze point is the first high-definition area, and the other areas except the first high-definition area are the first low-definition areas; m is greater than or equal to 2; Among them, the first device and the second device are three-dimensional display devices.

10. The system according to claim 9, characterized in that: The first device is configured to obtain information about a first gaze point of a first user at a first moment in the following manner: At a first moment, a facial image of a first user is captured by a first camera, and facial feature points are detected on the facial image. If a face is detected, eyes are recognized in the face area, and a left eye area and a right eye area are marked. A left pupil is recognized in the left eye area to determine the relative position of the left pupil in the left eye area, and a right pupil is recognized in the right eye area to determine the relative position of the right pupil in the right eye area. A position of an intersection of the first user's binocular sight lines on the display screen of the first device is determined based on the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and the intersection is used as a first gaze point of the first user at the first moment; The first device is configured to obtain information about a second gaze point of the first user at a second moment in the following manner: At the second moment, a facial image of the first user is captured by the first camera, and facial feature points are detected on the facial image; if a face is detected, eyes are recognized in the face area, and the left eye area and the right eye area are marked; the left pupil is recognized in the left eye area to determine the relative position of the left pupil in the left eye area; the right pupil is recognized in the right eye area to determine the relative position of the right pupil in the right eye area; the position of the intersection of the first user's binocular sight lines on the display screen of the first device is determined based on the relative position of the left pupil in the left eye area and the relative position of the right pupil in the right eye area, and the intersection position is used as the second gaze point of the first user at the second moment.

11. The system according to claim 9, characterized in that: The first device is configured to encode the first high-definition area and the first low-definition area respectively in the following manner so that the image resolution of the encoded first high-definition area is higher than the image resolution of the first low-definition area: Keeping the number of pixels in the first high definition area unchanged; The number of pixels in the first low-definition area is compressed horizontally to 1 / N of the original number, or vertically to 1 / N of the original number; wherein N is greater than or equal to 2.

12. The system according to claim 11, characterized in that: The first device is configured to decode the encoded data of the m first images to obtain the m second images in the following manner: For any encoded data of the first image, the first high-definition area and the first low-definition area of ​​the first image are decoded, and the low-definition area of ​​the first image is decompressed to obtain the second image.

13. The system of claim 9, wherein: The first device is configured to determine the second high-definition area and the second low-definition area of ​​the m second images according to the offset in the following manner: For any second image, marking the same area on the second image as a high-definition reference area according to the position of the first high-definition area of ​​the first image used to generate the second image, and marking the same area on the second image as a low-definition reference area according to the position of the first low-definition area of ​​the first image used to generate the second image; According to the offset, the high-definition reference area is translated to obtain a second high-definition area, and according to the offset, the low-definition reference area is translated to obtain a second low-definition area; Keeping pixels in the area where the high-definition reference area and the second high-definition area overlap unchanged; Processing pixels in the to-be-processed pixel area belonging to the second high-definition area in the low-definition reference area: The following processing is performed on any row of target pixel rows from the 1st row to the mth row of pixels from top to bottom in the pixel area to be processed: the pixel values ​​of the a-th column of pixels in the same row as the target pixel row and located in the high-definition reference area and the pixel values ​​of the a-th column of pixels included in the target pixel row are plotted on the coordinate axis to generate a first curve, the first curve is smoothed, and the original pixel values ​​on the target pixel row are replaced by the new pixel values ​​corresponding to the target pixel row on the smoothed first curve; wherein a is the number of pixels included in the horizontal direction of the offset; and m is the bottom row in the area where the high-definition reference area and the second high-definition area overlap; The following processing is performed on any target pixel column from the 1st column to the nth column from left to right in the pixel area to be processed: the pixel values ​​of the b-row pixels in the same column as the target pixel column and the pixel values ​​of the b-row pixels included in the target pixel column are plotted on the coordinate axis to generate a second curve, the second curve is smoothed, and the original pixel values ​​on the target pixel column are replaced by the new pixel values ​​corresponding to the target pixel column on the smoothed second curve; wherein b is the number of pixels included in the offset in the longitudinal direction; and n is the rightmost column in the area where the high-definition reference area and the second high-definition area overlap.

14. The system of claim 9, wherein: The first device is configured to determine a target display position of a third three-dimensional model on a display screen according to information of a second gaze point in the following manner, and display the third three-dimensional model at the target display position: Using left and right virtual cameras to shoot the third three-dimensional model according to the information of the second gaze point to obtain a left eye image and a right eye image, and combining the left eye image and the right eye image to generate a target image of the third three-dimensional model, wherein the left eye image is located to the left of the second gaze point, and the right eye image is located to the right of the second gaze point; The target image is displayed on a display screen of the first device.

15. The system of claim 9, wherein: The first camera is arranged in the middle of the upper frame of the display screen of the first device; The m cameras are respectively arranged in the left half area and the right half area of ​​the upper frame of the display screen of the second device, the left half area and the right half area of ​​the lower frame, the middle area of ​​the left frame and the middle area of ​​the right frame.

Images