Free viewpoint video playing method and device, storage medium and electronic equipment

Abstract

The invention discloses a free viewpoint video playing method and device, a storage medium and electronic equipment, and relates to the technical field of video playing. The method comprises the following steps: positioning a sight area of a user by adopting an eyeball tracking method based on a preset window size to obtain an effective window position; performing calculation processing based on the effective window position and a preset rotation angle value to obtain each neighborhood window position; performing image processing on a pre-acquired full-view video image based on the effective window position and each neighborhood window position to obtain a first view video image corresponding to the effective window position and a second view video image corresponding to each neighborhood window position; and in response to a window position switching operation of the user, switching the currently played first view angle video image into the second view angle video image of the target neighborhood view angle selected by the user. According to the method provided by the invention, the switching delay can be reduced, so that the free viewpoint video playing effect is smoother.

Description

Technical Field

[0001] The present invention relates to the field of video playback technology, and in particular to a free viewpoint video playback method, device, storage medium and electronic equipment. Background Art

[0002] With the development of cutting-edge technologies such as visual computing technology and modern communication technology, the acquisition, rendering, encoding compression, distribution and transmission of free viewpoint videos have become increasingly mature. However, as an immersive audio-visual cutting-edge application that can provide users with free perspective selection and meet individual viewing needs, it has always faced problems in terminal playback, such as poor presentation clarity and long latency, which seriously affect the user experience. Summary of the invention

[0003] In view of this, the present invention provides a free viewpoint video playback method, device, storage medium and electronic device, the main purpose of which is to solve the current problems of poor clarity and long delay in free viewpoint video terminal playback.

[0004] To solve the above problems, the present application provides a free viewpoint video playback method, comprising:

[0005] Based on the preset window size, the eye tracking method is used to locate the user's sight area to obtain the effective window position;

[0006] Calculation is performed based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0007] Performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions;

[0008] In response to the user's window position switching operation, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighborhood perspective selected by the user.

[0009] Optionally, the eye tracking method is used to locate the user's sight area to obtain the effective window position, specifically including:

[0010] Acquire an eyeball image of the user;

[0011] Extracting features from the eyeball image to obtain eyeball features corresponding to the user;

[0012] Performing calculation based on the eyeball features to obtain the sight direction;

[0013] The effective window position is obtained by performing calculation based on the sight line direction and the predetermined screen layout.

[0014] Optionally, performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions specifically includes:

[0015] Decoding the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions;

[0016] Performing image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image;

[0017] Performing image synthesis based on each of the second decoded video images and a second redundant image corresponding to each of the second decoded video images to obtain a second synthesized image;

[0018] Image rendering is performed on the first composite video image and each of the second composite video images respectively to obtain the first perspective video image and each of the second perspective video images.

[0019] Optionally, the decoding process of the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions specifically includes:

[0020] Cropping the full-view video image based on the effective window position to obtain a first initial video image corresponding to the effective window position;

[0021] Decoding the first initial video image using a preset first bit rate to obtain the first decoded video image;

[0022] Cropping the full-view video image based on the positions of the respective neighborhood windows to obtain a second initial video image corresponding to the positions of the respective neighborhood windows;

[0023] Each of the second initial video images is decoded using a preset first bit rate to obtain the second decoded video image corresponding to each of the neighborhood window positions.

[0024] Optionally, the performing image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image specifically includes:

[0025] Aligning a first decoded video image of a preset first code rate and the first redundant image of a preset second code rate to obtain a first aligned image;

[0026] fusing the first image data of the preset first code rate and the second image data of the preset second code rate in the first aligned image to obtain a first fused image;

[0027] Correct the first fused image to obtain the first composite image.

[0028] Optionally, respectively rendering the first composite video image and each of the second composite video images to obtain the first perspective video image and each of the second perspective video images specifically includes:

[0029] Performing format conversion on the first composite video image and each of the second composite video images to obtain first pixel data corresponding to the first composite video image and second pixel data corresponding to each of the second composite video images;

[0030] The first pixel data and each of the second pixel data are resolution-adapted based on a preset terminal resolution and size specification to obtain the first perspective video image and each of the second perspective video images.

[0031] Optionally, after the currently played first-perspective video image is switched to the second-perspective video image of the target neighborhood perspective selected by the user in response to the user's window position switching operation, the method further includes:

[0032] Determine the sight area corresponding to the target neighborhood viewing angle as the current effective window position;

[0033] Based on the current effective window position, each neighboring window position is updated to obtain each current neighboring window position;

[0034] The pre-acquired full-view video image is processed based on the current effective window position and each of the current neighboring window positions to obtain a first current-view video image corresponding to the current effective window position and a second current-view video image corresponding to each of the current neighboring window positions.

[0035] In order to solve the above problems, the present application provides a free viewpoint video playback device, comprising:

[0036] A positioning module, used to locate the user's sight area based on a preset window size by using an eye tracking method to obtain an effective window position;

[0037] A calculation processing module, used for performing calculation processing based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0038] An image processing module, configured to perform image processing on a pre-acquired full-view video image based on the effective window position and each of the neighboring window positions, to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions;

[0039] The image switching module is used to switch the first-perspective video image currently being played to the second-perspective video image of the target neighborhood perspective selected by the user in response to the user's window position switching operation.

[0040] In order to solve the above problem, the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned free viewpoint video playback method are implemented.

[0041] To solve the above problems, the present application provides an electronic device, which comprises at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned free viewpoint video playback method when executing the computer program on the memory.

[0042] Beneficial effects of the present application: The present application uses an eye tracking method to locate the user's sight area based on a preset window size to obtain an effective window position; based on the effective window position and the preset rotation angle value, calculation and processing are performed to obtain the positions of each neighboring window; based on the effective window position and each of the neighboring window positions, image processing is performed on the pre-acquired full-view video image to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions; the present application pre-transmits the full-space perspective video to the terminal in real time using the push-stream technology, and the player device implemented by the present application calculates the effective window based on the effective visual area of ​​the eyeball or the playback device and presents the window area at a high bit rate, thereby effectively solving the switching delay problem caused by the code stream switching caused by the use of multiple code streams in parallel transmission in current network audio-visual applications. And the delay problem of the terminal returning the window information to the audio-visual server to pull the code stream and then push the stream to play. In response to the user's window position switching operation, the currently played first-view video image is switched to the second-view video image of the target neighborhood perspective selected by the user. This application preloads the first-person perspective video and various second-person perspective videos, and the playback effect is smoother after the window position is switched.

[0043] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 A schematic diagram showing a flow chart of a free viewpoint video playback method provided in an embodiment of the present application is shown;

[0046] Figure 2 A schematic diagram showing a flow chart of a free viewpoint video playback method provided by another embodiment of the present application;

[0047] Figure 3 A structural block diagram of a free viewpoint video playback device provided in another embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0049] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of the embodiments. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0050] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0051] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0052] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0053] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0054] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0055] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0056] The present application embodiment provides a free viewpoint video playback method, such as Figure 1 As shown, including:

[0057] Step S101: positioning the user's sight area using an eye tracking method based on a preset window size to obtain a valid window position;

[0058] In the specific implementation of this step, the preset window size includes the horizontal angle of the window and the vertical angle of the window. The horizontal angle of the window can be set to 110°, and the vertical angle of the window can be set to 60°. In practical applications, the horizontal angle of the window and the vertical angle of the window can be set according to actual needs. Specifically, the eyeball image of the user is obtained; the feature of the eyeball image is extracted to obtain the eyeball feature corresponding to the user; the eyeball feature is calculated and processed to obtain the line of sight direction; the line of sight direction and the predetermined screen layout are calculated and processed to obtain the effective window position.

[0059] Step S102: performing calculation based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0060] In the specific implementation process of this step, the effective window position is used as a reference, and the positions of each neighboring window are determined based on the preset rotation angle value and each rotation direction. The preset rotation angle value can be an angle such as 5°, and the preset rotation angle value can be set according to actual needs. According to the effective window position and the preset rotation angle value, the adjacent window, i.e., the neighboring window, can be positioned to obtain the positions of each neighboring window.

[0061] Step S103: performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions;

[0062] In the specific implementation process of this step, the pre-acquired full-view video image is decoded based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions; specifically, the full-view video image is cropped based on the effective window position to obtain a first initial video image corresponding to the effective window position; the first initial video image is decoded at a preset first bit rate to obtain the first decoded video image; the full-view video image is cropped based on each of the neighboring window positions to obtain a second initial video image corresponding to each of the neighboring window positions; and the second initial video images are decoded at a preset first bit rate to obtain the second decoded video image corresponding to each of the neighboring window positions. Perform image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image; perform image synthesis based on each of the second decoded video images and a second redundant image corresponding to each of the second decoded video images to obtain a second synthesized image; perform image rendering on the first synthesized video image and each of the second synthesized video images respectively to obtain the first-perspective video image and each of the second-perspective video images.

[0063] Step S104: In response to the user's window position switching operation, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighborhood perspective selected by the user.

[0064] In the specific implementation of this step, the types of switching operations include remote control switching, manual sliding switching, and intelligent perception of eye or head movement switching. Specifically, when the user switches the current window position to the neighboring window position adjacent to the current window position, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighboring perspective selected by the user.

[0065] The present application uses an eye tracking method to locate the user's visual area based on a preset window size to obtain an effective window position; based on the effective window position and a preset rotation angle value, a calculation is performed to obtain each neighborhood window position; based on the effective window position and each neighborhood window position, a pre-acquired full-view video image is processed to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each neighborhood window position; the present application pre-transmits the full-space perspective video to the terminal in real time using the push-stream technology, and the player device implemented by the present application calculates the effective window based on the effective visual area of ​​the eyeball or the playback device and presents the window area at a high bit rate, thereby effectively solving the switching delay problem caused by the code stream switching caused by the use of multiple code streams in parallel transmission in current network audio-visual applications. And the delay problem of the terminal returning the window information to the audio-visual server to pull the code stream and then push the stream to play. In response to the user's window position switching operation, the currently played first-view video image is switched to the second-view video image of the target neighborhood perspective selected by the user. This application preloads the first-person perspective video and various second-person perspective videos, and the playback effect is smoother after the window position is switched.

[0066] Another embodiment of the present application provides another free viewpoint video playback method, such as Figure 2 As shown, including:

[0067] Step S201: positioning the user's sight area using an eye tracking method based on a preset window size to obtain a valid window position;

[0068] In the specific implementation of this step, the preset window size includes the horizontal angle of the window and the vertical angle of the window. The horizontal angle of the window can be set to 110°, and the vertical angle of the window can be set to 60°. In practical applications, the horizontal angle of the window and the vertical angle of the window can be set according to actual needs. Specifically, the eyeball image of the user is obtained; the feature of the eyeball image is extracted to obtain the eyeball feature corresponding to the user; the eyeball feature is calculated and processed to obtain the line of sight direction; the line of sight direction and the predetermined screen layout are calculated and processed to obtain the effective window position.

[0069] Step S202: performing calculation based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0070] In the specific implementation process of this step, the effective window position is used as a reference, and the positions of each neighboring window are determined based on the preset rotation angle value and each rotation direction. The preset rotation angle value can be an angle such as 5°, and the preset rotation angle value can be set according to actual needs. According to the effective window position and the preset rotation angle value, the adjacent window, i.e., the neighboring window, can be positioned to obtain the positions of each neighboring window.

[0071] Step S203: decoding the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions;

[0072] In the specific implementation process of this step, the full-view video image is cropped based on the effective window position to obtain a first initial video image corresponding to the effective window position; the first initial video image is decoded and processed using a preset first bit rate to obtain the first decoded video image; specifically, the full-view video image corresponding to the effective window is cropped based on the preset window size to obtain a first initial video image corresponding to the effective window of the preset window size. The cropping operation removes the portion of the video data that is not within the effective window range, and the scaling operation using the preset window size ensures that the video data is displayed in an appropriate proportion within the effective window. Then, the first initial video image is decoded and processed using a preset first bit rate to obtain the first decoded video image; the bit rate of a video stream refers to the data flow used by a video file in a unit of time, which is usually measured in bits per second (bps). Bit rate is an important factor affecting video quality and file size. Generally speaking, a higher bit rate can provide higher quality video images, but it also leads to a larger file size and higher bandwidth requirements. The preset first bit rate can be a bit rate of 20Mbps or more. In specific applications, the preset first bit rate can be set according to actual needs. The decoded video data has the characteristics of high resolution, high color fidelity, etc., ensuring the user's viewing experience. The full-view video image is cropped based on each of the neighborhood window positions to obtain a second initial video image corresponding to each of the neighborhood window positions; each of the second initial video images is decoded using the preset first bit rate to obtain the second decoded video image corresponding to each of the neighborhood window positions.

[0073] Step S204: performing image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image;

[0074] In the specific implementation process of this step, the first decoded video image of the preset first code rate and the first redundant image of the preset second code rate are aligned to obtain the first aligned image; specifically, the first redundant image except the first decoded video image is first decoded based on the preset second code rate, the second code rate is a low code rate, and can be configured to be decoded at a code rate below 1Mbps. For the video code stream in the first redundant image area, a lower code rate decoding method is used for processing, which helps to save bandwidth and computing resources, while ensuring the integrity of the video content. Specifically, the high-quality content of the first decoded video image and the low-code rate content of the first redundant image are accurately aligned to ensure that there will be no misalignment or ghosting when they are synthesized; the first decoded video image and the first redundant image can be aligned using an image registration algorithm or a motion estimation algorithm. The first image data of the preset first code rate and the second image data of the preset second code rate in the first aligned image are fused to obtain a first fused image; an image fusion algorithm is used to seamlessly fuse the high-quality content of the first decoded video image and the low-code rate content of the first redundant image to ensure that no obvious boundaries or artifacts appear in the transition area; the first fused image is corrected to obtain the first synthesized image. Specifically, the color and brightness of the first fused image are adjusted to ensure the consistency of the overall visual effect.

[0075] Step S205: performing image synthesis based on each of the second decoded video images and a second redundant image corresponding to each of the second decoded video images to obtain a second synthesized image;

[0076] In the specific implementation process of this step, image synthesis is performed for each second decoded video image and the second redundant image corresponding to each second decoded video image; specifically, the second decoded video image of the preset first code rate and the second redundant image of the preset second code rate are aligned to obtain a second aligned image; specifically, the second redundant image other than the second decoded video image is first decoded based on the preset second code rate, the second code rate is a low code rate, and can be configured to be decoded at a code rate below 1Mbps, and the video code stream of the second redundant image area is processed by a decoding method with a lower code rate, which helps to save bandwidth and computing resources, while ensuring the integrity of the video content. Specifically, the high-quality content of the second decoded video image and the low-code rate content of the second redundant image are accurately aligned to ensure that they will not be misaligned or ghosted when synthesized; the second decoded video image and the second redundant image can be aligned using an image registration algorithm or a motion estimation algorithm. The third image data of the preset first bit rate and the fourth image data of the preset second bit rate in the second aligned image are fused to obtain a second fused image; an image fusion algorithm is used to seamlessly fuse the high-quality content of the second decoded video image and the low-bit rate content of the second redundant image to ensure that no obvious boundaries or artifacts appear in the transition area; the second fused image is corrected to obtain the second synthesized image. Specifically, the second fused image is adjusted in color and brightness to ensure the consistency of the overall visual effect.

[0077] Step S206: performing image rendering on the first composite video image and each of the second composite video images respectively to obtain a first-view video image and each of the second-view video images;

[0078] In the specific implementation process of this step, the first composite video image and each second composite video image are converted into a format to obtain first pixel data corresponding to the first composite video image and second pixel data corresponding to each second composite video image; the first pixel data and each second pixel data are adapted to the resolution based on the resolution and size specifications of the preset terminal to obtain the first perspective video image and each second perspective video image. Rendering the composited image is to convert the image data into a format that can be displayed by the display device and add necessary visual effects and interactive functions.

[0079] Step S207: in response to the user's window position switching operation, switching the currently played first-perspective video image to the second-perspective video image of the target neighborhood perspective selected by the user;

[0080] In the specific implementation of this step, the types of switching operations include remote control switching, manual sliding switching, and intelligent perception of eye or head movement switching. Specifically, when the user switches the current window position to the neighboring window position adjacent to the current window position, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighboring perspective selected by the user.

[0081] Step S208: determining the sight area corresponding to the target neighborhood viewing angle as the current effective window position;

[0082] In the specific implementation of this step, during the user switching process, the effective window position is updated in real time, the window after the user switching operation is updated to the effective window position, and the neighboring window position corresponding to the updated effective window position is updated according to the updated effective window position.

[0083] Step S209: updating each neighboring window position based on the current valid window position to obtain each current neighboring window position;

[0084] In the specific implementation process of this step, calculation and processing are performed based on the current effective window position and the preset rotation angle value to obtain the positions of each neighboring window corresponding to the current effective window.

[0085] Step S210: Based on the current effective window position and each of the current neighboring window positions, the pre-acquired full-view video image is processed to obtain a first current-view video image corresponding to the current effective window position and a second current-view video image corresponding to each of the current neighboring window positions.

[0086] In the specific implementation process of this step, the pre-acquired full-view video image is decoded based on the current effective window position and each of the current neighboring window positions to obtain a first current decoded video image corresponding to the current effective window position and a second current decoded video image corresponding to each of the current neighboring window positions; image synthesis is performed based on the first current decoded video image and the first current redundant image corresponding to the first current decoded video image to obtain a first current synthesized image; image synthesis is performed based on each of the second current decoded video images and the second current redundant image corresponding to each of the second current decoded video images to obtain a second current synthesized image; and image rendering is performed on the first current synthesized video image and each of the second current synthesized video images, respectively, to obtain the first current view video image and each of the second current view video images.

[0087] The present application uses an eye tracking method to locate the user's line of sight area based on a preset window size to obtain an effective window position; performs calculation processing based on the effective window position and a preset rotation angle value to obtain the positions of each neighboring window; decodes a pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions; performs high-bitrate decoding on the video at the effective window position and low-bitrate decoding on the video in the redundant area, thereby achieving efficient utilization of terminal hardware performance and effectively solving the problem of low terminal performance or waste in decoding and playback of the full-aggregate code stream. Based on the first decoded video image and the first redundant image corresponding to the first decoded video image, an image is synthesized to obtain a first synthesized image; based on each of the second decoded video images and the second redundant image corresponding to each of the second decoded video images, an image is synthesized to obtain a second synthesized image; the first synthesized video image and each of the second synthesized video images are respectively rendered to obtain a first-view video image and each second-view video image; the present application pre-transmits the full-space view video to the terminal in real time using the push-stream technology, and the player device implemented by the present application calculates the effective window based on the effective visual area of ​​the eyeball or the playback device and presents the window area at a high bit rate, so it can effectively solve the switching delay problem caused by the code stream switching caused by the use of multiple code streams in parallel transmission in current network audio-visual applications. And the delay problem of the terminal returning the window information to the audio-visual server to pull the code stream and then push the stream to play. In response to the user's window position switching operation, the first view video image currently being played is switched to the second view video image of the target neighborhood view selected by the user. The present application pre-loads the first view video and each second view video, and the playback effect is smoother after the window position is switched.

[0088] Another embodiment of the present application provides a free viewpoint video playback device, such as Figure 3 As shown, including:

[0089] Positioning module 1, used to locate the user's sight area based on a preset window size by using an eye tracking method to obtain an effective window position;

[0090] A calculation processing module 2, used for performing calculation processing based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0091] An image processing module 3 is used to perform image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions;

[0092] The image switching module 4 is used to switch the first-perspective video image currently being played to the second-perspective video image of the target neighborhood perspective selected by the user in response to the user's window position switching operation.

[0093] In the specific implementation process, the positioning module 1 is specifically used to: obtain the eye image of the user; perform feature extraction on the eye image to obtain eye features corresponding to the user; perform calculations based on the eye features to obtain the line of sight direction; perform calculations based on the line of sight direction and a predetermined screen layout to obtain the effective window position.

[0094] In the specific implementation process, the calculation and processing module 2 is specifically used to: decode the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions; perform image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image; perform image synthesis based on each of the second decoded video images and a second redundant image corresponding to each of the second decoded video images to obtain a second synthesized image; perform image rendering on the first synthesized video image and each of the second synthesized video images respectively to obtain the first-view video image and each of the second-view video images.

[0095] In the specific implementation process, the calculation and processing module 2 is also used to: crop the full-view video image based on the effective window position to obtain a first initial video image corresponding to the effective window position; decode the first initial video image using a preset first bit rate to obtain the first decoded video image; crop the full-view video image based on each of the neighborhood window positions to obtain a second initial video image corresponding to each of the neighborhood window positions; decode each of the second initial video images using a preset first bit rate to obtain a second decoded video image corresponding to each of the neighborhood window positions.

[0096] In the specific implementation process, the calculation and processing module 2 is also used to: align the first decoded video image of the preset first code rate and the first redundant image of the preset second code rate to obtain a first aligned image; fuse the first image data of the preset first code rate and the second image data of the preset second code rate in the first aligned image to obtain a first fused image; correct the first fused image to obtain the first composite image.

[0097] During the specific implementation process, the computing and processing module 2 is also used to: perform format conversion on the first composite video image and each of the second composite video images to obtain first pixel data corresponding to the first composite video image and second pixel data corresponding to each of the second composite video images; perform resolution adaptation on the first pixel data and each of the second pixel data based on the resolution and size specifications of the preset terminal to obtain the first perspective video image and each of the second perspective video images.

[0098] During the specific implementation process, the device also includes: an update module, which is specifically used to: determine the line of sight area corresponding to the target neighborhood perspective as the current effective window position; update each neighborhood window position based on the current effective window position to obtain each current neighborhood window position; perform image processing on the pre-acquired full-perspective video image based on the current effective window position and each current neighborhood window position to obtain a first current perspective video image corresponding to the current effective window position and a second current perspective video image corresponding to each current neighborhood window position.

[0099] The present application uses an eye tracking method to locate the user's visual area based on a preset window size to obtain an effective window position; based on the effective window position and a preset rotation angle value, a calculation is performed to obtain each neighborhood window position; based on the effective window position and each neighborhood window position, a pre-acquired full-view video image is processed to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each neighborhood window position; the present application pre-transmits the full-space perspective video to the terminal in real time using the push-stream technology, and the player device implemented by the present application calculates the effective window based on the effective visual area of ​​the eyeball or the playback device and presents the window area at a high bit rate, thereby effectively solving the switching delay problem caused by the code stream switching caused by the use of multiple code streams in parallel transmission in current network audio-visual applications. And the delay problem of the terminal returning the window information to the audio-visual server to pull the code stream and then push the stream to play. In response to the user's window position switching operation, the currently played first-view video image is switched to the second-view video image of the target neighborhood perspective selected by the user. This application preloads the first-person perspective video and various second-person perspective videos, and the playback effect is smoother after the window position is switched.

[0100] Another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0101] Step 1: Based on the preset window size, the user's sight area is located using an eye tracking method to obtain an effective window position;

[0102] Step 2: Calculate and process based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0103] Step 3: performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions;

[0104] Step 4: In response to the user's window position switching operation, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighborhood perspective selected by the user.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0107] The specific implementation process of the above method steps can refer to the embodiments of any free viewpoint video playback method mentioned above, and this embodiment will not be repeated here.

[0108] The present application uses an eye tracking method to locate the user's visual area based on a preset window size to obtain an effective window position; based on the effective window position and a preset rotation angle value, a calculation is performed to obtain each neighborhood window position; based on the effective window position and each neighborhood window position, a pre-acquired full-view video image is processed to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each neighborhood window position; the present application pre-transmits the full-space perspective video to the terminal in real time using the push-stream technology, and the player device implemented by the present application calculates the effective window based on the effective visual area of ​​the eyeball or the playback device and presents the window area at a high bit rate, thereby effectively solving the switching delay problem caused by the code stream switching caused by the use of multiple code streams in parallel transmission in current network audio-visual applications. And the delay problem of the terminal returning the window information to the audio-visual server to pull the code stream and then push the stream to play. In response to the user's window position switching operation, the currently played first-view video image is switched to the second-view video image of the target neighborhood perspective selected by the user. This application preloads the first-person perspective video and various second-person perspective videos, and the playback effect is smoother after the window position is switched.

[0109] Another embodiment of the present application provides an electronic device, which may be a server, and the electronic device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external client via a network connection. When the electronic device program is executed by the processor, it implements functions or steps on the server side of a free viewpoint video playback method.

[0110] In one embodiment, an electronic device is provided, which may be a client. The electronic device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external server via a network connection. When the electronic device program is executed by the processor, it implements the functions or steps on the client side of a free viewpoint video playback method.

[0111] Another embodiment of the present application provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the following method steps when executing the computer program in the memory:

[0112] Step 1: Based on the preset window size, the user's sight area is located using an eye tracking method to obtain an effective window position;

[0113] Step 2: Calculate and process based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window;

[0114] Step 3: performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions;

[0115] Step 4: In response to the user's window position switching operation, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighborhood perspective selected by the user.

[0116] The specific implementation process of the above method steps can refer to the embodiments of any free viewpoint video playback method mentioned above, and this embodiment will not be repeated here.

[0117] The present application uses an eye tracking method to locate the user's visual area based on a preset window size to obtain an effective window position; based on the effective window position and a preset rotation angle value, a calculation is performed to obtain each neighborhood window position; based on the effective window position and each neighborhood window position, a pre-acquired full-view video image is processed to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each neighborhood window position; the present application pre-transmits the full-space perspective video to the terminal in real time using the push-stream technology, and the player device implemented by the present application calculates the effective window based on the effective visual area of ​​the eyeball or the playback device and presents the window area at a high bit rate, thereby effectively solving the switching delay problem caused by the code stream switching caused by the use of multiple code streams in parallel transmission in current network audio-visual applications. And the delay problem of the terminal returning the window information to the audio-visual server to pull the code stream and then push the stream to play. In response to the user's window position switching operation, the currently played first-view video image is switched to the second-view video image of the target neighborhood perspective selected by the user. This application preloads the first-person perspective video and various second-person perspective videos, and the playback effect is smoother after the window position is switched.

[0118] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A free viewpoint video playback method, characterized in that: include: Based on the preset window size, the eye tracking method is used to locate the user's sight area to obtain the effective window position; Calculation is performed based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window; Performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions; In response to the user's window position switching operation, the first-perspective video image currently being played is switched to the second-perspective video image of the target neighborhood perspective selected by the user.

2. The method according to claim 1, characterized in that The eye tracking method is used to locate the user's sight area to obtain the effective window position, specifically including: Acquire an eye image of the user; Extracting features from the eyeball image to obtain eyeball features corresponding to the user; Performing calculation based on the eyeball features to obtain the sight direction; The effective window position is obtained by performing calculation based on the sight line direction and the predetermined screen layout.

3. The method according to claim 1, characterized in that The performing image processing on the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions specifically includes: Decoding the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions; Performing image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image; Performing image synthesis based on each of the second decoded video images and a second redundant image corresponding to each of the second decoded video images to obtain a second synthesized image; Image rendering is performed on the first composite video image and each of the second composite video images respectively to obtain the first perspective video image and each of the second perspective video images.

4. The method according to claim 3, characterized in that The decoding process of the pre-acquired full-view video image based on the effective window position and each of the neighboring window positions to obtain a first decoded video image corresponding to the effective window position and a second decoded video image corresponding to each of the neighboring window positions specifically includes: Cropping the full-view video image based on the effective window position to obtain a first initial video image corresponding to the effective window position; Decoding the first initial video image using a preset first bit rate to obtain the first decoded video image; Cropping the full-view video image based on the positions of the respective neighborhood windows to obtain a second initial video image corresponding to the positions of the respective neighborhood windows; Each of the second initial video images is decoded using a preset first bit rate to obtain the second decoded video image corresponding to each of the neighborhood window positions.

5. The method according to claim 3, characterized in that The performing image synthesis based on the first decoded video image and a first redundant image corresponding to the first decoded video image to obtain a first synthesized image specifically includes: Aligning a first decoded video image of a preset first code rate and the first redundant image of a preset second code rate to obtain a first aligned image; fusing the first image data of the preset first code rate and the second image data of the preset second code rate in the first aligned image to obtain a first fused image; Correct the first fused image to obtain the first composite image.

6. The method according to claim 3, characterized in that The performing image rendering on the first composite video image and each of the second composite video images to obtain the first perspective video image and each of the second perspective video images specifically includes: Performing format conversion on the first composite video image and each of the second composite video images to obtain first pixel data corresponding to the first composite video image and second pixel data corresponding to each of the second composite video images; The first pixel data and each of the second pixel data are resolution-adapted based on a preset terminal resolution and size specification to obtain the first perspective video image and each of the second perspective video images.

7. The method according to claim 1, characterized in that After the currently played first-perspective video image is switched to the second-perspective video image of the target neighborhood perspective selected by the user in response to the user's window position switching operation, the method further includes: Determine the sight area corresponding to the target neighborhood viewing angle as the current effective window position; Based on the current effective window position, each neighboring window position is updated to obtain each current neighboring window position; The pre-acquired full-view video image is processed based on the current effective window position and each of the current neighboring window positions to obtain a first current-view video image corresponding to the current effective window position and a second current-view video image corresponding to each of the current neighboring window positions.

8. A free viewpoint video playback device, characterized in that: include: A positioning module, used to locate the user's sight area based on a preset window size by using an eye tracking method to obtain an effective window position; A calculation processing module, used for performing calculation processing based on the effective window position and the preset rotation angle value to obtain the position of each neighborhood window; An image processing module, configured to perform image processing on a pre-acquired full-view video image based on the effective window position and each of the neighboring window positions, to obtain a first-view video image corresponding to the effective window position and a second-view video image corresponding to each of the neighboring window positions; The image switching module is used to switch the first-perspective video image currently being played to the second-perspective video image of the target neighborhood perspective selected by the user in response to the user's window position switching operation.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the free viewpoint video playback method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method at least comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the free viewpoint video playback method according to any one of claims 1 to 7 when executing the computer program on the memory.

Images