Viewpoint switching method, device, and system for free viewpoint video

Abstract

A method, an apparatus, and a system for switching a viewpoint of a free viewpoint video are provided, which are related to the field of Internet technology. According to the method, after detecting that a user operates a first danmaku comment, a terminal can automatically switch a playback viewpoint of the free viewpoint video to a viewpoint corresponding to the first danmaku comment to improve a viewing experience of the free viewpoint video. The method includes a step of a terminal displaying a first interface, where the first interface has a video image of the first free viewpoint video at a first viewpoint and a first danmaku comment, and the first danmaku comment corresponds to a second viewpoint of the first free viewpoint video; and a step of displaying a second interface, where the second interface has a video image of the first free viewpoint video at the second viewpoint, when the terminal detects a first operation performed by a user on the first danmaku comment.

Description

[Technical Field]

[0001] The present application relates to the field of Internet technology, and in particular to a viewpoint switching method, apparatus, and system for free viewpoint video. [Background technology]

[0002] Free viewpoint video (FVV) is a method in which a target object is captured by multiple surrounding cameras to generate a video stream. ,So is encoded into a file like It will be , a video file in a special format. The generated free-viewpoint video is then uploaded to a media server for storage and then distributed to a terminal client through a content distribution network (CDN). When watching the free-viewpoint video using a terminal client, a user can freely switch the viewpoint by dragging their finger. Compared with traditional video, free-viewpoint video allows users to freely select the viewpoint, thus overcoming the drawback of traditional video in which the viewpoint is determined by a director. Free-viewpoint video provides users with an immersive and interactive playback experience.

[0003] However, when the target object has a vast shooting scene, the user has a limited viewing range on the client, and therefore has blind areas, in which the user may miss some highlights that can be viewed at a viewing range other than the current viewing range. In this case, it is necessary to improve the viewing experience for the user. Summary of the Invention

[0004] According to the viewpoint switching method, device and system for free viewpoint video provided in the present application, after detecting that a user has operated a first barrage comment, the terminal automatically switches the playback viewpoint of the free viewpoint video to a viewpoint corresponding to the first barrage comment, thereby improving the viewing experience of the free viewpoint video.

[0005] To achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect, there is provided a viewpoint switching method for free viewpoint video, the method comprising: displaying, by a terminal, a first interface, where the first interface has a video image of the first free viewpoint video at a first viewpoint and a first danmaku comment, the first danmaku comment corresponding to a second viewpoint of the first free viewpoint video; and, when the terminal detects a first operation performed by a user on the first danmaku comment, displaying a second interface, where the second interface has the video image of the first free viewpoint video at the second viewpoint.

[0007] It can be understood that when watching the first free viewpoint video, the user can quickly switch to the viewpoint corresponding to the first danmaku comment by operating the first danmaku comment, thereby allowing the user to easily view highlights of the viewpoint. It can be understood that even when the first free viewpoint video has a vast filmed scene with a limited viewpoint range on the terminal and even blind areas, the user can still grasp the viewpoint having highlights from the content of the danmaku comments displayed on the video image. A highlight may be a video clip that is highly praised by most users, or a video clip that viewers are interested in, etc. By operating a specific danmaku comment, the user can quickly switch to the viewpoint corresponding to the specific danmaku comment and view the highlights of the viewpoint, thereby improving the user's viewing experience.

[0008] In a possible implementation, displaying the first interface by the terminal includes displaying the first interface at a first playback instant, and displaying the second interface when the terminal detects a first operation performed by a user on the first danmaku comment includes displaying the second interface at a second playback instant when the terminal detects a first operation performed by a user on the first danmaku comment, where the second playback instant is related to the first playback instant.

[0009] In other words, the first interface has a video image of a first playback instant of a first free viewpoint video at a first viewpoint, and the second interface has a video image of a second playback instant of the first free viewpoint video at a second viewpoint.

[0010] In possible implementations, the second playback instant being related to the first playback instant includes the second playback instant being equal to the first playback instant, or the second playback instant being equal to the first playback instant minus a first preset time offset value; or the second playback instant being equal to the occurrence instant of the first danmaku comment, or the second playback instant being equal to the occurrence instant of the first danmaku comment minus a second preset time offset value.

[0011] That is, in some examples, after a user performs a first operation on a first danmaku comment in the first interface, the playback viewpoint is switched to a viewpoint corresponding to the first danmaku comment, and the playback instant is switched to the current playback instant of the first interface when the first operation was performed, i.e., the first playback instant. Alternatively, the playback instant is switched to an instant earlier than the first playback instant, i.e., the second playback instant is equal to the first playback instant minus a first preset time offset value. In some other examples, after a user performs a first operation on a first danmaku comment in the first interface, the playback viewpoint is switched to a viewpoint corresponding to the first danmaku comment, and the playback instant is switched to the occurrence instant of the first danmaku comment or to a playback instant earlier than the occurrence instant of the first danmaku comment, i.e., the occurrence instant of the first danmaku comment minus a second preset time offset value.

[0012] In a possible implementation, the first danmaku comment further includes a second viewpoint, i.e., a viewpoint corresponding to the first danmaku comment is further displayed within the first danmaku comment.

[0013] In a possible implementation form, displaying the second interface at the second playback instant when the terminal detects a first operation performed by the user on the first danmaku comment includes: sending a switching request to the media server when the terminal detects a first operation performed by the user on the first danmaku comment; the terminal receiving a first video file returned by the media server, where the first video file is a video file corresponding to the second playback instant of the first free viewpoint video at the second perspective; and the terminal displaying a video image of the second playback instant of the first free viewpoint video at the second perspective based on the first video file.

[0014] In a possible implementation, before the terminal displays the first interface at the first playback instant, the method further includes the terminal requesting danmaku comment data corresponding to the first playback instant of the first free viewpoint video from a danmaku comment server, wherein the danmaku comment data corresponding to the first playback instant has data of a first danmaku comment, and the data of the first danmaku comment includes a second viewpoint.

[0015] In a possible implementation, after the terminal displays the second interface at the second playback instant, the method further includes the terminal receiving a second operation to send a second danmaku comment, and the terminal sending data of the second danmaku comment to the danmaku comment server, where the data of the second danmaku comment includes content of the second danmaku comment and a third viewpoint, and the third viewpoint is a viewpoint corresponding to the first free viewpoint video during the sending of the second danmaku comment.

[0016] That is, when the terminal receives a second barrage comment input by a user for a video image at a third viewpoint, an association is established between the second barrage comment and the third viewpoint, and the content of the second barrage comment and information about the third viewpoint are transmitted to the barrage comment server.

[0017] In a possible implementation, the data of the second danmaku comment further includes an occurrence instant of the second danmaku comment, wherein the occurrence instant of the second danmaku comment is equal to the second playback instant, or the occurrence instant of the second danmaku comment is equal to the second playback instant minus a third preset time offset value.

[0018] In a possible implementation, the second interface further includes a first danmaku comment, i.e., when a user performs a first operation on the first danmaku comment, the second playback instant after the switch is still within a duration that continuously displays the first danmaku comment.

[0019] In a possible implementation form, after the terminal displays the second interface at the second playback instant, the method further includes the terminal displaying a third interface at a third playback instant, where the third playback instant is after the display time of the first danmaku comment, and the third interface has a video image of the first free viewpoint video at the second viewpoint, and the third interface does not have the first danmaku comment.

[0020] In a possible implementation form, the method further includes the terminal displaying a fourth interface at a fourth playback instant, where the fourth interface has a video image of the first free viewpoint video at the first viewpoint and a recommended viewpoint, and the recommended viewpoint includes the fourth viewpoint; and, when the terminal detects a third operation by the user to select the fourth viewpoint, displaying a video image of a fifth playback instant of the first free viewpoint video at the fourth viewpoint, where the fifth playback instant is related to the fourth playback instant.

[0021] In a possible implementation, the fifth playback instant being related to the fourth playback instant includes the fifth playback instant being equal to the fourth playback instant, or the fifth playback instant being equal to the fourth playback instant minus a fourth preset time offset value.

[0022] In a possible implementation form, before the step in which the terminal displays the fourth interface at the fourth playback instant, the method further includes the terminal requesting a second video file corresponding to the fourth playback instant of the first free viewpoint video at the first viewpoint from the media server, and requesting a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from the danmaku comment server, wherein the recommended viewpoint corresponding to the fourth playback instant is determined by the danmaku comment server based on the number of danmaku comments corresponding to the fourth playback instant of the first free viewpoint video.

[0023] In other words, the user can quickly switch the playback viewpoint of the first free viewpoint video by using the recommended viewpoint displayed on the terminal, thereby improving the user's video viewing experience.

[0024] In some embodiments, the barrage comment server may alternatively recommend viewpoints in which highlights are located to a user for selection based on the collected barrage comment data. For example, in the process of playing a free viewpoint video, the number of barrage comments corresponding to a video clip may reflect the popularity and highlight level of the video clip. If the number of barrage comments corresponding to a video clip is greater than a preset threshold, the video clip may be deemed a highlight. The user can quickly switch to a desired viewpoint by operating the recommended viewpoint.

[0025] According to a second aspect, a method for switching danmaku comments is provided and applied to a danmaku comment server, the method comprising: receiving data of a first danmaku comment transmitted by a first terminal, where the data of the first danmaku comment includes an identifier of a first free viewpoint video, a first viewpoint, and content of the first danmaku comment, and the first danmaku comment is transmitted by the first terminal when the first terminal plays back a video image of the first free viewpoint video at the first viewpoint; and storing the data of the first danmaku comment.

[0026] In a possible implementation, the data of the first barrage comment further includes the moment of occurrence of the first barrage comment.

[0027] In a possible implementation, the method further includes receiving a first request sent by the second terminal, where the first request is used to request danmaku comment data for a first playback instant of the first free viewpoint video, the first playback instant being less than or equal to an instant of occurrence of the first danmaku comment; and returning the data of the first danmaku comment to the second terminal in response to receiving the first request.

[0028] In a possible implementation form, after receiving the first request sent by the second terminal, the method further includes sending to the second terminal a recommended viewpoint list corresponding to a first playback instant of the first free viewpoint video.

[0029] In a possible implementation form, before sending the recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video to the second terminal, the method further includes separately collecting statistical information regarding the number of barrage comments of different viewpoints corresponding to the first playback moment of the first free viewpoint video; and determining the recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video based on the number of barrage comments of different viewpoints corresponding to the first playback moment of the first free viewpoint video.

[0030] In a possible implementation form, determining a recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video based on the number of danmaku comments of different viewpoints corresponding to the first playback moment of the first free viewpoint video further includes determining that the recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video includes the second viewpoint if it is determined that the number of danmaku comments of a second viewpoint corresponding to the first playback moment of the first free viewpoint video is greater than a predetermined threshold.

[0031] In a possible implementation, there are one or more second viewpoints.

[0032] According to a third aspect, there is provided a terminal comprising a processor, a memory, and a touchscreen, the memory and the touchscreen being coupled to the processor, the memory being configured to store computer program code, the computer program code including computer instructions, such that when the processor reads the computer instructions in the memory, the terminal is enabled to perform a method according to the first aspect and any one of the possible implementation forms of the first aspect.

[0033] According to a fourth aspect, there is provided a barrage comment server comprising a processor, a memory, and a communication interface, the memory and the communication interface being coupled to the processor, the memory being configured to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions in the memory, the barrage comment server is enabled to perform a method according to the second aspect and any one of the possible implementation forms of the second aspect.

[0034] According to a fifth aspect, there is provided a computer readable storage medium comprising computer instructions which, when executed on a terminal, enable the terminal to perform a method according to the first aspect or the second aspect and any one of the possible implementation forms of the first aspect or the second aspect.

[0035] According to a sixth aspect, there is provided a computer program product which, when executed on a computer, enables a terminal to perform a method according to the first aspect or the second aspect and any one of the possible implementations of the first aspect or the second aspect.

[0036] For technical effects that can be achieved by the terminal, the barrage comment server, the computer-readable storage medium, and the computer program product provided in the third to sixth aspects, please refer to the description of the technical effects in the first aspect and possible implementation forms of the first aspect. Details will not be described again here. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a schematic diagram of the structure of a free viewpoint video playback system according to an embodiment of the present application;

[0038] [Figure 2] 1 is a schematic diagram of the structure of a terminal according to an embodiment of the present application;

[0039] [Figure 3] FIG. 1 is a schematic diagram of the structure of a free viewpoint video client / application according to an embodiment of the present application.

[0040] [Figure 4] 1 is a schematic diagram of the structure of a server according to an embodiment of the present application;

[0041] [Figure 5A] 1 is a schematic flowchart of a method for sending barrage comments for a free viewpoint video according to an embodiment of the present application;

[0042] [Figure 5B] 1 is a schematic flowchart of a viewpoint switching method for free viewpoint video according to an embodiment of the present application;

[0043] [Figure 6(1)] 1 is a schematic diagram of several terminal interfaces according to an embodiment of the present application; [Figure 6(2)] 1 is a schematic diagram of several terminal interfaces according to an embodiment of the present application;

[0044] [Figure 7(1)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 7(2)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 7(3)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0045] [Figure 8(1)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 8(2)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0046] [Figure 9(1)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 9(2)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 9(3)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0047] [Figure 10(1)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 10(2)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 10(3)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0048] [Figure 11] 1 is a schematic flowchart of another viewpoint switching method for free viewpoint video according to an embodiment of the present application;

[0049] [Figure 12(1)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 12(2)] 3A-3C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0050] [Figure 13] 10 is a schematic flowchart of yet another viewpoint switching method for free viewpoint video according to an embodiment of the present application;

[0051] [Figure 14] 1 is a schematic diagram of the structure of a chip system according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0052] In the description of the embodiments of the present application, unless otherwise specified, " / " means "or." For example, A / B may represent A or B. In this specification, "and / or" only describes an associative relationship between related entities and indicates that three relationships may exist. For example, A and / or B may represent the following three cases: when only A is present, when both A and B are present, and when only B is present. The terms "first" and "second" are intended for descriptive purposes only and should not be understood as an indication or suggestion of relative importance or an implicit indication of the quantity of the indicated technical features. Thus, a feature qualified by "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. In addition, in the embodiments of the present application, the words "example" or "for example" are used to indicate providing an example, illustration, or explanation. Any embodiment or design scheme described in the embodiments of this application as an "example" or "for example" should not be described as being more preferred or having more advantages than another embodiment or design scheme. Rather, the use of words such as "example" or "for example" is intended to present a relative concept in a concrete manner.

[0053] In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, technical terms in the embodiments of the present application will be explained first.

[0054] A free-viewpoint video is a video file in a special format generated through joint image capture of a target object by multiple cameras surrounding the target object and specific video encoding of the video stream acquired through the joint image capture. The generated free-viewpoint video is then uploaded to a media server for storage and then distributed to a terminal-side client through a content distribution network (CDN). When a user watches the free-viewpoint video on a terminal-side client, the user can manually select a viewing point.

[0055] Camera position is the number of the position where each camera is placed when multiple cameras surrounding the target object are taking images of the target object. Camera position corresponds to one image capture angle, which can be used to capture images of a target object within a certain angle range.

[0056] The viewpoint is also called a viewing point or a playing viewpoint, and is an image capturing angle from which a user views a free viewpoint video. In other words, an image of a target object seen from a certain viewpoint is captured by a camera positioned at the viewpoint. (Camera position) The image may be considered as an image captured when image capture is performed at

[0057] Video playback is a process in which a video file (including free viewpoint video) is decoded, rendered, or otherwise processed, and then a group of images is displayed in chronological order in a display window, and the corresponding audio is played.

[0058] Danmaku comments are comments or annotations that are displayed on video (including free viewpoint video) at designated times or during designated time periods according to rules, bringing an interesting and unexpected experience to video viewing.

[0059] 1 is a schematic diagram of a system architecture according to an embodiment of the present application, which includes a camera array 500, a media processing server 200, a media server 300, a content distribution network (CDN) 400, a terminal 100, and a barrage comment server 600.

[0060] The camera array 500 includes multiple (e.g., 18-36) cameras or other devices with cameras surrounding the target object (object to be photographed / image capture scene). Camera position 0, Camera position 1, ..., and Camera position The camera array 500 performs image capture of a target object from multiple angles separately, and uploads the video streams acquired through image capture from the multiple angles to the media processing server 200.

[0061] The media processing server 200 is configured to transcode the received multi-angle video streams to generate free-viewpoint video, which the media processing server 200 then uploads to the media server 300 for storage.

[0062] For example, processing processes performed by the media processing server 200 on the video streams of multiple angles include, but are not limited to, focus alignment, frame alignment synchronization coding, rotation auxiliary stream generation, and virtual viewpoint synthesis. Focus alignment can be performed, for example, in an image cropping manner to ensure a smooth transition of the image of the captured object during the viewpoint switching process. Frame alignment synchronization coding is used to align video frames acquired through image capture by multiple cameras to the same moment. Virtual viewpoint synthesis is the insertion of a virtual video stream into two real video streams to reduce the number of cameras that need to be used during image capture or to enable a smooth transition of the image of the captured object during the viewpoint switching process. The rotation auxiliary stream is used for rotation display, which helps reduce total traffic usage.

[0063] It should be noted that the processing process performed by the media processing server 200 is merely an example, and the specific technical solutions used in the process of generating free-viewpoint videos based on the video streams uploaded by the camera array 500 are not limited to the embodiments of the present application.

[0064] It is further noted that the media processing server 200 herein may be at least one of a single independent physical server, multiple independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtualization center.

[0065] The media server 300 stores and maintains a large amount of free viewpoint videos. When the media server 300 receives a request for free viewpoint video content sent by the terminal 100, the media server 300 promptly returns the corresponding content of the free viewpoint video to the terminal 100 through the CDN 400.

[0066] It should be noted that the media server 300 in this specification may be at least one of a single independent physical server, multiple independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtualization center.

[0067] The CDN 400 enables users of terminals (e.g., terminal 100) to obtain required content, such as free-viewpoint video, locally by using edge servers deployed in various locations through functional modules of the central platform, such as a load balancing module, a content distribution module, and a scheduling module, thereby reducing network congestion and improving user access response speed and hit rate.

[0068] The terminal 100 can use a free viewpoint video client installed on the terminal 100 to request free viewpoint video content from the media server 300 and play free viewpoint video based on the free viewpoint video content returned by the media server 300 via the CDN 400. If the terminal 100 also enables the danmaku comment function when playing free viewpoint video, the terminal 100 also needs to request danmaku comment information corresponding to the current playback moment or time period from the danmaku comment server 600. In other words, the image played by the terminal 100 includes an image of the free viewpoint video and corresponding danmaku comment information. In addition, when playing free viewpoint video, the terminal 100 can also receive danmaku comment content input by a user and upload the danmaku comment content input by the user and other danmaku comment information to the danmaku comment server 600 for storage.

[0069] For example, the terminal 100 in this embodiment of the present application may be, for example, a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a netbook, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, an in-vehicle device, a smart screen, or an intelligent vehicle, etc. The specific form of the terminal 100 is not particularly limited in this application.

[0070] The danmaku comment server 600 is configured to receive danmaku comment information transmitted by each terminal (including, for example, terminal 100). The danmaku comment information includes, but is not limited to, an identifier of the free viewpoint video, the moment the danmaku comment occurred, the duration of the danmaku comment, the danmaku comment content, and the viewing point during transmission of the danmaku comment. The meanings of the parameters will be explained in detail later and will not be explained here.

[0071] It should be noted that the barrage comment server 600 in this specification may be at least one of a single independent physical server, multiple independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtualization center.

[0072] Original Vow It can be understood that the system structure shown in this embodiment does not constitute a specific limitation on the system architecture to which the technical solutions are applicable. In some other embodiments of the present application, the system architecture to which the technical solutions provided in the embodiments of the present application are applicable may further include more or fewer devices than those shown in this figure, or a combination of some of the devices, or a division of some of the devices, etc.

[0073] 2 is a schematic diagram of an example structure of a terminal 100 according to an embodiment of the present application. The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc.

[0074] Original Vow It may be understood that the structure shown in this embodiment does not constitute a specific limitation on the terminal 100. In some other embodiments of the present application, the terminal 100 may include more or fewer components than those shown in this figure, or some components may be combined, or some components may be divided, or a different component layout may exist. The components shown in this figure may be implemented by hardware, software, or a combination of software and hardware.

[0075] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. The different processing units may be separate components or may be integrated into one or more processors.

[0076] A video codec is configured to compress or decompress digital video. Terminal 100 may support one or more video codecs. Thus, terminal 100 can play or record video in multiple coding formats, such as Moving Picture Experts Group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4. In this embodiment of the present application, the video codec can decode free-viewpoint video downloaded by terminal 100 from media server 300 and play the free-viewpoint video on terminal 100.

[0077] The internal memory 121 may be configured to store computer-executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system and applications required by at least one function (e.g., a playback function for free viewpoint video). The data storage area may store data created during use of the terminal 100 (e.g., voice data and contacts). In addition, the internal memory 121 may include high-speed random access memory or non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS). The processor 110 executes instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor to perform various functional applications and data processing of the terminal 100.

[0078] In some embodiments of the present application, the internal memory 121 stores program code required by the free viewpoint video client / application. The free viewpoint video client / application can implement functions for downloading free viewpoint video from the media server 300 and playing the free viewpoint video, receive barrage comment content entered by a user, upload the barrage comment content entered by a user and other barrage comment information to the barrage comment server 600, and receive a viewpoint switching operation performed by a user to switch the viewing point of the free viewpoint video currently being played.

[0079] 3 is a schematic diagram of the structure of a free viewpoint video client / application according to an embodiment of the present application. The free viewpoint video client / application includes a user interface module, a user control module, a barrage comment module, and a playback module.

[0080] The user interface module is configured to provide a graphical user interface (GUI) for interaction between the client / application and the user, display video images of the free viewpoint video, and the like.

[0081] The user control module is configured to receive a viewpoint switching operation performed by the user (e.g., a sliding operation performed on the display by the user's finger, or a button press on a remote control by the user), calculate the viewpoint switching direction (e.g., viewpoint switching to the left or viewpoint switching to the right) and switching angle amplitude (e.g., a 10-degree or 20-degree switch), and transmit the calculated viewpoint switching direction and switching angle amplitude to the viewpoint control module in the playback module to switch the viewing point of the current free viewpoint video.

[0082] In some examples, the display of the terminal 100 is a touchscreen. In this case, the user control module includes a display slide control module. The display slide control module is configured to capture display events, where the display events include detected finger slide events and include a finger slide direction and a finger slide distance. The display events are converted into a view point switching direction and a view point switching angle. In some other examples, the terminal 100 is configured with a remote control. In this case, the user control module includes a remote control button control module. The remote control button control module is configured to receive remote control button events, where the button events include a button identifier and a number of button presses or a press duration. Next, a view point switching direction and a view point switching angle are determined based on the button events. For example, a button operated by a user is identified based on the button identifier, and the left button or the right button is tapped to determine the switching direction. A switch angle amplitude is identified based on the number of button presses or a press duration.

[0083] The playback module includes a viewpoint control module, a data decapsulation module, a video rendering module, a data download module, an audio and video encoding module, and an audio and video synchronization module.

[0084] The viewpoint control module is configured to receive the viewpoint switching direction and switching angle amplitude instructed by the user and sent by the user control module, calculate a target viewpoint after switching based on the current viewpoint of the free-viewpoint video, and notify the data download module to download the video file of the target viewpoint. The data download module requests the video file corresponding to the target viewpoint from the media server 300 by calling relevant software and hardware interfaces of the terminal 100. Next, the data decapsulation module of the client / application of the terminal 100 is used to decapsulate the video file returned by the server 300 to obtain audio and video data, the video file is decoded using the audio and video decoding module, the video image is rendered using the video rendering module, and the video and audio in the media data are synchronized using the audio and video synchronization module, and then the video corresponding to the target viewpoint is played, thereby implementing viewpoint switching of the free-viewpoint video.

[0085] The danmaku comment module includes a danmaku comment input module, a danmaku comment display module, and a danmaku comment control module. The danmaku comment input module is configured to receive danmaku comment content input by a user (which may further include a format of the danmaku comment content selected by the user, such as color, font, bold, and italic) and upload the danmaku comment content and other danmaku comment information to the danmaku comment server 600. Optionally, the danmaku comment input module may further automatically fill in a viewpoint corresponding to the free viewpoint video when the user inputs the danmaku comment content. The danmaku comment display module is configured to display danmaku comment information corresponding to a current playback instant of the free viewpoint video. The danmaku comment control module is configured to receive a viewpoint switching operation by a user (e.g., the user taps a control corresponding to a target danmaku comment or the user selects a specific viewpoint option from a recommended viewpoint list), obtain a target viewpoint to be switched to through syntax analysis based on the user operation, and send the target viewpoint to the viewpoint control module of the playback module, so that the viewpoint control module switches the played free viewpoint video to the target viewpoint.

[0086] The wireless communication function of the terminal 100 may be implemented through an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a modem processor, a baseband processor, and the like.

[0087] Mobile communication module 150 may provide wireless communication solutions, including 2G / 3G / 4G / 5G / 6G / next-generation wireless communication technologies, applied to terminal 100. In some embodiments, at least some of the functional modules in mobile communication module 150 may be disposed in processor 110. In some embodiments, at least some of the functional modules in mobile communication module 150 may be disposed in the same device as at least some of the modules in processor 110.

[0088] The wireless communication module 160 may provide solutions for wireless communication applied to the terminal 100, such as a wireless local area network (WLAN) (e.g., a wireless fidelity (Wi-Fi®) network), Bluetooth® (Bluetooth®, BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, and infrared (IR) technology.

[0089] Terminal 100 may implement display functions by using a GPU, a display 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0090] The terminal 100 may implement image capture functionality by using an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, and the like.

[0091] The external memory interface 120 can be used to connect an external memory card, for example, a microSD card, to expand the storage capabilities of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 and implements data storage functions. For example, music files or video files can be stored on the external storage card.

[0092] The terminal 100 may implement audio functions such as music playback and recording using an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, and an application processor.

[0093] The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive the wireless charging input through a wireless charging coil of the terminal 100. The charging management module 140 may further provide power to the terminal by using the power management module 141 when charging the battery 142.

[0094] Power management module 141 is configured to connect to battery 142, charging management module 140, and processor 110. Power management module 141 receives input from battery 142 and / or charging management module 140 and provides power to processor 110, internal memory 121, display 194, camera 193, wireless communication module 160, etc. Power management module 141 may be further configured to monitor parameters such as battery capacity, battery cycle count, and battery health (electrical leakage or impedance). In some other embodiments, power management module 141 may alternatively be disposed in processor 110. In some other embodiments, power management module 141 and charging management module 140 may alternatively be disposed in the same component.

[0095] Please refer to Figure 4. Figure 4 is a schematic diagram of the structure of the barrage comment server 600. The barrage comment server 600 includes one or more processors 610, one or more memories 620, and one or more communication interfaces 630.

[0096] The processor 610, memory 620, and communication interface 630 are connected via a bus. The processor 610 may include a general-purpose central processing unit (CPU) (e.g., CPU0 and CPU1), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural-network processing unit (NPU), or an integrated circuit configured to control program execution of the present solution.

[0097] Typically, the memory 620 may be configured to store computer-executable program code, where the executable program code includes instructions. The memory 620 may include a program storage area and a data storage area. The program storage area may store an operating system, application program code, etc. In some examples, the data storage area stores barrage comment data for each free-viewpoint video uploaded by a user. In some other examples, the data storage area further stores a highlight viewpoint list for each free-viewpoint video determined based on the barrage comment data uploaded by a user. In addition, the memory 620 may include a high-speed random access memory or a non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS), etc.

[0098] The processor 610 executes instructions stored in the memory 620 to perform various functional applications and data processing of the barrage comment server 600. The processor 610 may further include multiple CPUs, and may be a single-CPU processor or a multi-CPU processor. A processor herein may be one or more devices, circuits, or processing cores configured to process data (e.g., computer program instructions).

[0099] The communication interface 630 may be configured to communicate with another device or communication network, for example, an Ethernet or a wireless local area network (WLAN).

[0100] Please note that the media processing server 200 and media server 300 shown in Fig. 1 should be referred to the structure of the barrage comment server 600 shown in Fig. 3. The media processing server 200 and media server 300 may include more or fewer components than those of the barrage comment server 600, or some components may be combined, or some components may be divided, or a different component arrangement may be used. This is not limited to the embodiments of the present application.

[0101] The following describes in detail the technical solutions provided in the embodiments of the present application.

[0102] 5A is a schematic flowchart of a method for sending barrage comments for a free viewpoint video according to an embodiment of the present application. The process includes the following steps:

[0103] S500: The first terminal requests a video file a of a first free viewpoint video at viewpoint A from the media server.

[0104] S501: The media server returns a video file a to the first terminal.

[0105] S502: The first terminal plays back the video image of the first free viewpoint video at viewpoint A based on the video file a.

[0106] For example, in steps S500 to S502, a client / application used to play free-viewpoint video is installed on a first terminal (which may be any one of terminals 100 in FIG. 1 ), and the client / application may be referred to as a free-viewpoint video client / application. A user may launch the client / application and select to play the first free-viewpoint video within the display interface of the client / application. In some embodiments, the first terminal plays a video image of the first free-viewpoint video from viewpoint A by default. When detecting a user's operation selecting to play the first free-viewpoint video, the first terminal requests video file a of the first free-viewpoint video from viewpoint A via a CDN from a media server. In response, the media server returns video file a to the first terminal via the CDN, and the first terminal displays the video image of the first free-viewpoint video from viewpoint A based on video file a. In some other embodiments, the first terminal first plays a video image of the first free-viewpoint video from a viewpoint other than viewpoint A. When the first terminal detects a user operation instructing switching to viewpoint A, the first terminal requests video file a of the first free viewpoint video at viewpoint A from the media server via the CDN. In response, the media server returns video file a to the first terminal via the CDN, and the first terminal displays a video image of the first free viewpoint video at viewpoint A based on video file a.

[0107] For example, the first terminal is a mobile phone. A video application installed on the mobile phone supports playback of free-viewpoint videos. In this case, a user can launch the video application in the mobile phone and select to play the first free-viewpoint video within the interface of the video application. For example, FIG. 6(1) shows a playback interface 601 displayed by the mobile phone for playing the first free-viewpoint video. The playback interface 601 includes a video image 602 of the first free-viewpoint video at a certain viewpoint and danmaku comments (e.g., danmaku comments 603) that appear within the video image 602 according to rules. The playback interface 601 may further include several function controls, such as a playback progress bar 604, a playback control 609, a next control 610, a danmaku comment on / off control 605, a danmaku comment input box 606, a danmaku comment send control 607, and a viewpoint control 608.

[0108] A playback progress bar 604 indicates the playback progress of the current video and is used by the user to adjust the playback progress of the current video in a dragging manner. A playback control 609 is used to pause or continue playback of the current video. A next control 610 is used to switch to the next video. A danmaku comment on / off control 605 is used to enable or disable the danmaku comment function. A danmaku comment input box 606 is used to receive danmaku comment content entered by the user. A danmaku comment send control 607 is used to send danmaku comment content entered by the user or default danmaku comment content.

[0109] The viewpoint control 608 is used to display a viewpoint (e.g., 60 degrees) corresponding to the video image 602 currently being played by the mobile phone, and is used to call up a viewpoint ruler used to switch viewpoints. When it is detected that a user has operated the viewpoint control 608, the mobile phone displays a playback interface 611 shown in FIG. 6(2), and a viewpoint ruler 612 is displayed within the playback interface 611. The user can use their finger to slide left or right on the viewpoint ruler 612 to switch the viewpoint of the free viewpoint video currently being played by the mobile phone. For example, when it is detected that the user's finger is sliding right on the ruler, the mobile phone displays a playback interface 613 shown in FIG. 7(1). The playback interface 613 is a video image of a first free viewpoint video at viewpoint A (e.g., 70 degrees).

[0110] It should be noted that, for the purpose of explanation, an example is used in which a user operates the viewpoint control 608 to call up the viewpoint ruler 612. In some other examples, the mobile phone may directly display the viewpoint ruler 612 or the like within the free viewpoint video playback interface 601. The interface for displaying the viewpoint ruler 612 is not limited in the embodiments of the present application.

[0111] In some examples, after the mobile phone does not receive a user operation on the viewpoint ruler within a preset duration 1, the mobile phone may hide the viewpoint ruler, for example, as in the playback interface 614 shown in Fig. 7(2). In some other examples, when the mobile phone detects a user tapping operation on the display on the playback interface 614, the mobile phone displays the playback interface 615 shown in Fig. 7(3), and the above-mentioned function controls, such as the playback progress bar 604, the playback control 609, the next control 610, the danmaku comment on / off control 605, the danmaku comment input box 606, the danmaku comment send control 607, and the viewpoint control 608, are displayed again in the playback interface 615.

[0112] S503: While playing back a video image of a first free viewpoint video at viewpoint A, the first terminal receives operation 1 by the user to input a first danmaku comment, and the first terminal generates data of the first danmaku comment. The data of the first danmaku comment includes viewpoint A and playback instant T1.

[0113] In some embodiments, when playing back the video image of the first free viewpoint video at viewpoint A, the first terminal further enables a danmaku comment function. In this case, the first terminal needs to further request danmaku comment data corresponding to a corresponding playback instant of the first free viewpoint video from the danmaku comment server based on the playback progress of the first free viewpoint video (step S503' in FIG. 5A ). In response, after receiving the danmaku comment data returned by the danmaku comment server, the first terminal displays the corresponding danmaku comment on the displayed video image (step S503'' in FIG. 5A ). In some examples, when watching the first free viewpoint video, the user can further use the first terminal to input the content of the first danmaku comment and transmit the data of the first danmaku comment to the danmaku comment server.

[0114] For example, as shown in FIG. 7(3), when the mobile phone plays back a video image of a first free viewpoint video at viewpoint A, the mobile phone detects that the user has input the content of the first danmaku comment into danmaku comment input box 606 and operated danmaku comment send control 607. In this case, the mobile phone generates data of the first danmaku comment. In this example, operation 1 includes inputting the content of the first danmaku comment (e.g., "Awesome") into danmaku comment input box 606 and operating danmaku comment send control 607. In this embodiment of the present application, the data of the first danmaku comment generated by the mobile phone includes viewpoint A (e.g., 70 degrees). In addition, the data of the first danmaku comment may further include one or more of an identifier of the first free viewpoint video, an identifier of the first danmaku comment, the content of the first danmaku comment, a playback instant T1 (referred to as the moment of occurrence of the first danmaku comment or the position of occurrence of the first danmaku comment, for example, 20:10), a duration ΔT of the first danmaku comment, a format of the first danmaku comment, and a transmission position of the first danmaku comment (the playback instant at which the first danmaku comment was transmitted), etc.

[0115] Table 1 shows an example of a data format used for the data of the first barrage comment. [Table 1]

[0116] The value of the video identifier field is the identifier of the first free viewpoint video, for example, video 1. The value of the danmaku comment identifier field is the identifier of the first danmaku comment, for example, danmaku comment 1. The value of the danmaku comment content field is the content of the first danmaku comment entered by the user, for example, "Amazing."

[0117] The value of the occurrence instant field is the playback instant T1, i.e., the playback instant at which the first danmaku comment starts to be displayed on the display. In some embodiments, the playback instant T1 is equal to the transmission instant of the first danmaku comment. For example, if a user of a first terminal transmits a first danmaku comment at 20:10 in the free viewpoint video, the occurrence instant of the first danmaku comment, i.e., the playback instant T1, may be 20:10. In some other embodiments, it may be understood that the occurrence instant of the first danmaku comment is different from the transmission instant of the first danmaku comment. For example, in some scenarios, the duration of highlights in a video is long, or a user is accustomed to transmitting danmaku comments at the start of highlights. In this case, the transmission instant of the first danmaku comment is determined as the playback instant T1 to remind another user to pay attention to highlights in a video. In some other embodiments, the playback instant T1 is equal to the transmission instant of the first danmaku comment minus a preset time offset value 1. The preset time offset value 1 may be predefined or may be set by a user. This is not limited to the embodiment of the present application. The setting manner and calculation method of other preset time offset values ​​(e.g., preset time offset value 2 to preset time offset value 5) in this specification may be similar to those of the preset time offset value 1, and may be the same as or different from those of the preset time offset value 1. Details will not be described below. For example, if the sending moment of the first danmaku comment is 20:10 and the preset time offset value 1 is 5 seconds, the occurrence moment of the first danmaku comment, i.e., the playback moment T1, is 20:05. In some other scenarios, it may be understood that the duration of a highlight in a video is short, or that a user is accustomed to sending a danmaku comment at the end of a highlight. In this case, in order to remind another user in advance to pay attention to the highlight in the video, the sending moment of the first danmaku comment minus the preset time offset value 1 is determined as the playback moment T1.In some other embodiments, the playback instant T1 is equal to the sending instant of the first danmaku comment plus a preset time offset value 2. For example, if the sending instant of the first danmaku comment is 20:10 and the preset time offset value 2 is 3 seconds, the occurrence instant of the first danmaku comment, i.e., the playback instant T1, is 20:13. The preset time offset value 1 and the preset time offset value 2 may be the same or different. In some other scenarios, it may be understood that users are accustomed to discussing before highlights. In this case, in order to remind other users to pay attention to highlights in the video in time, the sending instant of the first danmaku comment plus the preset time offset value 2 is determined as the playback instant T1.

[0118] The value of the Duration field is the duration ΔT (e.g., 15 seconds) during which the first danmaku comment continuously appears on the display. In some examples, the first danmaku comment appears from the rightmost side of the terminal's display, and then moves from the rightmost side to the leftmost side of the display until the first danmaku comment disappears from the display. The entire process of displaying the first danmaku comment from the rightmost side to the leftmost side of the display takes ΔT, e.g., 15 seconds. The duration ΔT may be a preset value or may be set by the user. For example, the user may set a danmaku comment mode. Different danmaku comment modes correspond to different durations. Values ​​of the Font field, Color field, etc. are set by the user when the user enters the content of the first danmaku comment, or are set by the first terminal by default. The value of the Viewpoint field is Viewpoint A.

[0119] S504: The first terminal transmits data of the first barrage comment to the barrage comment server.

[0120] S505: The barrage comment server stores data of the first barrage comment sent by the first terminal.

[0121] In some embodiments, after receiving the data of the first barrage comment sent by the first terminal, the barrage comment server reviews the data of the first barrage comment, for example, performing a content review and a format review, and after the review is successful, the barrage comment server stores the data of the first barrage comment.

[0122] Next, when another user uses another terminal to view the first free viewpoint video and enables the barrage comment function, if the first free viewpoint video is played back up to the playback moment T1, the played back video image includes the content of the first barrage comment.

[0123] In the following, for the sake of explanation, it is assumed that another user is a second terminal (which may be any one of the terminals 100 in FIG. 1, or , th This example uses an example in which a first free viewpoint video is viewed using a terminal (which may be the first terminal).

[0124] 5B is a schematic flowchart of a viewpoint switching method for free viewpoint video according to an embodiment of the present application. The process includes the following steps:

[0125] S506: As shown in step S506a in Figure 5B, the second terminal requests the media server to play video file b of the first free viewpoint video at playback moment T2 from viewpoint B. Viewpoint B is different from viewpoint A. As shown in step S506b in Figure 5B, the second terminal further requests danmaku comment data corresponding to playback moment T2 of the first free viewpoint video from the danmaku comment server. The danmaku comment data corresponding to playback moment T2 includes data of the first danmaku comment.

[0126] It should be noted that step S506b can be performed before, after, or simultaneously with step S506a, and the order of performing step S506a and step S506b is not limited in the embodiment of the present application.

[0127] S507: As shown in step S507a in Fig. 5B, the media server returns video file b to the second terminal. As shown in step S507b in Fig. 5B, the danmaku comment server returns danmaku comment data corresponding to playback instant T2 of the first free viewpoint video to the second terminal.

[0128] It should be noted that step S507b can be performed before, after, or during the execution of step S507a, and the execution order of step S507a and step S507b is not limited in the embodiment of the present application.

[0129] S508: The second terminal displays, based on the video file b, a video image of the first free viewpoint video at the playback moment T2 at viewpoint B and the first danmaku comment. Optionally, the second terminal further displays viewpoint information of the first danmaku comment, i.e., viewpoint A.

[0130] The playback instant T2 is greater than or equal to the playback instant T1 and less than or equal to the playback instant T1 plus the duration ΔT of the first barrage comment.

[0131] In steps S506 to S508, a client / application used to play the free viewpoint video is installed on the second terminal. The client / application installed on the second terminal may be the same as or different from the client / application installed on the first terminal. A user may instruct the second terminal to play a video image of the first free viewpoint video from an arbitrary viewpoint by operating the free viewpoint video client / application on the second terminal. Optionally, after the user enables the danmaku comment function, the second terminal further requests danmaku comment data of a corresponding playback moment from the danmaku comment server based on the playback progress of the first free viewpoint video, and displays the danmaku comment at the corresponding moment. For related operations performed by the user on the second terminal, please refer to the related operations on the first terminal in steps S500 to S502. Details will not be described again here.

[0132] For example, the second terminal is a mobile phone. As shown in FIG. 8(1), before playback instant T1 (e.g., playback instant T0, e.g., 20:05), the mobile phone continues to play video images of the first free viewpoint video at viewpoint B. That is, the second terminal requests video files of the first free viewpoint video at viewpoint B (including video file b of the first free viewpoint video at playback instant T2 at viewpoint B) from the media server through the CDN, and plays the video images of the first free viewpoint video at viewpoint B based on the video files returned by the media server through the CDN. Optionally, when the second terminal is in the process of playing the first free viewpoint video, the user further enables a danmaku comment function, and the second terminal further requests danmaku comment data of the first free viewpoint video at the corresponding playback instant (including danmaku comment data corresponding to playback instant T2 of the first free viewpoint video) from the danmaku comment server. The second terminal further displays a corresponding barrage comment based on the barrage comment data of the first free viewpoint video returned by the barrage comment server.

[0133] It should be noted that in this example, the second terminal may first request a video file of the first free viewpoint video at viewpoint B from the media server, and then request danmaku comment data of the first free viewpoint video at a corresponding playback instant from the danmaku comment server, i.e., first execute step S506a and then execute step S506b. Optionally, the second terminal may first request danmaku comment data of the first free viewpoint video at a corresponding playback instant from the danmaku comment server, and then request a video file of the first free viewpoint video at viewpoint B from the media server, i.e., first execute step S506b and then execute step S506a.

[0134] As shown in FIG. 8(2), when the playback instant is equal to playback instant T1 (e.g., 20:10), the second terminal starts displaying the first danmaku comment. In some examples, the first danmaku comment displayed by the second terminal moves from one side of the display to the other at a specific speed until the first danmaku comment disappears. The duration from the moment the second terminal starts displaying the first danmaku comment to the moment the first danmaku comment disappears is equal to the duration ΔT of the first danmaku comment (e.g., 15 seconds). In other words, the second terminal displays the first danmaku comment during a time period from playback instant T1 to T1 plus the duration ΔT of the first danmaku comment (e.g., 20:10 to 20:25). In some other examples, the second terminal further displays a viewpoint corresponding to the first danmaku comment, i.e., viewpoint A. That is, when the user views the first danmaku comment on the second terminal, the user can further understand that the first danmaku comment corresponds to viewpoint A.

[0135] In this example, playback instant T2 is any instant in the time period from playback instant T1 to playback instant T1 plus the duration ΔT of the first barrage comment. For example, playback instant T2 can be any playback instant between 20:10 and 20:25.

[0136] For the sake of explanation, an example in which the second terminal is a mobile phone will still be used. As shown in Fig. 9(1), before playback instant T1 (for example, playback instant T0, e.g., 20:05), the second terminal continues to play back a video image of the first free viewpoint video at viewpoint E (where viewpoint E may be viewpoint A or another viewpoint other than viewpoint B). As shown in Fig. 9(2), when the playback instant is playback instant T1 (for example, 20:10), the second terminal starts displaying the first danmaku comment. Note that in this case, the second terminal displays a video image of the first free viewpoint video at viewpoint E. For the display style of the first danmaku comment, please refer to the description of the display style of the first danmaku comment in Fig. 8(1) and Fig. 8(2), and the details will not be described again here.

[0137] Next, as shown in FIG. 9(3), when the first free viewpoint video is played back up to the video image at playback instant T3 (e.g., 20:15), playback instant T3 is after playback instant T1 and before T1+ΔT, and the terminal detects a user's operation to switch to viewpoint B. In this case, the second terminal requests video file b of the first free viewpoint video at viewpoint B from the media server via the CDN, and plays back the video image of the first free viewpoint video at viewpoint B based on video file b returned by the media server via the CDN. Note that in this case, the second terminal still displays the content of the first danmaku comment. That is, in this case, the second terminal displays the video image of the first free viewpoint video at viewpoint B and the first danmaku comment.

[0138] In this example, playback instant T2 is any instant in the time period from playback instant T3 (the instant after the first free viewpoint video is switched to viewpoint B) to playback instant T1 plus the duration ΔT of the first danmaku comment. For example, playback instant T5 can be any playback instant between 20:15 and 20:25.

[0139] It may be noted that in this example, the second terminal first requests the barrage comment data of the first free viewpoint video at the corresponding playback moment from the barrage comment server, and then requests the video file of the first free viewpoint video at viewpoint B from the media server, i.e., first executes step S506b and then executes step S506a.

[0140] S509: When the second terminal plays back the video image of the playback moment T2 of the first free viewpoint video at viewpoint B, the second terminal detects that the user has performed operation 2 on the first barrage comment, in which case the second terminal sends a viewpoint switching request to the media server to request the video file c of the playback moment T4 of the first free viewpoint video at viewpoint A.

[0141] Operation 2 is, for example, the user's finger operating a virtual control corresponding to the first danmaku comment on the display of the second terminal (which is a touch screen), or the user operating a button on a remote control to select a control corresponding to the first danmaku comment displayed on the second terminal.

[0142] The playback instant T4 is related to the playback instant T1. For example, the playback instant T4 is equal to the playback instant T1, or the playback instant T4 is equal to the playback instant T1 minus a preset time offset value of 3 (e.g., 5 seconds or 10 seconds). Alternatively, the playback instant T4 is related to the playback instant T2. For example, the playback instant T4 is equal to the playback instant T2, or the playback instant T4 is equal to the playback instant T2 minus a preset time offset value of 4.

[0143] For the sake of explanation, an example in which the second terminal is a mobile phone is used. Fig. 10(1) shows a playback interface 1001 displayed by the mobile phone. The playback interface 1001 includes a video image of a first free viewpoint video at viewpoint B (e.g., 80 degrees) and a corresponding danmaku comment (including a control 1002 corresponding to the first danmaku comment). For example, the playback interface 1001 is a video image at a playback moment of 20:15 at an 80-degree viewpoint. When the mobile phone detects that a user has operated the control 1002 (e.g., by tapping, double-tapping, touch-and-holding, or gravity pressing) (operation 2), the mobile phone determines the first danmaku comment based on the control 1002 and determines the viewpoint of the first danmaku comment (viewpoint A, e.g., 70 degrees) as the viewpoint after switching.

[0144] In addition, the mobile phone further determines the playback instant after the viewpoint is switched. In some examples, the mobile phone determines the playback instant after the viewpoint is switched, i.e., playback instant T4, based on the occurrence instant (playback instant T1) of the first danmaku comment. For example, playback instant T4 is equal to playback instant T1. For example, as shown in FIG. 10(1), the mobile phone detects that the user taps the control 1002 corresponding to the first danmaku comment in the playback interface 1001 at playback instant T2 (e.g., at 20:15), and the mobile phone switches from viewpoint B to viewpoint A, and the mobile phone displays the playback interface 1003 in FIG. 10(2). The occurrence instant (playback instant T1) of the first danmaku comment is 20:10, and therefore playback instant T4 is 20:10. 5. With reference to the description of the playback instant T1 in step S503, it can be seen that the playback instant T1 is adjusted to the start instant of the highlight (for example, the playback instant T1 is equal to the sending instant of the first danmaku comment minus the preset time offset value 1). Therefore, when the playback instant T4 is equal to the playback instant T1, it means that the mobile phone starts playing the highlight after switching to viewpoint A. As another example, the playback instant T4 is equal to the playback instant T1 minus the preset time offset value 3. With reference to the description of the playback instant T1 in step S503, it can be seen that the playback instant T1 is not adjusted to the start instant of the highlight (for example, the playback instant T1 is equal to the sending instant of the first danmaku comment). In this case, the moment the user sends the danmaku comment is after the highlight starts. Therefore, the mobile phone can adjust the playback instant T4 (the playback instant T4 is equal to the playback instant T1 minus the preset time offset value 3) so that playback starts from the actual moment of the highlight after the mobile phone switches viewpoint. Alternatively, the playback instant T1 is adjusted to the start instant of the highlight (e.g., the playback instant T1 is equal to the sending instant of the first barrage comment minus a preset time offset value 1), and the mobile phone can still start playback from an earlier playback instant before the highlight, thus further ensuring that the user does not miss the highlight moment.For example, playback instant T1 is 20:10 and playback instant T4 is 20:08. As another example, playback instant T4 can alternatively be after playback instant T1. For example, playback instant T1 is 20:10 and playback instant T4 is 20:15.

[0145] In some other examples, the mobile phone determines the playback moment after the viewpoint is switched, i.e., playback moment T4, based on the playback moment at which the viewpoint is switched (playback moment T2). For example, playback moment T4 is equal to playback moment T2. As shown in FIG. 10(1), the mobile phone detects that at playback moment T2 (e.g., at 20:15), the user taps the control 1002 corresponding to the first barrage comment in the playback interface 1001, and the mobile phone switches from viewpoint B to viewpoint A, and the mobile phone displays the playback interface 1004 in FIG. 10(3). Playback moment T4 is the same as playback moment T2 (e.g., Regeneration moment T4 It may be noted that playback instant T4 is equal to playback instant T2 minus a preset time offset value of 4. For example, playback instant T2 is 20:15 and playback instant T4 is 20:08. Of course, in some other examples, playback instant T4 may alternatively be after playback instant T2. For example, playback instant T2 is 20:15 and playback instant T4 is 20:18.

[0146] After the mobile phone determines the viewpoint to be switched to and the playback instant after the viewpoint is switched, the mobile phone sends a viewpoint switching request to the media server to request the video file c of the playback instant T4 of the first free viewpoint video at viewpoint A.

[0147] The viewpoint switching request includes an identifier of the first free viewpoint video and viewpoint information after switching (viewpoint A). Optionally, the viewpoint switching request further includes a playback instant after the viewpoint is switched, that is, a playback instant T4.

[0148] In some other embodiments, the mobile phone may not determine the playback instant T4, but the media server determines the playback instant T4. For example, the request sent by the mobile phone to the media server further includes the playback instant T1 or the playback instant T2. In this case, after receiving the switching request, the media server determines the playback instant T4 based on the playback instant T1 or the playback instant T2, and returns the video file c of the playback instant T4 to the second terminal.

[0149] S510: The media server returns the video file c to the second terminal.

[0150] S511: The second terminal plays back the video image of the first free viewpoint video at the playback moment T4 at the viewpoint A based on the video file c.

[0151] In this way, the second terminal switches from playing the video image of the first free viewpoint video at viewpoint B to playing the video image of the first free viewpoint video at viewpoint A. It may be noted that in the viewpoint switching solution provided in this embodiment of the present application, before viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint B from the media server through the CDN. After viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint A from the media server through the CDN. That is, in the viewpoint switching process, the second terminal does not request the video file of the first free viewpoint video at another viewpoint other than viewpoint A and viewpoint B from the media server.

[0152] In a conventional solution in which a user switches viewpoints by sliding a finger on a display or operating a remote control, the terminal requests video files of all viewpoints from viewpoint B to viewpoint A of a first free viewpoint video from a media server. For example, if a user instructs the terminal to switch from 60 degrees to 120 degrees, the terminal requests video files of the first free viewpoint video at all viewpoints between 60 degrees and 120 degrees (e.g., 70 degrees, 80 degrees, 90 degrees, 100 degrees, and 110 degrees) and displays video images corresponding to the viewpoints by linearly switching between the angles. However, in this embodiment of the present application, the terminal requests only the video file of the first free viewpoint video at 120 degrees from the media server and displays the video image at 120 degrees.

[0153] In the viewpoint switching solution provided in this embodiment of the present application, compared with that of the prior art, the second terminal no longer requests the video file of the first free viewpoint video at a viewpoint other than viewpoint A and viewpoint B from the media server, thus reducing traffic. In addition, the terminal only needs to receive the video file of the first free viewpoint video at viewpoint B and process the video file of the first free viewpoint video at viewpoint B, thus reducing the video processing work of the terminal, improving the processing efficiency of the terminal, accelerating the response speed of the terminal, and reducing the power consumption of the terminal.

[0154] In addition, when a viewpoint switching solution in the prior art is used, in the process of switching from viewpoint B to viewpoint A, the second terminal needs to switch and display video images at all viewpoints between viewpoint B and viewpoint A. When a user watches a moving video image, frequent viewpoint switching is performed for the moving video image, which is likely to cause dizziness to the user. As a result, the video viewing experience is deteriorated. However, in the viewpoint switching solution provided in this embodiment of the present application, frequent viewpoint switching is not performed, which helps to avoid dizziness to the user and improve the video viewing experience.

[0155] In some other embodiments, some transition images may be added in the process of switching from the video image of the first free viewpoint video at viewpoint B to the video image of the first free viewpoint video at viewpoint A. For example, during switching, a transition image frame is generated based on the last N frames (N is a natural number greater than or equal to 1) of the image frames of the first free viewpoint video at viewpoint B displayed by the terminal and the first M frames (M is a natural number greater than or equal to 1) of the image frames of the first free viewpoint video at viewpoint A to be displayed by the terminal, and the transition image frame is inserted between the last frame of the image frames of the first free viewpoint video at viewpoint B and the first frame of the image frames of the first free viewpoint video at viewpoint A, making the viewpoint switching of the first free viewpoint video more natural and realistic. As another example, during the switching, image frames at some viewpoints from viewpoint B to viewpoint A are selected and inserted between the last image frame of the first free viewpoint video at viewpoint B and the first image frame of the first free viewpoint video at viewpoint A, so that the viewpoint switching of the first free viewpoint video becomes more natural and realistic. limit Not determined.

[0156] It may be further noted that the first terminal that sends the first danmaku comment and the second terminal that switches the viewpoint based on the first danmaku comment interact with the danmaku comment server separately. Therefore, it is not limited whether the first terminal and the second terminal are the same type of terminal, whether the first terminal and the second terminal run the same operating system, and whether the first terminal and the second terminal run the same free viewpoint video playback client / application. In other words, the technical solution provided in this embodiment of the present application can implement viewpoint switching between terminals, between execution platforms, and between applications, and is widely applicable.

[0157] In conclusion, in this embodiment of the present application, when a user sends a first danmaku comment by using a first terminal, the first terminal automatically establishes a correspondence between the first danmaku comment and the viewpoint of the video currently being viewed by the user. Subsequently, when another user views the same free viewpoint video by using a second terminal and sees the first danmaku comment, the user can quickly switch to the viewpoint corresponding to the first danmaku comment by operating the first danmaku comment. Therefore, even if the target object has a vast shooting scene with a limited viewpoint range on the client / application and even blind areas, the user can still grasp the viewpoint with highlights from the content of the danmaku comment displayed on the video image. The highlights may be video clips that are highly praised by most users, or video clips that viewers are interested in, etc. Furthermore, by operating a specific danmaku comment, the user can quickly switch to the viewpoint corresponding to the specific danmaku comment and view the highlights of the viewpoint.

[0158] In some other embodiments, the media server or the second terminal can automatically identify the content of the first free viewpoint video, for example, identify the first free viewpoint video as a different clip based on one or more of the plot, scene, image capture method (e.g., full-body capture), and people of the video, etc. For example, the playback instant range of Clip 1 is from 15:00 to 25:00.

[0159] In response to an operation performed by the user on the first danmaku comment at playback instant T2 (20:15), the second terminal switches to viewpoint A corresponding to the first danmaku comment and plays back a video image of instant T4, where playback instant T4 may be the start instant of clip 1 (15:00), or may be the occurrence instant T1 (20:10) of the first danmaku comment, or may be the current instant T2 (20:15).

[0160] In some embodiments, playback instant T4 may be the start instant (15:00) of Clip 1, and the playback instant range of Clip 1 is 15:00 to 25:00. After the second terminal finishes playing the clip at Viewpoint A, i.e., at the clip's end instant 25:00, the second terminal can automatically switch back to the original viewpoint (e.g., Viewpoint B) and continue playing the clip after instant 25:00. Of course, in some other examples, the second terminal can alternatively ask the user whether to switch back to the original viewpoint and continue playing the clip after instant 25:00. After receiving the user's consent, the second terminal switches back to the original viewpoint and continues playing the clip after 25:00. In some other embodiments, the second terminal may not switch back to the original viewpoint B, but continues playing the clip after time 25:00 in Viewpoint A.

[0161] Of course, in some other embodiments, in the scenario shown in FIG. 10(2) or FIG. 10(3), after the second terminal plays the video clip corresponding to the first danmaku comment, the first danmaku comment appears at 20:10 and lasts for 15 seconds, i.e., the second terminal plays the video clip corresponding to 20:10 to 20:25, and at 25:00, the second terminal does not switch back to the original perspective of 70 degrees, but can continue to play the clip after 20:25, when the first danmaku comment ended, at the perspective of 80 degrees.

[0162] Of course, in some other embodiments, in the scenario shown in FIG. 10(2) or FIG. 10(3), after the display of the first barrage comment has finished, the terminal may alternatively automatically switch back to the original perspective at 20:25 and continue playing the clip after 20:25.

[0163] In the above-described embodiment, the user selects to switch to a viewpoint of interest for viewing based on the barrage comment content. In some other embodiments, the barrage comment server may alternatively recommend a viewpoint where a highlight is located for the user to select based on the collected barrage comment data. For example, in the process of playing a free viewpoint video, the number of barrage comments corresponding to a video clip may reflect the popularity and highlight degree of the video clip. If the number of barrage comments corresponding to a video clip is greater than a preset threshold, the video clip may be considered a highlight. The solution provided in this embodiment will be described in detail below.

[0164] In this embodiment, each user in the entire network can choose to view the video image of the first free viewpoint video from any viewpoint, and can also switch the viewpoint of the first free viewpoint video at will. In the process of playing the first free viewpoint video, if a user thinks a certain video clip is excellent, the user can input related barrage comment content. The terminal transmits the barrage comment content input by the user and other data of the barrage comment to the barrage comment server. It should be noted that the barrage comment data transmitted by the terminal to the barrage comment server includes information such as the barrage comment content, the identifier of the first free viewpoint video, the viewpoint of the barrage comment, and the moment the barrage comment was generated. Naturally, the barrage comment data may also include other content. Please refer to the related content in Table 1. Additionally, for the process of the terminal transmitting barrage comment data to the server, please refer to the related content in steps S500 to S505 above. Details will not be described again here.

[0165] The following describes in detail the process by which the barrage comment server calculates a recommended viewpoint list based on collected barrage comment data, and another user selects to switch viewpoints by using the recommended viewpoint list.

[0166] FIG. 11 is a schematic flowchart of another viewpoint switching method for free viewpoint video according to an embodiment of the present application, which includes the following steps.

[0167] S1101: The barrage comment server determines a recommended viewpoint corresponding to each playback moment based on the barrage comment data of the first free viewpoint video.

[0168] In some embodiments, the danmaku comment server periodically (e.g., at one-minute intervals) calculates a recommended viewpoint corresponding to each playback moment in the first free viewpoint video. In a specific implementation, the playback moment in the first free viewpoint video is divided into multiple time periods (e.g., each time period is 10 to 15 seconds long). Next, using the time periods as units, statistical information is collected based on different viewpoints to calculate a recommended viewpoint corresponding to each time period. For example, the time period corresponding to each danmaku comment is determined based on the occurrence moment corresponding to each danmaku comment in the danmaku comment data. Next, for each time period, statistical information is collected about the number of danmaku comments corresponding to each viewpoint in the time period based on the viewpoints corresponding to the danmaku comments in the danmaku comment data. If the number of danmaku comments from one viewpoint or multiple viewpoints in the time period is greater than or equal to a preset threshold, the one viewpoint or multiple viewpoints are determined as recommended viewpoints. In an example, if the numbers of danmaku comments for multiple viewpoints within a certain time period are all greater than or equal to a preset threshold, the first R viewpoints may be selected as recommended viewpoints in descending order of the numbers of danmaku comments corresponding to the viewpoints, where R is an integer greater than 0. For example, R=1 or R=5, or the user can manually configure the value of R. It should be noted that the calculation steps in this specification are merely examples of specific implementation forms and do not constitute limitations on the method for calculating recommended viewpoints for each time period.

[0169] For example, Table 2 shows some examples of barrage comment data stored in the barrage comment server. Based on the occurrence moment corresponding to each barrage comment, it can be determined that barrage comments 1 to 4 correspond to time period 1 (10:00 to 10:15). Furthermore, for time period 1, it can be calculated that the number of barrage comments corresponding to viewpoint 1 is 3, and the number of barrage comments corresponding to viewpoint 2 is 1. The number of barrage comments corresponding to viewpoint 1 and the number of barrage comments corresponding to viewpoint 2 are below a preset threshold, and no viewpoint is recommended in time period 1.

[0170] Similarly, danmaku comments 5 to 8 correspond to time slot 2 (10:16 to 10:30). Furthermore, for time slot 2, the number of danmaku comments corresponding to viewpoint 1 can be calculated to be 2, the number of danmaku comments corresponding to viewpoint 2 to be 1, and the number of danmaku comments corresponding to viewpoint 3 to be 1. The number of danmaku comments corresponding to viewpoint 1, the number of danmaku comments corresponding to viewpoint 2, and the number of danmaku comments corresponding to viewpoint 3 are all below the preset threshold, and no viewpoint is recommended in time slot 2.

[0171] Similarly, barrage comments 100 to 201 correspond to time slot 3 (20:16 to 20:30). Furthermore, for time slot 3, the number of barrage comments corresponding to viewpoint 3 can be calculated to be 200, and the number of barrage comments corresponding to viewpoint 1 can be calculated to be 1. If the number of barrage comments corresponding to viewpoint 3 is greater than or equal to a preset threshold (e.g., 150), it is determined that viewpoint 3 is the recommended viewpoint corresponding to time slot 3. [Table 2]

[0172] In some other embodiments, the barrage comment server may further determine a recommended viewpoint by referring to the specific content of the barrage comments and the number of barrage comments. For example, the content of some barrage comments indicates that the video image of that viewpoint is excellent. For example, the content of the barrage comments may be "great" and "this is recommended," etc. Next, the viewpoint is determined as a recommended viewpoint by referring to the condition that the number of barrage comments of that viewpoint meets a specific condition (for example, the number is greater than a preset threshold). As another example, the content of some barrage comments indicates that the video image of that viewpoint is not excellent. For example, if the content of the barrage comments is "too bad" or "skip this," etc., the viewpoint may be excluded as a recommended viewpoint. In conclusion, the method for determining a recommended viewpoint is not particularly limited in the embodiments of the present application.

[0173] S1102: The barrage comment server stores a recommended viewpoint corresponding to each playback moment.

[0174] S1103: The second terminal requests the video file d of the first free viewpoint video at the playback moment T5 in the viewpoint C from the media server.

[0175] The playback instant T5 can be any playback instant of the first free viewpoint video.

[0176] S1104: The media server returns the video file d to the second terminal.

[0177] S1105: The second terminal requests the barrage comment server for a recommended viewpoint at playback moment T5 of the first free viewpoint video.

[0178] S1106: The barrage comment server returns recommended viewpoints for the playback moment T5 of the first free viewpoint video to the second terminal. The recommended viewpoints for the playback moment T5 of the first free viewpoint video include viewpoint D.

[0179] It should be noted that step S1105 can be performed before, after, or simultaneously with step S1103. The order of performing step S1105 and step S1103 is not limited in the embodiment of the present application.

[0180] Note that in some embodiments, when the second terminal enables the danmaku comment function, the second terminal not only requests danmaku comment data corresponding to each playback moment from the danmaku comment server based on the playback progress of the first free viewpoint video, but also requests recommended viewpoints corresponding to each playback moment from the danmaku comment server. In some other embodiments, when the second terminal does not enable the danmaku comment function but enables the viewpoint recommendation function, the second terminal requests recommended viewpoints corresponding to each playback moment from the danmaku comment server based on the playback progress of the first free viewpoint video. In some other embodiments, when the second terminal starts playing the first free viewpoint video, the second terminal requests recommended viewpoints corresponding to all playback moments from the danmaku comment server. If the recommended viewpoint calculated by the danmaku comment server is subsequently changed, the second terminal may be notified to obtain the latest recommended viewpoint. If the recommended viewpoint calculated by the danmaku comment server is not changed or is changed only slightly, the second terminal may not be notified to obtain the latest recommended viewpoint. In this way, the second terminal can then display the recommended viewpoint corresponding to the corresponding playback moment based on the recommended viewpoint corresponding to all playback moments requested when playback started, the updated recommended viewpoint obtained thereafter, and the playback progress of the first free viewpoint video.

[0181] S1107: The second terminal plays the video image of the first free viewpoint video at the playback moment T5 at the viewpoint C based on the video file d, and the second terminal further displays the recommended viewpoint at the playback moment T5.

[0182] For example, the second terminal is a mobile phone. FIG. 12(1) shows a playback interface 1201 of a first free viewpoint video displayed by a mobile phone. The playback interface 1201 includes a recommended viewpoint list 1202, which includes recommended viewpoints. There may be one or more recommended viewpoints. In some examples, when there are multiple recommended viewpoints corresponding to the playback instant at which the danmaku comment server collects statistical information, the recommended viewpoint list 1202 may include multiple recommended viewpoints. Optionally, the multiple recommended viewpoints may be sorted in descending order of the number of danmaku comments corresponding to the recommended viewpoints, and the recommended viewpoint with a larger number of danmaku comments is preferentially recommended to the user, or the recommended viewpoint with a larger number of danmaku comments is ranked first. In some other examples, the recommended viewpoint list 1202 further includes a recommendation index used to indicate the priority of the recommended viewpoints. The priority of the recommended viewpoints may be related to the order of the number of danmaku comments corresponding to the recommended viewpoints. In some other examples, the recommended viewpoint list 1202 further includes the number of recommended viewpoints, etc.

[0183] S1108: The second terminal detects the user's operation of selecting viewpoint D, and requests the video file e of the first free viewpoint video at playback moment T6 in viewpoint D from the media server.

[0184] In some embodiments, the playback instant T6 is relative to the playback instant T5, for example, the playback instant T6 is equal to the playback instant T5, or the playback instant T6 is equal to the playback instant T5 minus a preset time offset value of 5 (e.g., the preset time offset value is 5 seconds).

[0185] For example, the second terminal is a mobile phone. In the playback interface 1201 shown in Fig. 12(1), the mobile phone displays a video image at a playback moment T5 (e.g., 20:16) at a viewpoint C (e.g., 70 degrees). When the mobile phone detects a user operation to select a viewpoint D (e.g., 60 degrees) in the recommended viewpoint list 1202 in the playback interface 1201 shown in Fig. 12(1), the mobile phone requests a video file e of the first free viewpoint video at a playback moment T6 (e.g., 20:16 or 20:11) at viewpoint D from the media server.

[0186] In some other embodiments, playback instant T6 is associated with the clip corresponding to playback instant T5. Specifically, the media server or the second terminal can automatically identify the content of the first free viewpoint video, for example, identifying the first free viewpoint video as a different clip based on one or more of the video's plot, scene, image capture method (e.g., full-body capture), and person, etc. Playback instant T6 is the playback start instant of the clip corresponding to playback instant T5. For example, the playback instant is 20:16, and the playback instant range of clip 2 is, for example, 20:00 to 25:00. It can be determined that playback instant T5 is 20:16 and corresponds to clip 2. Therefore, playback instant T6 can be 20:00.

[0187] S1109: The media server returns the video file e to the second terminal.

[0188] S1110: The second terminal plays back the video image of the first free viewpoint video at the viewpoint D at the playback moment T6 based on the video file e.

[0189] For example, in the playback interface 1203 shown in FIG. 12(2), the mobile phone displays a video image at viewpoint D.

[0190] In this way, the second terminal switches from playing the video image of the first free viewpoint video at viewpoint C to playing the video image of the first free viewpoint video at viewpoint D. It may be noted that in the viewpoint switching solution provided in this embodiment of the present application, before viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint C from the media server through the CDN. After viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint D from the media server through the CDN. That is, in the viewpoint switching process, the second terminal does not request the video file of the first free viewpoint video at another viewpoint other than viewpoint C and viewpoint D from the media server.

[0191] In addition, if the second terminal does not enable the barrage comment function or does not meet the conditions for displaying barrage comments, the second terminal may alternatively display a list of recommended viewpoints, and the user may display barrage comments based on the list of recommended viewpoints.

[0192] For other contents of this embodiment, please refer to the above description, and the details will not be described again here.

[0193] FIG. 13 is a schematic flowchart of another viewpoint switching method for free viewpoint video according to an embodiment of the present application, which includes the following steps.

[0194] S1310: The terminal displays a first interface, where the first interface has a video image of a first free viewpoint video at a first viewpoint and a first barrage comment, and the first barrage comment corresponds to a second viewpoint of the first free viewpoint video.

[0195] Optionally, the first barrage of comments in the first interface further comprises a second perspective.

[0196] In some examples, the terminal displays the first interface at the first playback instant, i.e., the first interface has a video image of the first free viewpoint video at the first viewpoint at the first playback instant.

[0197] In some other examples, before displaying the first interface at the first playback instant, the terminal requests danmaku comment data corresponding to the first playback instant of the first free viewpoint video from a danmaku comment server, where the danmaku comment data corresponding to the first playback instant includes data of a first danmaku comment, and the data of the first danmaku comment includes a second viewpoint.

[0198] S1320: When the terminal detects a first operation performed by the user on the first barrage comment, it displays a second interface, where the second interface includes a video image of the first free viewpoint video at a second viewpoint.

[0199] In some examples, when the terminal detects a first operation performed by the user on the first danmaku comment, it displays the second interface at a second playback instant. The second playback instant is related to the first playback instant. For example, the second playback instant is equal to the first playback instant, or the second playback instant is equal to the first playback instant minus a first preset time offset value. As another example, the first playback instant is related to the occurrence instant of the first danmaku comment. The second playback instant is also related to the occurrence instant of the first danmaku comment. In this case, the second playback instant is equal to the occurrence instant of the first danmaku comment, or the second playback instant is equal to the occurrence instant of the first danmaku comment minus a second preset time offset value.

[0200] In some other examples, when the terminal detects a first operation performed by the user on the first barrage comment, it sends a switching request to the media server. The media server returns a first video file to the terminal, where the first video file is a video file corresponding to a second playback instant of the first free viewpoint video at a second viewpoint. The terminal displays a video image of the second playback instant of the first free viewpoint video at the second viewpoint based on the first video file.

[0201] Optionally, the second interface includes the first barrage of comments.

[0202] Optionally, the terminal plays a third interface at a third playback instant, where the third playback instant is after the display time of the first danmaku comment, and the third interface has a video image of the first free viewpoint video at the second viewpoint, and the third interface does not have the first danmaku comment.

[0203] In some other examples, the terminal displays a fourth interface at a fourth playback instant, where the fourth interface has a video image of the first free viewpoint video at the first viewpoint and a recommended viewpoint, where the recommended viewpoint includes the fourth viewpoint. When the terminal detects a third operation by the user selecting the fourth viewpoint, the terminal displays a video image of the first free viewpoint video at a fifth playback instant at the fourth viewpoint, where the fifth playback instant is related to the fourth playback instant. For example, the fifth playback instant is equal to the fourth playback instant, or the fifth playback instant is equal to the fourth playback instant minus a fourth preset time offset value.

[0204] Optionally, before displaying the fourth interface at the fourth playback instant, the terminal requests a second video file corresponding to the fourth playback instant of the first free viewpoint video at the first viewpoint from the media server, and requests a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from the danmaku comment server, where the recommended viewpoint corresponding to the fourth playback instant is determined by the danmaku comment server based on the number of danmaku comments corresponding to the fourth playback instant of the first free viewpoint video.

[0205] The embodiment of the present application further provides a chip system. As shown in FIG. 14 , the chip system includes at least one processor 1301 and at least one interface circuit 1302. The processor 1301 and the interface circuit 1302 may be interconnected through a line. For example, the interface circuit 1302 may be configured to receive a signal from another device (e.g., a memory in the terminal 100). As another example, the interface circuit 1302 may be configured to transmit a signal to another device (e.g., the processor 1301). For example, the interface circuit 1302 may read an instruction stored in the memory and transmit the instruction to the processor 1301. When the instruction is executed by the processor 1301, the terminal may be able to perform the steps performed by the terminal 100 (e.g., a mobile phone) in the above-described embodiment. Of course, the chip system may further include another discrete device. This is not particularly limited in the embodiment of the present application.

[0206] An embodiment of the present application further provides an apparatus. The apparatus is included in a terminal, and the apparatus has a function for implementing the behavior of the terminal in any of the above-described embodiments. The function may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above-described function, such as a detection module or unit, a display module or unit, a determination module or unit, and a calculation module or unit.

[0207] An embodiment of the present application further provides a computer-readable storage medium containing computer instructions, which, when executed on a terminal, enable the terminal to perform any of the methods in the above-mentioned embodiments.

[0208] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to perform any of the methods in the above-described embodiments.

[0209] An embodiment of the present application further provides a graphical user interface on a terminal, the terminal including a display, a camera, a memory, and one or more processors, the one or more processors configured to execute one or more computer programs stored in the memory; the graphical user interface includes a graphical user interface displayed when the terminal executes any of the methods in the above-described embodiments.

[0210] It can be understood that to implement the above-mentioned functions, the terminal etc. includes corresponding hardware structures and / or software modules for performing the functions. In combination with the examples described in the embodiments disclosed herein, those skilled in the art should easily recognize that the units, algorithms, and steps can be implemented by hardware, or a combination of hardware and computer software in the embodiments of the present application. Whether the functions are performed by hardware or hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but the implementation form is not limited to the above. Original Vow should not be considered to go beyond the scope of the embodiments.

[0211] In the embodiment of the present application, a terminal or the like may be divided into functional modules based on the above-mentioned method examples. For example, each functional module may be obtained through division based on each corresponding function, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. Original Vow It should be noted that in this embodiment, the module division is an example and is merely a logical function division. In actual implementation, other division styles may be used.

[0212] The above description of the implementation form allows those skilled in the art to clearly understand that the division of the above-mentioned functional modules is taken as an example for explanation, for the purpose of convenient and concise description. In actual application, the above-mentioned functions can be allocated to different modules and implemented according to requirements, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above-mentioned functions. For detailed work processes of the above-mentioned systems, devices, and units, please refer to the corresponding processes in the above-mentioned method embodiments, and details will not be described again here.

[0213] The functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0214] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions in the embodiments of the present application may be essentially implemented in the form of a software product, or a portion of the technical solutions may be implemented in the form of a software product. A computer software product is stored in a storage medium and includes a plurality of instructions for instructing a computer device (which may be a personal computer, a server, a network device, etc.) or a processor to perform all or part of the steps of the methods described in the embodiments of the present application. The above-mentioned storage medium includes any medium capable of storing program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0215] The above description is merely a specific implementation form of the present application and is not intended to limit the scope of protection of the present application. Any modifications or replacements within the technical scope disclosed in this application shall be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims. (Other possible items) (Item 1) 1. A viewpoint switching method for free viewpoint video, comprising: a terminal displaying a first interface, wherein the first interface has a video image of a first free viewpoint video at a first viewpoint and a first barrage comment, the first barrage comment corresponding to a second viewpoint of the first free viewpoint video; and and displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment, wherein the second interface has a video image of the first free viewpoint video at the second viewpoint. A method comprising: (Item 2) The step of displaying the first interface by the terminal includes: the terminal displays the first interface at a first playback instant; having; The step of displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment includes: and displaying the second interface at a second playback instant when the terminal detects the first operation performed by the user on the first barrage comment, wherein the second playback instant is related to the first playback instant. Item 1. The method according to item 1, comprising: (Item 3) The second playback instant is related to the first playback instant by: The second playback instant is equal to the first playback instant, or the second playback instant is equal to the first playback instant minus a first preset time offset value; or the second playback instant is equal to the occurrence instant of the first danmaku comment, or the second playback instant is equal to the occurrence instant of the first danmaku comment minus a second preset time offset value. The method according to item 2, comprising: (Item 4) 4. The method according to any one of items 1 to 3, wherein the first barrage comments further include the second perspective. (Item 5) When the terminal detects the first operation performed by the user on the first barrage comment, the step of displaying the second interface at a second playback moment includes: sending a switching request to a media server when the terminal detects the first operation performed by the user on the first barrage comment; and receiving, by the terminal, a first video file returned by the media server, where the first video file is a video file corresponding to the second playback instant of the first free viewpoint video at the second viewpoint; and a step in which the terminal displays the video image at the second playback moment of the first free viewpoint video at the second viewpoint based on the first video file; The method according to any one of Items 2 to 4, comprising: (Item 6) Before the terminal displays the first interface at a first playback instant, the method includes: a step in which the terminal requests barrage comment data corresponding to the first playback instant of the first free viewpoint video from a barrage comment server, wherein the barrage comment data corresponding to the first playback instant has data of the first barrage comment, and the data of the first barrage comment includes the second viewpoint. The method according to any one of items 2 to 5, further comprising: (Item 7) After the terminal displaying the second interface at a second playback instant, the method further comprises: a step of receiving, by the terminal, a second operation to send a second danmaku comment, and a step of transmitting, by the terminal, data of the second danmaku comment to a danmaku comment server, wherein the data of the second danmaku comment includes content of the second danmaku comment and a third viewpoint, and the third viewpoint is a viewpoint corresponding to the first free viewpoint video during transmission of the second danmaku comment. The method according to any one of items 2 to 6, further comprising: (Item 8) 8. The method of claim 7, wherein the data of the second danmaku comment further includes a generation instant of the second danmaku comment, and the generation instant of the second danmaku comment is equal to the second playback instant, or the generation instant of the second danmaku comment is equal to the second playback instant minus a third preset time offset value. (Item 9) The second interface further includes the first barrage comment. The method according to any one of items 1 to 8. (Item 10) After the terminal displaying the second interface at a second playback instant, the method further comprises: a step of displaying a third interface at a third playback instant by the terminal, where the third playback instant is after a display time of the first danmaku comment, the third interface having the video image of the first free viewpoint video at the second viewpoint, and the third interface not having the first danmaku comment; The method according to any one of items 2 to 9, further comprising: (Item 11) The method comprises: displaying a fourth interface at a fourth playback instant, the fourth interface having the video image of the first free viewpoint video at the first viewpoint and a recommended viewpoint, the recommended viewpoint including a fourth viewpoint; and and when the terminal detects a third operation of selecting the fourth viewpoint by the user, displaying a video image of a fifth playback instant of the first free viewpoint video at the fourth viewpoint, wherein the fifth playback instant is related to the fourth playback instant. 11. The method according to any one of items 1 to 10, further comprising: (Item 12) The fifth replay moment is related to the fourth replay moment by: the fifth playback instant is equal to the fourth playback instant, or the fifth playback instant is equal to the fourth playback instant minus a fourth preset time offset value; Item 12. The method according to item 11, comprising: (Item 13) Before the terminal displays a fourth interface at a fourth playback instant, the method includes: a step of the terminal requesting a second video file corresponding to the fourth playback instant of the first free viewpoint video at the first viewpoint from a media server, and a step of requesting a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from a danmaku comment server, wherein the recommended viewpoint corresponding to the fourth playback instant is determined by the danmaku comment server based on the number of danmaku comments corresponding to the fourth playback instant of the first free viewpoint video. 13. The method according to item 11 or 12, further comprising: (Item 14) 14. A terminal comprising a processor, a memory, and a touch screen, wherein the memory and the touch screen are coupled to the processor, the memory is configured to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions in the memory, the terminal is capable of executing the viewpoint switching method for free viewpoint video described in any one of items 1 to 13. (Item 15) A computer-readable storage medium comprising computer instructions, which, when executed on a terminal, enable the terminal to execute the viewpoint switching method for free viewpoint video described in any one of items 1 to 13.

Claims

1. 1. A viewpoint switching method for free viewpoint video, comprising: a terminal displaying a first interface, wherein the first interface has a video image of a first free viewpoint video at a first viewpoint and a first danmaku comment, the first danmaku comment corresponding to a second viewpoint of the first free viewpoint video; and a step of displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment, wherein the second interface has a video image of the first free viewpoint video at the second viewpoint. A method comprising:

2. The step of displaying the first interface by the terminal includes: the terminal displays the first interface at a first playback instant; having The step of displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment includes: and displaying the second interface at a second playback instant when the terminal detects the first operation performed by the user on the first barrage comment, wherein the second playback instant is related to the first playback instant.

2. The method of claim 1, comprising:

3. The second playback instant is related to the first playback instant by: The second playback instant is equal to the first playback instant, or the second playback instant is equal to the first playback instant minus a first preset time offset value; or the second playback instant is equal to the occurrence instant of the first danmaku comment, or the second playback instant is equal to the occurrence instant of the first danmaku comment minus a second preset time offset value. The method of claim 2 , comprising:

4. The method of claim 1 , wherein the first barrage of comments further includes the second perspective.

5. When the terminal detects the first operation performed by the user on the first barrage comment, the step of displaying the second interface at a second playback moment includes: sending a switching request to a media server when the terminal detects the first operation performed by the user on the first barrage comment; and receiving, by the terminal, a first video file returned by the media server, where the first video file is a video file corresponding to the second playback instant of the first free viewpoint video at the second viewpoint; and a step in which the terminal displays the video image at the second playback moment of the first free viewpoint video at the second viewpoint based on the first video file; The method of claim 2 , comprising:

6. Before the terminal displays the first interface at a first playback instant, the method includes: a step in which the terminal requests barrage comment data corresponding to the first playback instant of the first free viewpoint video from a barrage comment server, wherein the barrage comment data corresponding to the first playback instant has data of the first barrage comment, and the data of the first barrage comment includes the second viewpoint. The method of claim 2 further comprising:

7. After the terminal displays the second interface at a second playback instant, the method further comprises: a step of receiving, by the terminal, a second operation to send a second danmaku comment, and a step of transmitting, by the terminal, data of the second danmaku comment to a danmaku comment server, wherein the data of the second danmaku comment includes content of the second danmaku comment and a third viewpoint, and the third viewpoint is a viewpoint corresponding to the first free viewpoint video during transmission of the second danmaku comment. The method of claim 2 further comprising:

8. 8. The method of claim 7, wherein the data of the second danmaku comment further includes a generation instant of the second danmaku comment, and the generation instant of the second danmaku comment is equal to the second playback instant, or the generation instant of the second danmaku comment is equal to the second playback instant minus a third preset time offset value.

9. The second interface further includes the first barrage of comments. The method of claim 1.

10. After the terminal displays the second interface at a second playback instant, the method further comprises: a step of the terminal displaying a third interface at a third playback instant, where the third playback instant is after a display time of the first danmaku comment, the third interface having the video image of the first free viewpoint video at the second viewpoint, and the third interface not having the first danmaku comment; The method of claim 2 further comprising:

11. The method comprises: displaying a fourth interface at a fourth playback instant, the fourth interface having the video image of the first free viewpoint video at the first viewpoint and a recommended viewpoint, the recommended viewpoint including a fourth viewpoint; and and when the terminal detects a third operation by the user to select the fourth viewpoint, displaying a video image of a fifth playback instant of the first free viewpoint video at the fourth viewpoint, wherein the fifth playback instant is related to the fourth playback instant. The method of claim 1 further comprising:

12. The fifth playback instant is related to the fourth playback instant by: the fifth playback instant is equal to the fourth playback instant, or the fifth playback instant is equal to the fourth playback instant minus a fourth preset time offset value; The method of claim 11 , comprising:

13. Before the terminal displays a fourth interface at a fourth playback instant, the method includes: a step of the terminal requesting a second video file corresponding to the fourth playback instant of the first free viewpoint video at the first viewpoint from a media server, and a step of requesting a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from a danmaku comment server, wherein the recommended viewpoint corresponding to the fourth playback instant is determined by the danmaku comment server based on the number of danmaku comments corresponding to the fourth playback instant of the first free viewpoint video. The method of claim 11 further comprising:

14. 14. A terminal comprising a processor, a memory and a touchscreen, wherein the memory and the touchscreen are coupled to the processor, and wherein the memory is configured to store computer program code, the computer program code comprising computer instructions, and wherein when the processor reads the computer instructions in the memory, the terminal is enabled to perform a method according to any one of claims 1 to 13.

15. A computer readable storage medium comprising computer instructions, which when executed on a terminal enable the terminal to carry out the method of any one of claims 1 to 13.

16. A computer program comprising computer program instructions which, when executed by a processing device, cause the processing device to carry out a method according to any one of claims 1 to 13.

Images