Method, device and system for switching viewpoints for free viewpoint video

Abstract

A method, an apparatus, and a system for switching a viewpoint of a free viewpoint video are provided, which are related to the field of Internet technology. According to the method, after detecting that a user operates a first danmaku comment, a terminal can automatically switch a playback viewpoint of the free viewpoint video to a viewpoint corresponding to the first danmaku comment to improve a viewing experience of the free viewpoint video. The method includes a step of a terminal displaying a first interface, where the first interface has a video image of the first free viewpoint video at a first viewpoint and a first danmaku comment, and the first danmaku comment corresponds to a second viewpoint of the first free viewpoint video; and a step of displaying a second interface, where the second interface has a video image of the first free viewpoint video at the second viewpoint, when the terminal detects a first operation performed by a user on the first danmaku comment.

Description

[Technical field]

[0001] The present application relates to the field of Internet technology, and in particular to a viewpoint switching method, apparatus and system for free viewpoint video. [Background technology]

[0002] Free viewpoint video (FVV) is a method of capturing a target object using multiple cameras to generate a video stream. ,So is encoded into a file like It will be , a video file in a special format. Then, the generated free viewpoint video is uploaded to a media server for storage, and then distributed to the terminal-side client through a content distribution network (CDN). When watching the free viewpoint video by using the terminal-side client, the user can freely switch the viewpoint by using a finger drag operation. Compared with the traditional video, the free viewpoint video allows the user to freely select the viewpoint, thus overcoming the shortcomings of the traditional video in which the viewpoint is determined by a director. The free viewpoint video provides the user with an immersive and interactive playback experience.

[0003] However, when the target object has a vast shooting scene, the user has a limited viewing range on the client, and therefore has blind areas, in which the user may miss some highlights that can be viewed at a viewing range other than the current viewing range. In this case, it is necessary to improve the viewing experience for the user. Summary of the Invention

[0004] According to the viewpoint switching method, device, and system for free viewpoint video provided in the present application, after detecting that a user has operated a first danmaku comment, the terminal automatically switches the playback viewpoint of the free viewpoint video to a viewpoint corresponding to the first danmaku comment, thereby improving the viewing experience of the free viewpoint video.

[0005] To achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect, a viewpoint switching method for a free viewpoint video is provided, the method comprising: a terminal displaying a first interface, where the first interface has a video image of the first free viewpoint video at a first viewpoint and a first barrage comment, the first barrage comment corresponding to a second viewpoint of the first free viewpoint video; and, when the terminal detects a first operation performed by a user on the first barrage comment, displaying a second interface, where the second interface has a video image of the first free viewpoint video at the second viewpoint.

[0007] It can be understood that when watching the first free viewpoint video, the user can quickly switch to a viewpoint corresponding to the first danmaku comment by operating the first danmaku comment, so that the user can easily watch the highlight of the viewpoint. It can be understood that when the first free viewpoint video has a vast shooting scene with a limited viewpoint range on the terminal and even a blind area, the user can still grasp the viewpoint having the highlight from the content of the danmaku comment displayed on the video image. The highlight may be a video clip that is highly praised by most users, or a video clip that is of interest to viewers, etc. By operating a specific danmaku comment, the user can quickly switch to a viewpoint corresponding to the specific danmaku comment to watch the highlight of the viewpoint, thus improving the user's viewing experience.

[0008] In a possible implementation, displaying the first interface by the terminal includes the terminal displaying the first interface at a first playback instant. Displaying the second interface when the terminal detects a first operation performed by the user on the first danmaku comment includes displaying the second interface at a second playback instant when the terminal detects a first operation performed by the user on the first danmaku comment, where the second playback instant is related to the first playback instant.

[0009] In other words, the first interface has a video image of a first playback instant of the first free viewpoint video at a first viewpoint, and the second interface has a video image of a second playback instant of the first free viewpoint video at a second viewpoint.

[0010] In possible implementation forms, the second playback instant being related to the first playback instant includes: the second playback instant being equal to the first playback instant, or the second playback instant being equal to the first playback instant minus a first preset time offset value; or the second playback instant being equal to the occurrence instant of the first danmaku comment, or the second playback instant being equal to the occurrence instant of the first danmaku comment minus a second preset time offset value.

[0011] That is, in some examples, after a user performs a first operation on a first danmaku comment in a first interface, the playback viewpoint is switched to a viewpoint corresponding to the first danmaku comment, and the playback instant is switched to a current playback instant of the first interface when the first operation is performed, i.e., a first playback instant. Alternatively, the playback instant is switched to an instant earlier than the first playback instant, i.e., the second playback instant is equal to the first playback instant minus a first preset time offset value. In some other examples, after a user performs a first operation on a first danmaku comment in a first interface, the playback viewpoint is switched to a viewpoint corresponding to the first danmaku comment, and the playback instant is switched to an occurrence instant of the first danmaku comment, or to an instant earlier than the occurrence instant of the first danmaku comment, i.e., the occurrence instant of the first danmaku comment minus a second preset time offset value.

[0012] In a possible implementation, the first barrage comment further includes a second perspective, i.e., a perspective corresponding to the first barrage comment is further displayed within the first barrage comment.

[0013] In a possible implementation form, displaying the second interface at the second playback instant when the terminal detects a first operation performed by the user on the first danmaku comment includes: sending a switching request to the media server when the terminal detects the first operation performed by the user on the first danmaku comment; the terminal receiving a first video file returned by the media server, where the first video file is a video file corresponding to the second playback instant of the first free viewpoint video at the second perspective; and the terminal displaying a video image of the second playback instant of the first free viewpoint video at the second perspective based on the first video file.

[0014] In a possible implementation form, before the terminal displays the first interface at the first playback instant, the method further includes the terminal requesting danmaku comment data corresponding to the first playback instant of the first free viewpoint video from a danmaku comment server, where the danmaku comment data corresponding to the first playback instant has data of a first danmaku comment, and the data of the first danmaku comment includes a second viewpoint.

[0015] In a possible implementation form, after the terminal displays the second interface at the second playback instant, the method further includes the terminal receiving a second operation of sending a second danmaku comment, and the terminal sending data of the second danmaku comment to the danmaku comment server, where the data of the second danmaku comment includes content of the second danmaku comment and a third viewpoint, and the third viewpoint is a viewpoint corresponding to the first free viewpoint video during the sending of the second danmaku comment.

[0016] In other words, when the terminal receives a second barrage comment input by a user for a video image at a third viewpoint, an association is established between the second barrage comment and the third viewpoint, and the content of the second barrage comment and information regarding the third viewpoint are transmitted to the barrage comment server.

[0017] In a possible implementation form, the data of the second barrage comment further includes an occurrence instant of the second barrage comment, wherein the occurrence instant of the second barrage comment is equal to the second playback instant, or the occurrence instant of the second barrage comment is equal to the second playback instant minus a third preset time offset value.

[0018] In a possible implementation, the second interface further includes a first danmaku comment, i.e., when the user performs a first operation on the first danmaku comment, the second playback instant after the switch is still within the duration of the continuous display of the first danmaku comment.

[0019] In a possible implementation form, after the terminal displays the second interface at the second playback instant, the method further includes the terminal displaying a third interface at a third playback instant, where the third playback instant is after the display time of the first danmaku comment, and the third interface has a video image of the first free viewpoint video at the second viewpoint, and the third interface does not have the first danmaku comment.

[0020] In a possible implementation form, the method further includes the terminal displaying a fourth interface at a fourth playback instant, where the fourth interface has a video image of the first free viewpoint video at the first viewpoint and a recommended viewpoint, the recommended viewpoint including the fourth viewpoint; and, when the terminal detects a third operation by the user to select the fourth viewpoint, displaying a video image of a fifth playback instant of the first free viewpoint video at the fourth viewpoint, where the fifth playback instant is associated with the fourth playback instant.

[0021] In a possible implementation, the fifth playback instant being related to the fourth playback instant includes the fifth playback instant being equal to the fourth playback instant, or the fifth playback instant being equal to the fourth playback instant minus a fourth preset time offset value.

[0022] In a possible implementation form, before the terminal displays the fourth interface at the fourth playback instant, the method further includes the terminal requesting a second video file corresponding to the fourth playback instant of the first free viewpoint video at the first viewpoint from the media server, and requesting a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from the danmaku comment server, wherein the recommended viewpoint corresponding to the fourth playback instant is determined by the danmaku comment server based on the number of danmaku comments corresponding to the fourth playback instant of the first free viewpoint video.

[0023] In other words, the user can quickly switch the playback viewpoint of the first free viewpoint video by using the recommended viewpoint displayed on the terminal, thus improving the user's video viewing experience.

[0024] In some embodiments, the barrage comment server may alternatively recommend to the user the viewpoints in which the highlights are located, for the user to select, based on the collected barrage comment data. For example, in the process of playing a free viewpoint video, the number of barrage comments corresponding to a video clip may reflect the popularity and highlight degree of the video clip. If the number of barrage comments corresponding to a video clip is greater than a preset threshold, the video clip may be regarded as a highlight. The user can quickly switch to a desired viewpoint by operating the recommended viewpoint.

[0025] According to a second aspect, a method for switching barrage comments is provided and applied to a barrage comment server, the method comprising: receiving data of a first barrage comment transmitted by a first terminal, where the data of the first barrage comment includes an identifier of a first free viewpoint video, a first viewpoint, and content of the first barrage comment, and the first barrage comment is transmitted by the first terminal when the first terminal plays a video image of the first free viewpoint video at the first viewpoint; and storing the data of the first barrage comment.

[0026] In a possible implementation, the data of the first barrage comment further includes a moment of occurrence of the first barrage comment.

[0027] In a possible implementation form, the method further includes receiving a first request sent by the second terminal, where the first request is used to request danmaku comment data for a first playback instant of the first free viewpoint video, the first playback instant being less than or equal to an occurrence instant of the first danmaku comment; and, in response to receiving the first request, returning the data of the first danmaku comment to the second terminal.

[0028] In a possible implementation form, after receiving the first request sent by the second terminal, the method further includes sending a recommended viewpoint list corresponding to a first playback instant of the first free viewpoint video to the second terminal.

[0029] In a possible implementation form, before sending the recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video to the second terminal, the method further includes separately collecting statistical information regarding the number of barrage comments of different viewpoints corresponding to the first playback moment of the first free viewpoint video; and determining the recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video based on the number of barrage comments of different viewpoints corresponding to the first playback moment of the first free viewpoint video.

[0030] In a possible implementation form, determining a recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video based on the number of danmaku comments of different viewpoints corresponding to the first playback moment of the first free viewpoint video further includes determining that the recommended viewpoint list corresponding to the first playback moment of the first free viewpoint video includes the second viewpoint if it is determined that the number of danmaku comments of a second viewpoint corresponding to the first playback moment of the first free viewpoint video is greater than a predetermined threshold.

[0031] In a possible implementation, there are one or more second viewpoints.

[0032] According to a third aspect, there is provided a terminal comprising a processor, a memory and a touch screen, the memory and the touch screen being coupled to the processor, the memory being configured to store computer program code, the computer program code including computer instructions, such that when the processor reads the computer instructions in the memory, the terminal is enabled to perform a method according to the first aspect and any one of the possible implementation forms of the first aspect.

[0033] According to a fourth aspect, there is provided a barrage comment server comprising a processor, a memory and a communication interface, the memory and the communication interface being coupled to the processor, the memory being configured to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions in the memory, the barrage comment server is enabled to perform a method according to the second aspect and any one of the possible implementation forms of the second aspect.

[0034] According to a fifth aspect, a computer readable storage medium is provided, comprising computer instructions which, when executed on a terminal, enable the terminal to perform a method according to the first aspect or the second aspect and any one of the possible implementation forms of the first aspect or the second aspect.

[0035] According to a sixth aspect, there is provided a computer program product which, when executed on a computer, enables a terminal to perform a method according to the first aspect or the second aspect and any one of the possible implementations of the first aspect or the second aspect.

[0036] For technical effects that can be achieved by the terminal, the barrage comment server, the computer-readable storage medium, and the computer program product provided in the third to sixth aspects, please refer to the description of the technical effects in the first aspect and possible implementation forms of the first aspect. Details will not be described again here. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a schematic diagram of the structure of a free viewpoint video playback system according to an embodiment of the present application.

[0038] [Diagram 2] 1 is a schematic diagram of a structure of a terminal according to an embodiment of the present application;

[0039] [Diagram 3] FIG. 2 is a schematic diagram of a free viewpoint video client / application structure according to an embodiment of the present application.

[0040] [Figure 4] FIG. 2 is a schematic diagram of a server structure according to an embodiment of the present application;

[0041] [Figure 5A] 1 is a schematic flowchart of a method for sending barrage comments about a free viewpoint video according to an embodiment of the present application;

[0042] [Figure 5B] 1 is a schematic flowchart of a viewpoint switching method for a free viewpoint video according to an embodiment of the present application;

[0043] [Figure 6(1)] 2 is a schematic diagram of several terminal interfaces according to an embodiment of the present application; [Figure 6 (2)] 2 is a schematic diagram of several terminal interfaces according to an embodiment of the present application;

[0044] [Figure 7(1)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 7(2)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 7(3)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0045] [Figure 8(1)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 8 (2)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0046] [Figure 9(1)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 9 (2)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 9 (3)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0047] [Figure 10(1)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 10(2)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 10(3)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0048] [Figure 11] 4 is a schematic flowchart of another viewpoint switching method for free viewpoint video according to an embodiment of the present application;

[0049] [Figure 12(1)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application. [Figure 12 (2)] 4A-4C are schematic diagrams of several other terminal interfaces according to embodiments of the present application.

[0050] [Figure 13] 1 is a schematic flowchart of yet another viewpoint switching method for free viewpoint video according to an embodiment of the present application;

[0051] [Figure 14] FIG. 1 is a schematic diagram of a structure of a chip system according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0052] In the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may represent A or B. In this specification, "and / or" describes only the associative relationship between related objects and represents that three relationships may exist. For example, A and / or B may represent the following three cases: when only A is present, when both A and B are present, and when only B is present. The terms "first" and "second" are intended for descriptive purposes only and should not be understood as an indication or suggestion of relative importance or an implicit indication of the quantity of the technical features shown. Thus, a feature qualified by "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. In addition, in the embodiments of the present application, the words "example" or "for example" are used to represent providing an example, illustration, or explanation. Any embodiment or design scheme described in the embodiments of this application as an "example" or "for example" should not be described as being more preferred or having more advantages over another embodiment or design scheme. Rather, the use of words such as "example" or "for example" is intended to present a relative concept in a concrete manner.

[0053] In order to facilitate the understanding of the technical solutions provided in the embodiments of the present application, technical terms in the embodiments of the present application are first explained.

[0054] A free viewpoint video is a video file in a special format that is generated through joint image capture performed on a target object by multiple cameras surrounding the target object and a specific video encoding of the video stream acquired through the joint image capture. The generated free viewpoint video is then uploaded to a media server for storage, and then distributed to a terminal-side client through a content distribution network (CDN). When a user watches the free viewpoint video on a terminal-side client, the user can manually select a viewing point.

[0055] Camera location is the number of the position where each camera is placed when multiple cameras surrounding a target object are taking images of the target object. Camera location corresponds to an image capture angle, which can be used to capture images of a target object within a certain angle range.

[0056] A viewpoint is also called a viewing point or a playing viewpoint, and is an image capturing angle at which a user views a free viewpoint video. In other words, an image of a target object seen from a certain viewpoint is captured when a camera is positioned at the viewpoint. (Camera position) The image may be regarded as an image captured when image capture is performed at

[0057] Video playback is a process in which a group of images are displayed in a time sequence within a display window and corresponding audio is played back after a video file (including a free viewpoint video) has been decoded or rendered.

[0058] Danmaku comments are comments or annotations that are displayed on video (including free viewpoint video) at specified times or during specified time periods according to rules, and bring an interesting and unexpected experience to video viewing.

[0059] 1 is a schematic diagram of a system architecture according to an embodiment of the present application. The system architecture includes a camera array 500, a media processing server 200, a media server 300, a content distribution network (CDN) 400, a terminal 100, and a barrage comment server 600.

[0060] The camera array 500 includes multiple (e.g., 18-36) cameras around the target object (object to be photographed / image capture scene), or another device with cameras. Camera location 0, Camera location 1, ..., and Camera location n separately. The camera array 500 performs image capture from multiple angles on a target object separately, and uploads video streams acquired through image capture from multiple angles to the media processing server 200.

[0061] The media processing server 200 is configured to transcode the received multiple angle video streams to generate free viewpoint video, which the media processing server 200 then uploads to the media server 300 for storage.

[0062] For example, the processing processes performed by the media processing server 200 on the video streams of multiple angles include, but are not limited to, focus alignment, frame alignment synchronization coding, rotation auxiliary stream generation, and virtual viewpoint synthesis. Focus alignment can be performed, for example, in an image cropping manner to ensure a smooth transition of the image of the captured object in the process of switching the viewpoint. Frame alignment synchronization coding is used to align the video frames obtained through image capture by multiple cameras to the same moment. Virtual viewpoint synthesis is the insertion of a virtual video stream into two real video streams to reduce the number of cameras that need to be used during image capture or to enable a smooth transition of the image of the captured object in the process of switching the viewpoint. The rotation auxiliary stream is used for rotation display, which helps to reduce the total traffic usage.

[0063] It should be noted that the processing process performed by the media processing server 200 is merely an example, and the specific technical solutions used in the process of generating free viewpoint videos based on the video streams uploaded by the camera array 500 are not limited to the embodiments of the present application.

[0064] It should be further noted that the media processing server 200 in this specification may be at least one of an independent physical server, multiple independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtualization center.

[0065] The media server 300 stores and maintains a large amount of free viewpoint videos. When the media server 300 receives a request sent by the terminal 100 for the content of the free viewpoint video, the media server 300 promptly returns the corresponding content of the free viewpoint video to the terminal 100 through the CDN 400.

[0066] It should be noted that the media server 300 in this specification may be at least one of an independent physical server, multiple independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtualization center.

[0067] The CDN 400 enables a user of a terminal (e.g., terminal 100) to obtain required content, such as free viewpoint video, locally by using edge servers deployed in various locations through functional modules of a central platform, such as a load balancing module, a content distribution module, and a scheduling module, thus reducing network congestion and improving user access response speed and hit rate.

[0068] The terminal 100 can request the content of the free viewpoint video from the media server 300 by using the free viewpoint video client installed in the terminal 100, and play the free viewpoint video based on the content of the free viewpoint video returned by the media server 300 through the CDN 400. If the terminal 100 further enables the barrage comment function when playing the free viewpoint video, the terminal 100 needs to further request the barrage comment information corresponding to the current playback moment or time period from the barrage comment server 600. In other words, the image played by the terminal 100 includes the image of the free viewpoint video and the corresponding barrage comment information. In addition, when playing the free viewpoint video, the terminal 100 can further receive the barrage comment content input by the user, and upload the barrage comment content input by the user and other barrage comment information to the barrage comment server 600 for storage.

[0069] For example, the terminal 100 in this embodiment of the present application may be, for example, a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a netbook, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, an in-vehicle device, a smart screen, or an intelligent vehicle, etc. The specific form of the terminal 100 is not particularly limited in this application.

[0070] The danmaku comment server 600 is configured to receive danmaku comment information transmitted by each terminal (including, for example, the terminal 100). The danmaku comment information includes, but is not limited to, an identifier of the free viewpoint video, the moment when the danmaku comment occurred, the duration of the danmaku comment, the danmaku comment content, and the viewing point during the transmission of the danmaku comment. The meaning of the parameters will be explained in detail later and will not be explained here.

[0071] It should be noted that the barrage comment server 600 in this specification may be at least one of an independent physical server, a plurality of independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtualization center.

[0072] Original Application It can be understood that the system structure shown in this embodiment of does not constitute a specific restriction on the system architecture to which the technical solutions are applicable. In some other embodiments of the present application, the system architecture to which the technical solutions provided in the embodiments of the present application are applicable may further include more or less devices than those shown in this figure, or a combination of some of the devices, or a division of some of the devices, etc.

[0073] 2 is a schematic diagram of an example of the structure of a terminal 100 according to an embodiment of the present application. The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a voice module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc.

[0074] Original Application It may be understood that the structure shown in this embodiment of does not constitute a specific limitation for the terminal 100. In some other embodiments of the present application, the terminal 100 may include more or fewer components than those shown in this figure, or some components may be combined, or some components may be split, or a different component layout may exist. The components shown in this figure may be implemented by hardware, software, or a combination of software and hardware.

[0075] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. The different processing units may be separate components or may be integrated into one or more processors.

[0076] A video codec is configured to compress or decompress digital video. The terminal 100 may support one or more video codecs. Thus, the terminal 100 can play or record video in multiple coding formats, such as moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4. In this embodiment of the present application, the video codec can decode free-viewpoint video downloaded by the terminal 100 from the media server 300 and play the free-viewpoint video on the terminal 100.

[0077] The internal memory 121 may be configured to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system and applications required by at least one function (e.g., a playback function for free viewpoint video), etc. The data storage area may store data (e.g., voice data and contacts) created during use of the terminal 100, etc. In addition, the internal memory 121 may include a high-speed random access memory, or may include a non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS), etc. The processor 110 executes instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor to perform various functions, applications, and data processing of the terminal 100.

[0078] In some embodiments of the present application, the internal memory 121 stores program codes required by a free viewpoint video client / application. The free viewpoint video client / application can implement a function of downloading a free viewpoint video from the media server 300 and playing the free viewpoint video, receive barrage comment content input by a user, upload the barrage comment content input by a user and other barrage comment information to the barrage comment server 600, and receive a viewpoint switching operation performed by a user to switch the viewing point of the free viewpoint video currently being played, etc.

[0079] For example, Figure 3 is a schematic diagram of the structure of a free viewpoint video client / application according to an embodiment of the present application. The free viewpoint video client / application includes a user interface module, a user control module, a barrage comment module, and a playback module.

[0080] The user interface module is configured to provide a graphical user interface (GUI) for interaction between a client / application and a user, and to display video images of the free viewpoint video, etc.

[0081] The user control module is configured to receive a viewpoint switching operation performed by a user (e.g., a sliding operation performed on the display by the user's finger, or an operation of the user pressing a button on a remote control), calculate the viewpoint switching direction (e.g., viewpoint switching to the left or viewpoint switching to the right) and switching angle amplitude (e.g., a switch of 10 degrees or 20 degrees), and transmit the calculated viewpoint switching direction and switching angle amplitude to the viewpoint control module in the playback module to switch the viewing point of the current free viewpoint video.

[0082] In some examples, the display of the terminal 100 is a touch screen. In this case, the user control module includes a display slide control module. The display slide control module is configured to capture display events, where the display events include a detected finger slide event and include a finger slide direction and a finger slide distance. The display events are converted into a switching direction and a switching angle of the viewing point. In some other examples, the terminal 100 is configured with a remote control. In this case, the user control module includes a button control module for the remote control. The button control module for the remote control is configured to receive button events of the remote control, where the button events include a button identifier and a number of times or a duration of pressing the button. Then, based on the button events, a switching direction and a switching angle of the viewing point are determined. For example, a button operated by a user is identified based on the button identifier, and, for example, a left button or a right button is tapped to determine a switching direction. A switching angle amplitude is identified based on a number of times or a duration of pressing the button.

[0083] The playback module includes a viewpoint control module, a data decapsulation module, a video rendering module, a data download module, an audio and video encoding module, and an audio and video synchronization module.

[0084] The viewpoint control module is configured to receive the viewpoint switching direction and switching angle amplitude instructed by the user and sent by the user control module, calculate the target viewpoint after switching based on the current viewpoint of the free viewpoint video, and notify the data download module to download the video file of the target viewpoint. The data download module requests the video file corresponding to the target viewpoint from the media server 300 by calling the relevant software and hardware interfaces of the terminal 100. Then, the data decapsulation module of the client / application of the terminal 100 is used to decapsulate the video file returned by the server 300 to obtain audio and video data, the video file is decoded by using the audio and video decoding module, the video image is rendered by using the video rendering module, the video and audio in the media data are synchronized by using the audio and video synchronization module, and then the video corresponding to the target viewpoint is played, thus implementing viewpoint switching of the free viewpoint video.

[0085] The barrage comment module includes a barrage comment input module, a barrage comment display module, and a barrage comment control module. The barrage comment input module is configured to receive a barrage comment content input by a user (which may further include a format of the barrage comment content selected by the user, such as color, font, bold, and italics), and upload the barrage comment content and other barrage comment information to the barrage comment server 600. Optionally, the barrage comment input module may further automatically fill in a viewpoint corresponding to the free viewpoint video when the user inputs the barrage comment content. The barrage comment display module is configured to display the barrage comment information corresponding to the current playback instant of the free viewpoint video. The barrage comment control module is configured to receive a user's viewpoint switching operation (e.g., the user taps a control corresponding to a target barrage comment, or the user selects a specific viewpoint option from a recommended viewpoint list), and obtains a target viewpoint to be switched to through syntax analysis based on the user operation, and transmits the target viewpoint to the viewpoint control module of the playback module, so that the viewpoint control module switches the played free viewpoint video to the target viewpoint.

[0086] The wireless communication function of the terminal 100 may be implemented through an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a modem processor, a baseband processor, and the like.

[0087] The mobile communication module 150 may provide a wireless communication solution including 2G / 3G / 4G / 5G / 6G / next-generation wireless communication technologies applied to the terminal 100. In some embodiments, at least some of the functional modules in the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules in the mobile communication module 150 may be disposed in the same device as at least some of the modules in the processor 110.

[0088] The wireless communication module 160 may provide a solution for wireless communication applied to the terminal 100, such as a wireless local area network (WLAN) (e.g., a wireless fidelity (Wi-Fi®) network), Bluetooth® (Bluetooth®, BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, and infrared (IR) technology.

[0089] The terminal 100 may implement display functions by using a GPU, a display 194, an application processor, and the like. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0090] The terminal 100 may implement image capture functionality by using an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, and the like.

[0091] The external memory interface 120 can be used to connect an external memory card, for example a microSD card, to expand the storage capabilities of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, music files or video files are stored in the external storage card.

[0092] The terminal 100 may implement audio functions, such as music playback and recording, using an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, and an application processor.

[0093] The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive the wireless charging input through a wireless charging coil of the terminal 100. The charging management module 140 may further provide power to the terminal by using the power management module 141 when charging the battery 142.

[0094] The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, etc. The power management module 141 may be further configured to monitor parameters such as battery capacity, battery cycle count, and battery health (electrical leakage or impedance). In some other embodiments, the power management module 141 may alternatively be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may alternatively be disposed in the same component.

[0095] Please refer to Fig. 4. Fig. 4 is a schematic diagram of the structure of a barrage comment server 600. The barrage comment server 600 includes one or more processors 610, one or more memories 620, and one or more communication interfaces 630.

[0096] The processor 610, the memory 620, and the communication interface 630 are connected through a bus. The processor 610 may include a general-purpose central processing unit (CPU) (e.g., CPU0 and CPU1), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural-network processing unit (NPU), or an integrated circuit configured to control program execution of the solution of the present application.

[0097] Typically, the memory 620 may be configured to store computer executable program code, where the executable program code includes instructions. The memory 620 may include a program storage area and a data storage area. The program storage area may store an operating system and application program code, etc. In some examples, the data storage area stores barrage comment data for each free viewpoint video uploaded by a user. In some other examples, the data storage area further stores a viewpoint list of highlights for each free viewpoint video, etc., determined based on the barrage comment data uploaded by a user. In addition, the memory 620 may include a high-speed random access memory, or may include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory, or a universal flash storage (UFS), etc.

[0098] The processor 610 executes various functional applications and data processing of the barrage comment server 600 by executing instructions stored in the memory 620. The processor 610 may further include multiple CPUs, and the processor 610 may be a single-CPU processor or a multi-CPU processor. A processor in this specification may be one or more devices, circuits, or processing cores configured to process data (e.g., computer program instructions).

[0099] The communication interface 630 may be configured to communicate with another device or communication network, for example, an Ethernet or a wireless local area network (WLAN).

[0100] Please note that for the media processing server 200 and the media server 300 shown in Fig. 1, refer to the structure of the barrage comment server 600 shown in Fig. 3. The media processing server 200 and the media server 300 may include more or fewer components than those of the barrage comment server 600, or some components may be combined, or some components may be divided, or a different component arrangement may be used. This is not limited to the embodiments of the present application.

[0101] The following describes in detail the technical solutions provided in the embodiments of the present application.

[0102] 5A is a schematic flowchart of a method for sending barrage comments for a free viewpoint video according to an embodiment of the present application. The process includes the following steps:

[0103] S500: The first terminal requests a video file a of a first free viewpoint video at viewpoint A from the media server.

[0104] S501: The media server returns a video file a to the first terminal.

[0105] S502: The first terminal plays back a video image of the first free viewpoint video at viewpoint A based on the video file a.

[0106] For example, in steps S500 to S502, a client / application used to play a free viewpoint video is installed in a first terminal (which may be any one of the terminals 100 in FIG. 1), and the client / application may be referred to as a free viewpoint video client / application. A user may launch the client / application and select to play the first free viewpoint video in a display interface of the client / application. In some embodiments, the first terminal plays a video image of the first free viewpoint video at viewpoint A by default. When detecting an operation by the user to select to play the first free viewpoint video, the first terminal requests a video file a of the first free viewpoint video at viewpoint A from a media server through a CDN. In response, the media server returns the video file a to the first terminal through the CDN, and the first terminal displays the video image of the first free viewpoint video at viewpoint A based on the video file a. In some other embodiments, the first terminal first plays a video image of the first free viewpoint video at a viewpoint other than viewpoint A. When the first terminal detects a user's operation instructing switching to viewpoint A, the first terminal requests the media server via the CDN for video file a of the first free viewpoint video at viewpoint A. In response, the media server returns video file a to the first terminal via the CDN, and the first terminal displays the video image of the first free viewpoint video at viewpoint A based on video file a.

[0107] For example, the first terminal is a mobile phone. A video application installed on the mobile phone supports playing free viewpoint videos. In this case, a user can launch the video application in the mobile phone and select to play the first free viewpoint video in the interface of the video application. For example, FIG. 6(1) shows a playing interface 601 displayed by the mobile phone for playing the first free viewpoint video. The playing interface 601 includes a video image 602 of the first free viewpoint video at a certain viewpoint, and a barrage comment (including, for example, a barrage comment 603) that appears in the video image 602 according to a rule. The playing interface 601 may further include some function controls, such as a playing progress bar 604, a playing control 609, a next control 610, a barrage comment on / off control 605, a barrage comment input box 606, a barrage comment send control 607, and a viewpoint control 608.

[0108] A playback progress bar 604 indicates the playback progress of the current video and is used by the user to adjust the playback progress of the current video in a drag manner. A playback control 609 is used to pause or continue the playback of the current video. A next control 610 is used to switch to the next video. A danmaku comment on / off control 605 is used to enable or disable the danmaku comment function. A danmaku comment input box 606 is used to receive danmaku comment content entered by the user. A danmaku comment send control 607 is used to send danmaku comment content entered by the user or default danmaku comment content.

[0109] The viewpoint control 608 is used to display a viewpoint (e.g., 60 degrees) corresponding to the video image 602 currently being played by the mobile phone, and is used to call up a viewpoint ruler used to switch viewpoints. When it is detected that the user has operated the viewpoint control 608, the mobile phone displays a playback interface 611 shown in FIG. 6(2), and a viewpoint ruler 612 is displayed in the playback interface 611. The user can use his / her finger to slide leftward or rightward on the viewpoint ruler 612 to switch the viewpoint of the free viewpoint video currently being played by the mobile phone. For example, when it is detected that the user's finger is slid rightward on the ruler, the mobile phone displays a playback interface 613 shown in FIG. 7(1). The playback interface 613 is a video image of a first free viewpoint video at viewpoint A (e.g., 70 degrees).

[0110] It should be noted that, for the purpose of explanation, an example is used in which a user operates the viewpoint control 608 to call up the viewpoint ruler 612. In some other examples, the mobile phone may directly display the viewpoint ruler 612 or the like in the free viewpoint video playback interface 601. The interface for displaying the viewpoint ruler 612 is not limited in the embodiment of the present application.

[0111] In some examples, after the mobile phone does not receive a user operation on the viewpoint ruler within a preset duration 1, the mobile phone may hide the viewpoint ruler, for example, as in the playback interface 614 shown in Fig. 7(2). In some other examples, when the mobile phone detects a user tapping operation on the display on the playback interface 614, the mobile phone displays the playback interface 615 shown in Fig. 7(3), and the above-mentioned functional controls such as the playback progress bar 604, the playback control 609, the next control 610, the barrage comment on / off control 605, the barrage comment input box 606, the barrage comment send control 607, and the viewpoint control 608 are displayed again in the playback interface 615.

[0112] S503: When playing back a video image of a first free viewpoint video at viewpoint A, the first terminal receives an operation 1 of inputting a first danmaku comment by a user, and the first terminal generates data of the first danmaku comment. The data of the first danmaku comment includes viewpoint A and a playback instant T1.

[0113] In some embodiments, when playing the video image of the first free viewpoint video at viewpoint A, the first terminal further enables the barrage comment function. In this case, the first terminal needs to further request the barrage comment data corresponding to the corresponding playback instant of the first free viewpoint video from the barrage comment server based on the playback progress of the first free viewpoint video. Step S503' in FIG. 5A. In response, after receiving the barrage comment data returned by the barrage comment server, the first terminal displays the corresponding barrage comment on the displayed video image. Step S503'' in FIG. 5A. In some examples, when watching the first free viewpoint video, the user can further input the content of the first barrage comment and transmit the data of the first barrage comment to the barrage comment server by using the first terminal.

[0114] For example, as shown in FIG. 7(3), when the mobile phone plays back the video image of the first free viewpoint video at viewpoint A, the mobile phone detects that the user has input the content of the first danmaku comment into the danmaku comment input box 606 and operated the danmaku comment send control 607. In this case, the mobile phone generates data of the first danmaku comment. In this example, operation 1 includes inputting the content of the first danmaku comment (e.g., "Awesome") into the danmaku comment input box 606 and operating the danmaku comment send control 607. In this embodiment of the present application, the data of the first danmaku comment generated by the mobile phone includes viewpoint A (e.g., 70 degrees). In addition, the data of the first barrage comment may further include one or more of an identifier of the first free viewpoint video, an identifier of the first barrage comment, content of the first barrage comment, a playback instant T1 (referred to as the instant of occurrence of the first barrage comment or the position of occurrence of the first barrage comment, e.g., 20:10), a duration ΔT of the first barrage comment, a format of the first barrage comment, and a transmission position of the first barrage comment (the playback instant at which the first barrage comment was transmitted), etc.

[0115] Table 1 shows an example of a data format used for the data of the first barrage comment. [Table 1]

[0116] The value of the video identifier field is an identifier of the first free viewpoint video, e.g., video 1. The value of the danmaku comment identifier field is an identifier of the first danmaku comment, e.g., danmaku comment 1. The value of the danmaku comment content field is the content of the first danmaku comment entered by the user, e.g., "Amazing."

[0117] The value of the occurrence instant field is the playback instant T1, i.e., the playback instant at which the first danmaku comment starts to be displayed on the display. In some embodiments, the playback instant T1 is equal to the transmission instant of the first danmaku comment. For example, if the user of the first terminal transmits the first danmaku comment at 20:10 of the free viewpoint video, the occurrence instant of the first danmaku comment, i.e., the playback instant T1, may be 20:10. In some other embodiments, it may be understood that the occurrence instant of the first danmaku comment is different from the transmission instant of the first danmaku comment. For example, in some scenarios, the duration of a highlight in a video is long, or the user is accustomed to transmitting a danmaku comment at the beginning of a highlight. In this case, the transmission instant of the first danmaku comment is determined as the playback instant T1 in order to remind another user to pay attention to the highlight in the video. In some other embodiments, the playback instant T1 is equal to the transmission instant of the first danmaku comment minus a preset time offset value 1. The preset time offset value 1 may be predefined or may be set by a user. This is not limited in the embodiment of the present application. The setting manner and calculation method of other preset time offset values ​​(e.g., preset time offset value 2 to preset time offset value 5) in this specification are similar to those of the preset time offset value 1, and may be the same as those of the preset time offset value 1, or may be different. In the following, details will not be described. For example, if the sending moment of the first barrage comment is 20:10 and the preset time offset value 1 is 5 seconds, the occurrence moment of the first barrage comment, that is, the playback moment T1, is 20:05. It may be understood that in some other scenarios, the duration of the highlight in the video is short, or the user is accustomed to sending the barrage comment at the end of the highlight. In this case, in order to remind another user in advance to pay attention to the highlight in the video, the sending moment of the first barrage comment minus the preset time offset value 1 is determined as the playback moment T1.In some other embodiments, the playback instant T1 is equal to the sending instant of the first barrage comment plus the preset time offset value 2. For example, if the sending instant of the first barrage comment is 20:10 and the preset time offset value 2 is 3 seconds, the occurrence instant of the first barrage comment, i.e., the playback instant T1, is 20:13. The preset time offset value 1 and the preset time offset value 2 may be the same or different. In some other scenarios, it may be understood that users are accustomed to discussing before highlights, etc. In this case, in order to remind other users to pay attention to the highlights in the video in time, the sending instant of the first barrage comment plus the preset time offset value 2 is determined as the playback instant T1.

[0118] The value of the duration field is a duration ΔT (e.g., 15 seconds) during which the first danmaku comment continuously appears on the display. In some examples, the first danmaku comment appears from the rightmost side of the display of the terminal, and then the first danmaku comment moves from the rightmost side of the display to the leftmost side until the first danmaku comment disappears from the display. The whole process of displaying the first danmaku comment from the rightmost side of the display to the leftmost side of the display takes ΔT, e.g., 15 seconds. The duration ΔT may be a preset value or may be set by a user. For example, a user may set a danmaku comment mode. Different danmaku comment modes correspond to different durations. Values ​​of the font field, color field, etc. are set by a user when the user inputs the content of the first danmaku comment, or are set by the first terminal by default. The value of the viewpoint field is viewpoint A.

[0119] S504: The first terminal transmits data of the first barrage comment to the barrage comment server.

[0120] S505: The barrage comment server stores data of the first barrage comment transmitted by the first terminal.

[0121] In some embodiments, after receiving the data of the first barrage comment sent by the first terminal, the barrage comment server reviews the data of the first barrage comment, for example, performing a content review and a format review, and after the review is successful, the barrage comment server stores the data of the first barrage comment.

[0122] Next, when another user uses another terminal to view the first free viewpoint video and activates the barrage comment function, if the first free viewpoint video is played back up to the playback moment T1, the played back video image includes the content of the first barrage comment.

[0123] In the following, for the sake of explanation, it is assumed that another user is a second terminal (which may be any one of the terminals 100 in FIG. 1 or , No. Here, an example will be used in which a first free viewpoint video is viewed using a first terminal (which may be the first terminal).

[0124] 5B is a schematic flowchart of a viewpoint switching method for a free viewpoint video according to an embodiment of the present application. The process includes the following steps:

[0125] S506: As shown in step S506a in Fig. 5B, the second terminal requests the media server to play the video file b of the first free viewpoint video at the playback moment T2 in the viewpoint B. The viewpoint B is different from the viewpoint A. As shown in step S506b in Fig. 5B, the second terminal further requests the danmaku comment server for danmaku comment data corresponding to the playback moment T2 of the first free viewpoint video. The danmaku comment data corresponding to the playback moment T2 includes data of the first danmaku comment.

[0126] It should be noted that step S506b can be performed before, after, or simultaneously with step S506a. The order of performing steps S506a and S506b is not limited in the embodiment of the present application.

[0127] S507: As shown in step S507a in Fig. 5B, the media server returns the video file b to the second terminal. As shown in step S507b in Fig. 5B, the danmaku comment server returns the danmaku comment data corresponding to the playback instant T2 of the first free viewpoint video to the second terminal.

[0128] It should be noted that step S507b can be performed before, after, or during the execution of step S507a. The execution order of steps S507a and S507b is not limited in the embodiment of the present application.

[0129] S508: The second terminal displays the video image of the first free viewpoint video at the playback moment T2 in viewpoint B and the first barrage comment based on the video file b. Optionally, the second terminal further displays viewpoint information of the first barrage comment, i.e., viewpoint A.

[0130] The playback instant T2 is greater than or equal to the playback instant T1 and less than or equal to the playback instant T1 plus the duration ΔT of the first barrage comment.

[0131] In steps S506 to S508, a client / application used to play the free viewpoint video is installed on the second terminal. The client / application installed on the second terminal may be the same as or different from the client / application installed on the first terminal. The user may instruct the second terminal to play the video image of the first free viewpoint video at any viewpoint by operating the client / application of the free viewpoint video on the second terminal. Optionally, after the user enables the barrage comment function, the second terminal further requests the barrage comment data of the corresponding playback moment from the barrage comment server based on the playback progress of the first free viewpoint video, and displays the barrage comment at the corresponding moment. For related operations performed by the user on the second terminal, please refer to the related operations on the first terminal in steps S500 to S502. Details will not be described again here.

[0132] For example, the second terminal is a mobile phone. As shown in FIG. 8(1), before the playback instant T1 (for example, playback instant T0, for example 20:05), the mobile phone continues to play the video image of the first free viewpoint video at viewpoint B. That is, the second terminal requests the video file of the first free viewpoint video at viewpoint B (including the video file b of the first free viewpoint video at playback instant T2 at viewpoint B) from the media server through the CDN, and plays the video image of the first free viewpoint video at viewpoint B based on the video file returned by the media server through the CDN. Optionally, when the second terminal is in the process of playing the first free viewpoint video, the user further enables the barrage comment function, and the second terminal further requests the barrage comment data of the first free viewpoint video at the corresponding playback instant (including the barrage comment data corresponding to the playback instant T2 of the first free viewpoint video) from the barrage comment server. The second terminal further displays a corresponding danmaku comment based on the danmaku comment data of the first free viewpoint video returned by the danmaku comment server.

[0133] Please note that in this example, the second terminal may first request the video file of the first free viewpoint video at viewpoint B from the media server, and then request the barrage comment data of the first free viewpoint video at the corresponding playback moment from the barrage comment server, i.e., first execute step S506a and then execute step S506b. Optionally, the second terminal may first request the barrage comment data of the first free viewpoint video at the corresponding playback moment from the barrage comment server, and then request the video file of the first free viewpoint video at viewpoint B from the media server, i.e., first execute step S506b and then execute step S506a.

[0134] As shown in FIG. 8(2), when the playback instant is equal to the playback instant T1 (e.g., 20:10), the second terminal starts displaying the first danmaku comment. In some examples, the first danmaku comment displayed by the second terminal moves from one side of the display to the other side at a certain speed until the first danmaku comment disappears. The duration from the moment the second terminal starts displaying the first danmaku comment to the moment the first danmaku comment disappears is equal to the duration ΔT of the first danmaku comment (e.g., 15 seconds). In other words, the second terminal displays the first danmaku comment in a time period from the playback instant T1 to T1 plus the duration ΔT of the first danmaku comment (e.g., 20:10 to 20:25). In some other examples, the second terminal further displays a viewpoint corresponding to the first danmaku comment, i.e., viewpoint A. That is, when the user views the first danmaku comment on the second terminal, the user can further understand that the first danmaku comment corresponds to viewpoint A.

[0135] In this example, the playback instant T2 is any instant in the time period from the playback instant T1 to the playback instant T1 plus the duration ΔT of the first barrage comment. For example, the playback instant T2 can be any playback instant between 20:10 and 20:25.

[0136] For the sake of explanation, an example in which the second terminal is a mobile phone is still used. As shown in FIG. 9(1), before the playback instant T1 (for example, playback instant T0, for example, 20:05), the second terminal continues to play the video image of the first free viewpoint video at viewpoint E (wherein, viewpoint E can be viewpoint A, or another viewpoint other than viewpoint B). As shown in FIG. 9(2), when the playback instant is playback instant T1 (for example, 20:10), the second terminal starts to display the first danmaku comment. In this case, it should be noted that the second terminal displays the video image of the first free viewpoint video at viewpoint E. For the display style of the first danmaku comment, refer to the description of the display style of the first danmaku comment in FIG. 8(1) and FIG. 8(2), and the details will not be described again here.

[0137] Next, as shown in FIG. 9(3), when the first free viewpoint video is played back up to the video image at the playback instant T3 (for example, 20:15), the playback instant T3 is after the playback instant T1 and before T1+ΔT, and the terminal detects the user's operation of switching to viewpoint B. In this case, the second terminal requests the video file b of the first free viewpoint video at viewpoint B from the media server through the CDN, and plays the video image of the first free viewpoint video at viewpoint B based on the video file b returned by the media server through the CDN. Please note that in this case, the second terminal still displays the content of the first barrage comment. That is, in this case, the second terminal displays the video image of the first free viewpoint video at viewpoint B and the first barrage comment.

[0138] In this example, the playback instant T2 is any instant in the time period from the playback instant T3 (the instant after the first free viewpoint video is switched to viewpoint B) to the playback instant T1 plus the duration ΔT of the first barrage comment. For example, the playback instant T5 can be any playback instant between 20:15 and 20:25.

[0139] It may be noted that in this example, the second terminal first requests the barrage comment data of the first free viewpoint video at the corresponding playback moment from the barrage comment server, and then requests the video file of the first free viewpoint video at viewpoint B from the media server, i.e., first executes step S506b and then executes step S506a.

[0140] S509: When the second terminal plays back the video image of the playback moment T2 of the first free viewpoint video from viewpoint B, the second terminal detects that the user has performed operation 2 on the first barrage comment, in this case, the second terminal sends a viewpoint switching request to the media server to request a video file c of the playback moment T4 of the first free viewpoint video from viewpoint A.

[0141] Operation 2 is, for example, the user's finger operating a virtual control corresponding to the first danmaku comment on the display of the second terminal (which is a touch screen), or the user operating a button on a remote control to select a control corresponding to the first danmaku comment displayed on the second terminal.

[0142] The playback instant T4 is related to the playback instant T1. For example, the playback instant T4 is equal to the playback instant T1 or the playback instant T4 is equal to the playback instant T1 minus a preset time offset value 3 (for example, 5 seconds or 10 seconds). Alternatively, the playback instant T4 is related to the playback instant T2. For example, the playback instant T4 is equal to the playback instant T2 or the playback instant T4 is equal to the playback instant T2 minus a preset time offset value 4.

[0143] For the sake of explanation, an example in which the second terminal is a mobile phone is used. FIG. 10(1) shows a playback interface 1001 displayed by a mobile phone. The playback interface 1001 includes a video image of a first free viewpoint video at viewpoint B (e.g., 80 degrees) and a corresponding danmaku comment (including a control 1002 corresponding to the first danmaku comment). For example, the playback interface 1001 is a video image at a playback moment 20:15 at a viewpoint of 80 degrees. When the mobile phone detects that a user operates the control 1002 (e.g., performs a tap, double tap, touch and hold, or gravity press) (operation 2), the mobile phone determines a first danmaku comment based on the control 1002, and determines the viewpoint of the first danmaku comment (viewpoint A, e.g., 70 degrees) as the viewpoint after switching.

[0144] In addition, the mobile phone further determines the playback instant after the viewpoint is switched. In some examples, the mobile phone determines the playback instant after the viewpoint is switched, i.e., playback instant T4, based on the occurrence instant (playback instant T1) of the first barrage comment. For example, the playback instant T4 is equal to the playback instant T1. For example, as shown in FIG. 10(1), the mobile phone detects that the user taps the control 1002 corresponding to the first barrage comment in the playback interface 1001 at the playback instant T2 (for example, at 20:15), the mobile phone switches from the viewpoint B to the viewpoint A, and the mobile phone displays the playback interface 1003 in FIG. 10(2). The occurrence instant (playback instant T1) of the first barrage comment is 20:10, and therefore the playback instant T4 is 20:10. With reference to the description of the playback instant T1 in step S503, it can be seen that the playback instant T1 is adjusted to the start instant of the highlight (for example, the playback instant T1 is equal to the sending instant of the first barrage comment minus the preset time offset value 1). Therefore, when the playback instant T4 is equal to the playback instant T1, it means that the mobile phone starts playing the highlight after switching to the viewpoint A. As another example, the playback instant T4 is equal to the playback instant T1 minus the preset time offset value 3. With reference to the description of the playback instant T1 in step S503, it can be seen that the playback instant T1 is not adjusted to the start instant of the highlight (for example, the playback instant T1 is equal to the sending instant of the first barrage comment). In this case, the moment when the user sends the barrage comment is after the highlight starts. Therefore, the mobile phone can adjust the playback instant T4 (the playback instant T4 is equal to the playback instant T1 minus the preset time offset value 3) so that the playback starts from the actual highlight moment after the mobile phone switches the viewpoint. Alternatively, the playback instant T1 is adjusted to the start instant of the highlight (e.g., the playback instant T1 is equal to the sending instant of the first barrage comment minus a preset time offset value 1), and the mobile phone can still start playback from an earlier playback instant before the highlight, thus further ensuring that the user does not miss the highlight moment.For example, the playback instant T1 is 20:10 and the playback instant T4 is 20:08. As another example, the playback instant T4 can alternatively be after the playback instant T1. For example, the playback instant T1 is 20:10 and the playback instant T4 is 20:15.

[0145] In some other examples, the mobile phone determines the playback instant after the viewpoint is switched, i.e., playback instant T4, based on the playback instant at which the viewpoint is switched (playback instant T2). For example, playback instant T4 is equal to playback instant T2. As shown in FIG. 10(1), the mobile phone detects that at playback instant T2 (e.g., at 20:15), the user taps the control 1002 corresponding to the first barrage comment in the playback interface 1001, so the mobile phone switches from viewpoint B to viewpoint A, and the mobile phone displays the playback interface 1004 in FIG. 10(3). The playback instant T4 is the same as playback instant T2 (e.g., Regeneration moment T4 It may be noted that the playback instant T4 is equal to the playback instant T2 minus a preset time offset value 4. For example, the playback instant T2 is 20:15 and the playback instant T4 is 20:08. Of course, in some other examples, the playback instant T4 may alternatively be after the playback instant T2. For example, the playback instant T2 is 20:15 and the playback instant T4 is 20:18.

[0146] After the mobile phone determines the viewpoint to be switched to and the playback instant after the viewpoint is switched, the mobile phone sends a viewpoint switching request to the media server to request a video file c of the playback instant T4 of the first free viewpoint video at viewpoint A.

[0147] The viewpoint switching request includes an identifier of the first free viewpoint video and viewpoint information after switching (viewpoint A). Optionally, the viewpoint switching request further includes a playback instant after the viewpoint is switched, that is, a playback instant T4.

[0148] In some other embodiments, the mobile phone may not determine the playing moment T4, but the media server determines the playing moment T4. For example, the request sent by the mobile phone to the media server further includes the playing moment T1 or the playing moment T2. In this case, after receiving the switching request, the media server determines the playing moment T4 based on the playing moment T1 or the playing moment T2, and returns the video file c of the playing moment T4 to the second terminal.

[0149] S510: The media server returns the video file c to the second terminal.

[0150] S511: The second terminal plays back the video image at the playback moment T4 of the first free viewpoint video at the viewpoint A based on the video file c.

[0151] In this way, the second terminal switches from playing the video image of the first free viewpoint video at viewpoint B to playing the video image of the first free viewpoint video at viewpoint A. It may be noted that in the viewpoint switching solution provided in this embodiment of the present application, before the viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint B from the media server through the CDN. After the viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint A from the media server through the CDN. That is, in the viewpoint switching process, the second terminal does not request the video file of the first free viewpoint video at another viewpoint other than viewpoint A and viewpoint B from the media server.

[0152] In the prior art, in a solution in which a user switches viewpoints by sliding a finger on a display or by operating a remote control, a terminal requests video files of all viewpoints from viewpoint B to viewpoint A of a first free viewpoint video from a media server. For example, when a user instructs to switch from 60 degrees to 120 degrees, the terminal requests video files of the first free viewpoint video at all viewpoints between 60 degrees and 120 degrees (e.g., 70 degrees, 80 degrees, 90 degrees, 100 degrees, and 110 degrees), and displays video images corresponding to the viewpoints by linearly switching for each angle. However, in this embodiment of the present application, the terminal requests only the video file of the first free viewpoint video at 120 degrees from a media server, and displays the video image at 120 degrees.

[0153] In the viewpoint switching solution provided in this embodiment of the present application, compared with that of the prior art, the second terminal no longer requests the video file of the first free viewpoint video at another viewpoint other than viewpoint A and viewpoint B from the media server, thus reducing traffic. In addition, the terminal only needs to receive the video file of the first free viewpoint video at viewpoint B and process the video file of the first free viewpoint video at viewpoint B, thus reducing the video processing work of the terminal, improving the processing efficiency of the terminal, accelerating the response speed of the terminal, and reducing the power consumption of the terminal.

[0154] In addition, when the viewpoint switching solution in the prior art is used, in the process of switching from viewpoint B to viewpoint A, the second terminal needs to switch and display the video images at all viewpoints between viewpoint B and viewpoint A. When a user watches a moving video image, frequent viewpoint switching is performed on the moving video image, which is likely to cause dizziness to the user. As a result, the video viewing experience is deteriorated. However, in the viewpoint switching solution provided in this embodiment of the present application, frequent viewpoint switching is not performed, which is helpful in avoiding dizziness to the user and improving the video viewing experience.

[0155] In some other embodiments, in the process of switching from the video image of the first free viewpoint video at viewpoint B to the video image of the first free viewpoint video at viewpoint A, some transition images may be added. For example, during switching, a transition image frame is generated based on the last N frames (N is a natural number greater than or equal to 1) of the image frame of the first free viewpoint video at viewpoint B displayed by the terminal and the first M frames (M is a natural number greater than or equal to 1) of the image frame of the first free viewpoint video at viewpoint A to be displayed by the terminal, and the transition image frame is inserted between the last frame of the image frame of the first free viewpoint video at viewpoint B and the first frame of the image frame of the first free viewpoint video at viewpoint A, so that the viewpoint switching of the first free viewpoint video is more natural and realistic. As another example, during switching, image frames at some viewpoints from viewpoint B to viewpoint A are selected and inserted between the last frame of the image frame of the first free viewpoint video at viewpoint B and the first frame of the image frame of the first free viewpoint video at viewpoint A, so that the viewpoint switching of the first free viewpoint video is more natural and realistic. The transition images in the viewpoint switching process are Limit Not determined.

[0156] It may be further noted that the first terminal that transmits the first barrage comment and the second terminal that switches the viewpoint based on the first barrage comment interact with the barrage comment server separately. Therefore, it is not limited whether the first terminal and the second terminal are the same type of terminal, whether the first terminal and the second terminal run the same operating system, and whether the first terminal and the second terminal run the same free viewpoint video playback client / application. In other words, the technical solution provided in this embodiment of the present application can implement viewpoint switching between terminals, between execution platforms, and between applications, and is widely applicable.

[0157] In conclusion, in this embodiment of the present application, when a user sends a first danmaku comment by using a first terminal, the first terminal automatically establishes a correspondence relationship between the first danmaku comment and the viewpoint of the video currently being viewed by the user. After that, when another user views the same free viewpoint video by using a second terminal and sees the first danmaku comment, the user can quickly switch to the viewpoint corresponding to the first danmaku comment by operating the first danmaku comment. Therefore, when the target object has a vast shooting scene with a limited viewpoint range on the client / application and even a blind area, the user can still grasp the viewpoint with highlights from the content of the danmaku comment displayed on the video image. The highlights may be video clips that are highly praised by most users, or may be video clips that viewers are interested in, etc. Furthermore, the user can quickly switch to the viewpoint corresponding to the specific danmaku comment by operating the specific danmaku comment to view the highlights of the viewpoint.

[0158] In some other embodiments, the media server or the second terminal can automatically identify the content of the first free viewpoint video, for example, identify the first free viewpoint video as a different clip based on one or more of its plot, scene, image capture method (e.g., full body capture), and person, etc. For example, the playback instant range of clip 1 is 15:00-25:00.

[0159] In response to an operation performed by a user on the first danmaku comment at playback instant T2 (20:15), the second terminal switches to viewpoint A corresponding to the first danmaku comment and plays a video image of instant T4, where playback instant T4 may be the start instant (15:00) of clip 1, or may be the occurrence instant T1 (20:10) of the first danmaku comment, or may be the current instant T2 (20:15).

[0160] In some embodiments, the playback instant T4 may be the start instant (15:00) of clip 1, and the playback instant range of clip 1 is 15:00-25:00. After the second terminal finishes playing the clip at viewpoint A, i.e., at the end instant 25:00 of the clip, the second terminal can automatically switch back to the original viewpoint (e.g., viewpoint B) and continue to play the clip after instant 25:00. Of course, in some other examples, the second terminal can alternatively ask the user whether to switch back to the original viewpoint and continue to play the clip after instant 25:00. After receiving the user's consent, it switches back to the original viewpoint and continues to play the clip after 25:00. In some other embodiments, it may not switch back to the original viewpoint B, but continues to play the clip after time 25:00 in viewpoint A.

[0161] Of course, in some other embodiments, in the scenario shown in FIG. 10(2) or FIG. 10(3), after the second terminal plays the video clip corresponding to the first danmaku comment, the first danmaku comment appears at 20:10 and lasts for 15 seconds, i.e., the second terminal plays the video clip corresponding to 20:10 to 20:25, and at 25:00, the second terminal does not switch back to the original perspective of 70 degrees, but can continue to play the clip after 20:25, when the first danmaku comment ends, at the perspective of 80 degrees.

[0162] Of course, in some other embodiments, in the scenario shown in FIG. 10(2) or FIG. 10(3), after the display of the first barrage comment has finished, the terminal may alternatively automatically switch back to the original perspective at 20:25 and continue playing the clip after 20:25.

[0163] In the above embodiment, the user selects to switch to a viewpoint that the user is interested in for viewing based on the barrage comment content. In some other embodiments, the barrage comment server may alternatively recommend a viewpoint where a highlight is located for the user to select based on the collected barrage comment data. For example, in the process of playing a free viewpoint video, the number of barrage comments corresponding to a video clip may reflect the popularity and highlight degree of the video clip. If the number of barrage comments corresponding to a video clip is greater than a preset threshold, the video clip may be regarded as a highlight. The solution provided in this embodiment will be described in detail below.

[0164] In this embodiment, each user in the entire network can choose to watch the video image of the first free viewpoint video at any viewpoint, and can also switch the viewpoint of the first free viewpoint video at will. In the process of playing the first free viewpoint video, if a user thinks that a certain video clip is great, the user can input related barrage comment content. The terminal transmits the barrage comment content input by the user and other data of the barrage comment to the barrage comment server. It should be noted that the data of the barrage comment transmitted by the terminal to the barrage comment server includes information such as the barrage comment content, the identifier of the first free viewpoint video, the viewpoint of the barrage comment, and the moment of occurrence of the barrage comment. Of course, the data of the barrage comment may further include other content. Please refer to the related content in Table 1. In addition, for the process of the terminal transmitting the barrage comment data to the server, please refer to the related content in steps S500 to S505 above. Details will not be described again here.

[0165] The following describes in detail the process in which the barrage comment server calculates a recommended viewpoint list based on collected barrage comment data, and another user selects to switch viewpoints by using the recommended viewpoint list.

[0166] FIG. 11 is a schematic flowchart of another viewpoint switching method for free viewpoint video according to an embodiment of the present application, which includes the following steps.

[0167] S1101: The danmaku comment server determines a recommended viewpoint corresponding to each playback instant based on the danmaku comment data of the first free viewpoint video.

[0168] In some embodiments, the danmaku comment server periodically (e.g., at intervals of one minute) calculates a recommended viewpoint corresponding to each playback instant in the first free viewpoint video. In a particular implementation, the playback instant of the first free viewpoint video is divided into a plurality of time periods (e.g., the length of each time period is 10 to 15 seconds). Then, by using the time periods as units, statistical information is collected based on different viewpoints to calculate a recommended viewpoint corresponding to each time period. For example, the time period corresponding to each danmaku comment is determined based on the occurrence instant corresponding to each danmaku comment in the danmaku comment data. Then, for each time period, statistical information on the number of danmaku comments corresponding to each viewpoint in the time period is collected based on the viewpoints corresponding to the danmaku comments in the danmaku comment data. If the number of danmaku comments of one viewpoint or multiple viewpoints in the time period is greater than or equal to a preset threshold, the one viewpoint or multiple viewpoints are determined as recommended viewpoints. In an example, if the quantity of barrage comments in multiple viewpoints in a certain time period are all greater than or equal to a preset threshold, the first R viewpoints may be selected as recommended viewpoints in descending order of the quantity of barrage comments corresponding to the viewpoints, where R is an integer greater than 0. For example, R=1 or R=5, or the user can manually configure the value of R. It should be noted that the calculation steps in this specification are merely examples of a specific implementation form and do not constitute limitations on the method for calculating the recommended viewpoints in each time period.

[0169] For example, Table 2 shows some examples of danmaku comment data stored in the danmaku comment server. Based on the occurrence moment corresponding to each danmaku comment, it can be determined that danmaku comment 1 to danmaku comment 4 correspond to time period 1 (10:00 to 10:15). Furthermore, for time period 1, it can be calculated that the number of danmaku comments corresponding to viewpoint 1 is 3, and the number of danmaku comments corresponding to viewpoint 2 is 1. The number of danmaku comments corresponding to viewpoint 1 and the number of danmaku comments corresponding to viewpoint 2 are less than a preset threshold, and no viewpoint is recommended in time period 1.

[0170] Similarly, danmaku comments 5 to 8 correspond to time period 2 (10:16 to 10:30). Furthermore, for time period 2, the number of danmaku comments corresponding to viewpoint 1 can be calculated to be 2, the number of danmaku comments corresponding to viewpoint 2 to be 1, and the number of danmaku comments corresponding to viewpoint 3 to be 1. The number of danmaku comments corresponding to viewpoint 1, the number of danmaku comments corresponding to viewpoint 2, and the number of danmaku comments corresponding to viewpoint 3 are all below the preset threshold, and no viewpoint is recommended in time period 2.

[0171] Similarly, danmaku comments 100 to 201 correspond to time period 3 (20:16 to 20:30). Furthermore, for time period 3, the number of danmaku comments corresponding to viewpoint 3 can be calculated to be 200, and the number of danmaku comments corresponding to viewpoint 1 can be calculated to be 1. If the number of danmaku comments corresponding to viewpoint 3 is greater than or equal to a preset threshold (e.g., 150), it is determined that the recommended viewpoint corresponding to time period 3 is viewpoint 3. [Table 2]

[0172] In some other embodiments, the barrage comment server may further determine the recommended viewpoint by referring to the specific content of the barrage comment and the number of the barrage comments. For example, the content of some barrage comments indicates that the video image of the viewpoint is excellent. For example, the content of the barrage comments is "great" and "this is recommended", etc. Then, the viewpoint is determined as the recommended viewpoint by referring to the condition that the number of barrage comments of the viewpoint meets a specific condition (for example, the number is greater than a preset threshold). As another example, the content of some barrage comments indicates that the video image of the viewpoint is not excellent. For example, if the content of the barrage comments is "too bad" or "skip this", etc., the viewpoint may be excluded as a recommended viewpoint. In conclusion, the method for determining the recommended viewpoint is not particularly limited in the embodiments of the present application.

[0173] S1102: The barrage comment server stores a recommended viewpoint corresponding to each playback moment.

[0174] S1103: The second terminal requests the video file d of the first free viewpoint video at the playback moment T5 in the viewpoint C from the media server.

[0175] The playback instant T5 can be any playback instant of the first free viewpoint video.

[0176] S1104: The media server returns the video file d to the second terminal.

[0177] S1105: The second terminal requests the danmaku comment server for a recommended viewpoint at a playback moment T5 of the first free viewpoint video.

[0178] S1106: The barrage comment server returns a recommended viewpoint for the playback moment T5 of the first free viewpoint video to the second terminal. The recommended viewpoint for the playback moment T5 of the first free viewpoint video includes viewpoint D.

[0179] It should be noted that step S1105 can be performed before, after, or simultaneously with step S1103. The order of performing steps S1105 and S1103 is not limited in the embodiment of the present application.

[0180] It should be noted that in some embodiments, when the second terminal enables the danmaku comment function, the second terminal not only requests the danmaku comment server for danmaku comment data corresponding to each playback moment based on the playback progress of the first free viewpoint video, but also requests the danmaku comment server for a recommended viewpoint corresponding to each playback moment. In some other embodiments, when the second terminal does not enable the danmaku comment function but enables the viewpoint recommendation function, the second terminal requests the danmaku comment server for a recommended viewpoint corresponding to each playback moment based on the playback progress of the first free viewpoint video. In some other embodiments, when the second terminal starts playing the first free viewpoint video, the second terminal requests the danmaku comment server for recommended viewpoints corresponding to all playback moments. After that, if the recommended viewpoint calculated by the danmaku comment server is changed, the second terminal may be notified to obtain the latest recommended viewpoint. If the recommended viewpoint calculated by the danmaku comment server is not changed or is changed only slightly, the second terminal may not be notified to obtain the latest recommended viewpoint. In this way, the second terminal may then display recommended viewpoints corresponding to corresponding playback moments based on the recommended viewpoints corresponding to all playback moments requested when playback was started, the updated recommended viewpoints obtained thereafter, and the playback progress of the first free viewpoint video.

[0181] S1107: The second terminal plays the video image of the first free viewpoint video at the playback moment T5 at the viewpoint C based on the video file d, and the second terminal further displays the recommended viewpoint at the playback moment T5.

[0182] For example, the second terminal is a mobile phone. FIG. 12(1) shows a playback interface 1201 of the first free viewpoint video displayed by the mobile phone. The playback interface 1201 includes a recommended viewpoint list 1202, and the recommended viewpoint list 1202 includes a recommended viewpoint. There are one or more recommended viewpoints. In some examples, when there are multiple recommended viewpoints corresponding to the playback instant at which the danmaku comment server collects statistical information, the recommended viewpoint list 1202 may include multiple recommended viewpoints. Optionally, the multiple recommended viewpoints may be sorted in descending order of the number of danmaku comments corresponding to the recommended viewpoints, and the recommended viewpoints with a larger number of danmaku comments are preferentially recommended to the user, or the recommended viewpoints with a larger number of danmaku comments are ranked first. In some other examples, the recommended viewpoint list 1202 further includes a recommendation index used to represent the priority of the recommended viewpoints. The priority of the recommended viewpoints may be related to the order of the number of danmaku comments corresponding to the recommended viewpoints. In some other examples, the recommended viewpoint list 1202 further includes the quantity of the recommended viewpoints, etc.

[0183] S1108: The second terminal detects an operation by the user to select viewpoint D, and the second terminal requests a video file e of the first free viewpoint video at playback moment T6 in viewpoint D from the media server.

[0184] In some embodiments, the playback instant T6 is relative to the playback instant T5, for example, the playback instant T6 is equal to the playback instant T5, or the playback instant T6 is equal to the playback instant T5 minus a preset time offset value of 5 (for example, the preset time offset value is 5 seconds).

[0185] For example, the second terminal is a mobile phone. In the playback interface 1201 shown in Fig. 12(1), the mobile phone displays a video image of a playback moment T5 (e.g., 20:16) at a viewpoint C (e.g., 70 degrees). When the mobile phone detects a user's operation to select a viewpoint D (e.g., 60 degrees) in the recommended viewpoint list 1202 in the playback interface 1201 shown in Fig. 12(1), the mobile phone requests a video file e of a playback moment T6 (e.g., 20:16 or 20:11) of the first free viewpoint video at viewpoint D from the media server.

[0186] In some other embodiments, the playback instant T6 is associated with the clip corresponding to the playback instant T5. Specifically, the media server or the second terminal can automatically identify the content of the first free viewpoint video, for example, identify the first free viewpoint video as a different clip based on one or more of the plot, scene, image shooting method (e.g., full body shooting), and person of the video. The playback instant T6 is the playback start instant of the clip corresponding to the playback instant T5. For example, the playback instant is 20:16, and the playback instant range of clip 2 is, for example, 20:00-25:00. It can be understood that the playback instant T5 is 20:16 and corresponds to clip 2. Therefore, the playback instant T6 can be 20:00.

[0187] S1109: The media server returns the video file e to the second terminal.

[0188] S1110: The second terminal plays back the video image at the playback moment T6 of the first free viewpoint video at the viewpoint D based on the video file e.

[0189] For example, in the playback interface 1203 shown in FIG.

[0190] In this way, the second terminal switches from playing the video image of the first free viewpoint video at viewpoint C to playing the video image of the first free viewpoint video at viewpoint D. It may be noted that in the viewpoint switching solution provided in this embodiment of the present application, before the viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint C from the media server through the CDN. After the viewpoint switching, the second terminal requests the video file of the first free viewpoint video at viewpoint D from the media server through the CDN. That is, in the viewpoint switching process, the second terminal does not request the video file of the first free viewpoint video at another viewpoint other than viewpoint C and viewpoint D from the media server.

[0191] In addition, if the second terminal does not enable the danmaku comment function or does not meet the conditions for displaying danmaku comments, the second terminal may alternatively display a list of recommended viewpoints, and the user may display the danmaku comments based on the recommended viewpoint list.

[0192] For other contents of this embodiment, please refer to the above description, and the details will not be described again here.

[0193] FIG. 13 is a schematic flowchart of another viewpoint switching method for free viewpoint video according to an embodiment of the present application, which includes the following steps.

[0194] S1310: The terminal displays a first interface, where the first interface has a video image of a first free viewpoint video at a first viewpoint and a first barrage comment, where the first barrage comment corresponds to a second viewpoint of the first free viewpoint video.

[0195] Optionally, the first barrage of comments in the first interface further includes a second perspective.

[0196] In some examples, the terminal displays the first interface at the first playback instant, i.e., the first interface has a video image of the first free viewpoint video at the first viewpoint at the first playback instant.

[0197] In some other examples, before displaying the first interface at the first playback instant, the terminal requests danmaku comment data corresponding to the first playback instant of the first free viewpoint video from a danmaku comment server, where the danmaku comment data corresponding to the first playback instant includes data of a first danmaku comment, and the data of the first danmaku comment includes a second viewpoint.

[0198] S1320: When the terminal detects a first operation performed by a user on a first barrage comment, it displays a second interface, where the second interface includes a video image of the first free viewpoint video at a second viewpoint.

[0199] In some examples, when the terminal detects a first operation performed by the user on the first danmaku comment, the terminal displays the second interface at a second playback instant. The second playback instant is related to the first playback instant. For example, the second playback instant is equal to the first playback instant, or the second playback instant is equal to the first playback instant minus a first preset time offset value. As another example, the first playback instant is related to the occurrence instant of the first danmaku comment. The second playback instant is also related to the occurrence instant of the first danmaku comment. In this case, the second playback instant is equal to the occurrence instant of the first danmaku comment, or the second playback instant is equal to the occurrence instant of the first danmaku comment minus a second preset time offset value.

[0200] In some other examples, when the terminal detects a first operation performed by the user on the first barrage comment, the terminal sends a switching request to the media server. The media server returns a first video file to the terminal, where the first video file is a video file corresponding to a second playback instant of the first free viewpoint video in a second viewpoint. The terminal displays a video image of the second playback instant of the first free viewpoint video in the second viewpoint based on the first video file.

[0201] Optionally, the second interface includes the first barrage of comments.

[0202] Optionally, the terminal plays a third interface at a third playback instant, where the third playback instant is after a display time of the first barrage comment, and the third interface has a video image of the first free viewpoint video at the second viewpoint, and the third interface does not have the first barrage comment.

[0203] In some other examples, the terminal displays a fourth interface at a fourth playback instant, where the fourth interface has a video image of the first free viewpoint video at the first viewpoint and the recommended viewpoint, where the recommended viewpoint includes the fourth viewpoint. When the terminal detects a third operation by the user to select the fourth viewpoint, the terminal displays a video image of a fifth playback instant of the first free viewpoint video at the fourth viewpoint, where the fifth playback instant is related to the fourth playback instant. For example, the fifth playback instant is equal to the fourth playback instant, or the fifth playback instant is equal to the fourth playback instant minus a fourth preset time offset value.

[0204] Optionally, before displaying the fourth interface at the fourth playback instant, the terminal requests a second video file corresponding to the fourth playback instant of the first free viewpoint video at the first viewpoint from the media server, and requests a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from the danmaku comment server, where the recommended viewpoint corresponding to the fourth playback instant is determined by the danmaku comment server based on the quantity of danmaku comments corresponding to the fourth playback instant of the first free viewpoint video.

[0205] The embodiment of the present application further provides a chip system. As shown in FIG. 14, the chip system includes at least one processor 1301 and at least one interface circuit 1302. The processor 1301 and the interface circuit 1302 may be interconnected through a line. For example, the interface circuit 1302 may be configured to receive a signal from another device (e.g., a memory in the terminal 100). As another example, the interface circuit 1302 may be configured to transmit a signal to another device (e.g., the processor 1301). For example, the interface circuit 1302 may read an instruction stored in a memory and transmit the instruction to the processor 1301. When the instruction is executed by the processor 1301, the terminal may be able to execute the steps executed by the terminal 100 (e.g., a mobile phone) in the above-mentioned embodiment. Of course, the chip system may further include another discrete device. This is not particularly limited in the embodiment of the present application.

[0206] An embodiment of the present application further provides an apparatus. The apparatus is included in a terminal, and the apparatus has a function of implementing the behavior of the terminal in any manner in the above-mentioned embodiments. The function may be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above-mentioned function, such as a detection module or unit, a display module or unit, a determination module or unit, and a calculation module or unit.

[0207] An embodiment of the present application further provides a computer-readable storage medium containing computer instructions, which, when executed on a terminal, enable the terminal to perform any of the methods in the above-mentioned embodiments.

[0208] An embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to perform any of the methods in the above-mentioned embodiments.

[0209] An embodiment of the present application further provides a graphical user interface on a terminal, the terminal including a display, a camera, a memory, and one or more processors, the one or more processors being configured to execute one or more computer programs stored in the memory; the graphical user interface includes a graphical user interface displayed when the terminal executes any of the methods in the above-mentioned embodiments.

[0210] It can be understood that in order to implement the above-mentioned functions, the terminal etc. includes corresponding hardware structures and / or software modules for performing the functions. Those skilled in the art should easily recognize that the units, algorithms, and steps can be implemented by hardware, or a combination of hardware, and computer software in the embodiments of the present application, in combination with the examples described in the embodiments disclosed herein. Whether the functions are performed by hardware or hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but it is understood that the implementation form is not limited to the above. Original Application should not be considered as going beyond the scope of the embodiments.

[0211] In the embodiment of the present application, the terminal etc. may be divided into functional modules based on the above-mentioned method examples. For example, each functional module may be obtained through division based on each corresponding function, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. Original Application It should be noted that in this embodiment, the module division is an example and is merely a logical function division. In actual implementation, other division styles may be used.

[0212] The above description of the implementation form allows those skilled in the art to clearly understand that for the purpose of convenient and concise description, the above-mentioned division of functional modules is taken as an example for explanation. In practical application, the above-mentioned functions can be allocated to different modules and implemented according to requirements, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above-mentioned functions. For the detailed work process of the above-mentioned system, device and unit, please refer to the corresponding processing in the above-mentioned method embodiment, and the details will not be described again here.

[0213] The functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0214] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such understanding, the technical solutions in the embodiments of the present application may be essentially implemented in the form of a software product, or the part that contributes to the prior art, or all or part of the technical solutions. The computer software product is stored in a storage medium and includes a plurality of instructions for instructing a computer device (which may be a personal computer, a server, a network device, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The above-mentioned storage medium includes any medium that can store program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0215] The above description is merely a specific implementation form of the present application, and is not intended to limit the protection scope of the present application. Any variation or replacement within the technical scope disclosed in this application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. (Other possible items) (Item 1) 1. A viewpoint switching method for a free viewpoint video, comprising: A terminal displays a first interface, where the first interface has a video image of a first free viewpoint video at a first viewpoint and a first barrage comment, the first barrage comment corresponding to a second viewpoint of the first free viewpoint video; and displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment, where the second interface has a video image of the first free viewpoint video at the second viewpoint. A method comprising: (Item 2) The step of displaying the first interface by the terminal includes: the terminal displaying the first interface at a first playback instant; having; The step of displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment includes: displaying the second interface at a second playback instant when the terminal detects the first operation performed by the user on the first barrage comment, where the second playback instant is related to the first playback instant. 2. The method according to item 1, comprising the steps of: (Item 3) The second playback instant is related to the first playback instant by: The second playback instant is equal to the first playback instant, or the second playback instant is equal to the first playback instant minus a first preset time offset value; or the second playback instant is equal to the occurrence instant of the first barrage comment, or the second playback instant is equal to the occurrence instant of the first barrage comment minus a second preset time offset value. The method according to item 2, comprising: (Item 4) The method according to any one of items 1 to 3, wherein the first barrage comments further include the second viewpoint. (Item 5) When the terminal detects the first operation performed by the user on the first barrage comment, the step of displaying the second interface at a second playback moment includes: sending a switching request to a media server when the terminal detects the first operation performed by the user on the first barrage comment; and receiving a first video file returned by the media server by the terminal, where the first video file is a video file corresponding to the second playback instant of the first free viewpoint video in the second viewpoint; and a step of displaying the video image of the second playback moment of the first free viewpoint video at the second viewpoint based on the first video file by the terminal; The method according to any one of Items 2 to 4, comprising: (Item 6) Prior to the terminal displaying the first interface at a first playback instant, the method further comprises: A step in which the terminal requests barrage comment data corresponding to the first playback instant of the first free viewpoint video from a barrage comment server, where the barrage comment data corresponding to the first playback instant has data of the first barrage comment, and the data of the first barrage comment includes the second viewpoint. The method according to any one of items 2 to 5, further comprising: (Item 7) After the terminal displaying the second interface at a second playback instant, the method further comprises: A step of receiving a second operation of transmitting a second barrage comment by the terminal, and a step of transmitting data of the second barrage comment by the terminal to a barrage comment server, where the data of the second barrage comment includes a content of the second barrage comment and a third viewpoint, and the third viewpoint is a viewpoint corresponding to the first free viewpoint video during the transmission of the second barrage comment. The method according to any one of items 2 to 6, further comprising: (Item 8) 8. The method of claim 7, wherein the data of the second barrage comment further includes a generation instant of the second barrage comment, the generation instant of the second barrage comment being equal to the second playback instant, or the generation instant of the second barrage comment being equal to the second playback instant minus a third preset time offset value. (Item 9) The second interface further includes the first barrage comment. The method according to any one of items 1 to 8. (Item 10) After the terminal displaying the second interface at a second playback instant, the method further comprises: The terminal displays a third interface at a third playback instant, where the third playback instant is after a display time of the first barrage comment, the third interface has the video image of the first free viewpoint video at the second viewpoint, and the third interface does not have the first barrage comment. The method according to any one of items 2 to 9, further comprising: (Item 11) The method includes: displaying a fourth interface at a fourth playback instant, where the fourth interface has the video image of the first free viewpoint video at the first viewpoint and a recommended viewpoint, the recommended viewpoint including a fourth viewpoint; and and displaying a video image of a fifth playback instant of the first free viewpoint video in the fourth viewpoint when the terminal detects a third operation of selecting the fourth viewpoint by the user, where the fifth playback instant is related to the fourth playback instant. The method according to any one of items 1 to 10, further comprising: (Item 12) The fifth instant of play is related to the fourth instant of play in that: the fifth playback instant being equal to the fourth playback instant, or the fifth playback instant being equal to the fourth playback instant minus a fourth preset time offset value. Item 12. The method according to item 11, comprising: (Item 13) Before the terminal displays a fourth interface at a fourth playback instant, the method further comprises: A step of the terminal requesting a second video file corresponding to the fourth playback moment of the first free viewpoint video in the first viewpoint from a media server, and a step of requesting a recommended viewpoint corresponding to the fourth playback moment of the first free viewpoint video from a barrage comment server, where the recommended viewpoint corresponding to the fourth playback moment is determined by the barrage comment server based on a quantity of barrage comments corresponding to the fourth playback moment of the first free viewpoint video. 13. The method according to item 11 or 12, further comprising: (Item 14) A terminal comprising a processor, a memory, and a touch screen, the memory and the touch screen are coupled to the processor, the memory is configured to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions in the memory, the terminal becomes capable of executing the viewpoint switching method for free viewpoint video described in any one of items 1 to 13. (Item 15) A computer-readable storage medium comprising computer instructions, the computer instructions being executed on a terminal to enable the terminal to execute the viewpoint switching method for free viewpoint video described in any one of items 1 to 13.

Claims

1. 1. A viewpoint switching method for a free viewpoint video, comprising: A terminal displays a first interface, where the first interface has a video image of a first free viewpoint video at a first viewpoint and a first barrage comment, the first barrage comment corresponding to a second viewpoint of the first free viewpoint video; and displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment, where the second interface has a video image of the first free viewpoint video at the second viewpoint. A method comprising:

2. The step of displaying the first interface by the terminal includes: the terminal displays the first interface at a first playback instant. having The step of displaying a second interface when the terminal detects a first operation performed by a user on the first barrage comment includes: When the terminal detects the first operation performed by the user on the first barrage comment, displaying the second interface at a second playback instant, where the second playback instant is related to the first playback instant.

2. The method of claim 1, comprising:

3. The second playback instant is related to the first playback instant by: The second playback instant is equal to the first playback instant, or the second playback instant is equal to the first playback instant minus a first preset time offset value; or the second playback instant is equal to the occurrence instant of the first barrage comment, or the second playback instant is equal to the occurrence instant of the first barrage comment minus a second preset time offset value. The method of claim 2 , comprising:

4. The method according to any one of claims 1 to 3, wherein the first barrage of comments further includes the second perspective.

5. When the terminal detects the first operation performed by the user on the first barrage comment, the step of displaying the second interface at a second playback moment includes: sending a switching request to a media server when the terminal detects the first operation performed by the user on the first barrage comment; and receiving a first video file returned by the media server by the terminal, where the first video file is a video file corresponding to the second playback instant of the first free viewpoint video in the second viewpoint; and a step of displaying the video image of the second playback moment of the first free viewpoint video from the second viewpoint based on the first video file by the terminal; The method according to any one of claims 2 to 4, comprising:

6. Prior to the terminal displaying the first interface at a first playback instant, the method further comprises: A step in which the terminal requests danmaku comment data corresponding to the first playback instant of the first free viewpoint video from a danmaku comment server, where the danmaku comment data corresponding to the first playback instant has data of the first danmaku comment, and the data of the first danmaku comment includes the second viewpoint. The method of any one of claims 2 to 5, further comprising:

7. After the terminal displays the second interface at a second playback instant, the method further comprises: a step of receiving a second operation of transmitting a second barrage comment by the terminal, and a step of transmitting data of the second barrage comment by the terminal to a barrage comment server, wherein the data of the second barrage comment includes a content of the second barrage comment and a third viewpoint, and the third viewpoint is a viewpoint corresponding to the first free viewpoint video during the transmission of the second barrage comment. The method of any one of claims 2 to 6, further comprising:

8. 8. The method of claim 7, wherein the data of the second barrage comment further includes a generation instant of the second barrage comment, and the generation instant of the second barrage comment is equal to the second playback instant, or the generation instant of the second barrage comment is equal to the second playback instant minus a third preset time offset value.

9. The second interface further includes the first barrage of comments. The method according to any one of claims 1 to 8.

10. After the terminal displays the second interface at a second playback instant, the method further comprises: A step of the terminal displaying a third interface at a third playback instant, where the third playback instant is after a display time of the first barrage comment, the third interface having the video image of the first free viewpoint video at the second viewpoint, and the third interface not having the first barrage comment. The method of any one of claims 2 to 9, further comprising:

11. The method comprises: displaying a fourth interface at a fourth playback instant, where the fourth interface has the video images of the first free viewpoint video at the first viewpoint and a recommended viewpoint, the recommended viewpoint including a fourth viewpoint; and and displaying a video image of a fifth playback instant of the first free viewpoint video in the fourth viewpoint when the terminal detects a third operation of selecting the fourth viewpoint by the user, where the fifth playback instant is related to the fourth playback instant. The method of any one of claims 1 to 10, further comprising:

12. The fifth playback instant is related to the fourth playback instant in that: the fifth playback instant being equal to the fourth playback instant, or the fifth playback instant being equal to the fourth playback instant minus a fourth preset time offset value. The method of claim 11 , comprising:

13. Before the terminal displays a fourth interface at a fourth playback instant, the method further comprises: A step of the terminal requesting a second video file corresponding to the fourth playback instant of the first free viewpoint video in the first viewpoint from a media server, and a step of requesting a recommended viewpoint corresponding to the fourth playback instant of the first free viewpoint video from a barrage comment server, where the recommended viewpoint corresponding to the fourth playback instant is determined by the barrage comment server based on a quantity of barrage comments corresponding to the fourth playback instant of the first free viewpoint video. The method of claim 11 or 12, further comprising:

14. A terminal comprising a processor, a memory, and a touch screen, the memory and the touch screen being coupled to the processor, the memory being configured to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions in the memory, the terminal is capable of executing the viewpoint switching method for free viewpoint video according to any one of claims 1 to 13.

15. A computer-readable storage medium comprising computer instructions, the computer instructions being capable of causing a terminal to execute the viewpoint switching method for free viewpoint video according to any one of claims 1 to 13, when executed on the terminal.

Images