Method, device and system for displaying barrage comments on free viewpoint video

Abstract

The method, device and system for displaying barrage comments of a free viewpoint video relates to the field of Internet technology, and helps a user to concentrate on viewing a current video image and the contents of the barrage comments related to the current video image, thereby improving the viewing experience. The method includes: a terminal displays a first interface of a first free viewpoint video at a first playback time point, the first interface including a video image of the first free viewpoint video at a first viewpoint and a first barrage comment, and receives a first operation, the first operation being an operation of switching a playback viewpoint of the first free viewpoint video from the first viewpoint to a second viewpoint. In response to receiving the first operation, the terminal displays a second interface of the first free viewpoint video at a first playback time point, the second interface including a video image of the first free viewpoint video at a second viewpoint, and the second interface does not include the first barrage comment.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to Chinese Patent Application No. 202111510462.4, entitled “METHOD FOR DISPLAYING BULLET COMMENT OF FREE VIEWPOINT VIDEO, APPARATUS, AND SYSTEM,” filed with the State Intellectual Property Office of China on December 10, 2021, the entire contents of which are incorporated herein by reference. [Technical field] The present application relates to the field of Internet technology, and in particular to a method, an apparatus and a system for displaying bullet comments of a free viewpoint video. [Background technology]

[0002] A free viewpoint video (FVV) is a special format video file in which a target object is jointly photographed by multiple surrounding cameras to generate a video stream that is encoded into a file such as a special format video file. The generated free viewpoint video is then uploaded to a media server for storage and then distributed to a terminal through a content distribution network (CDN). When viewing the free viewpoint video by using a terminal, a user may freely switch viewpoints through a finger drag operation. Compared with traditional videos, free viewpoint video allows a user to freely select a viewpoint, thereby overcoming the shortcomings of traditional videos in which the viewpoint is determined by a director. The free viewpoint video provides a user with an immersive and interactive playback experience.

[0003] During the playback of the free viewpoint video, the user can enable the barrage comment function. The barrage comment means scrolling, leaving, or displaying a large amount of user comments on the video image in a more action-effect manner, which is currently a popular pastime. It should be noted that the terminal not only displays the video image of the free viewpoint video at a certain viewpoint (e.g., viewpoint A), but also displays the barrage comments at all viewpoints at the current playback time. For example, the terminal not only displays the barrage comment at viewpoint A, but also displays the barrage comment at viewpoint B, the barrage comment at viewpoint C, etc. In this case, in some scenarios, the content of the barrage comment at viewpoint B and / or the barrage comment at viewpoint C may not be consistent with the content of the video image at viewpoint A, resulting in confusion and a confused and unsatisfactory video experience for the user. Summary of the Invention

[0004] The method, device, and system for displaying danmaku comments in free viewpoint video provided in this application improve the viewing experience by helping a user focus on viewing a current video image and the content of the danmaku comments related to the current video image.

[0005] To achieve the above objectives, the embodiments of this application provide the following technical solutions:

[0006] According to a first aspect, there is provided a method for displaying danmaku comments of a free viewpoint video, which is applied to a terminal. The method includes the following: The terminal displays a first interface of a first free viewpoint video at a first playback time point, the first interface including a video image of the first free viewpoint video at a first viewpoint and a first danmaku comment. The terminal receives a first operation, the first operation being an operation of switching a playback viewpoint of the first free viewpoint video from the first viewpoint to a second viewpoint. In response to receiving the first operation, the terminal displays a second interface of the first free viewpoint video at the first playback time point, the second interface including a video image of the first free viewpoint video at a second viewpoint, and the second interface not including the first danmaku comment.

[0007] An offset angle between the first viewpoint and a third viewpoint corresponding to the first danmaku comment is within a first preset angle range. Thus, the first danmaku comment is displayed in the first interface. An offset angle between the second viewpoint and a third viewpoint corresponding to the first danmaku comment is outside the first preset angle range. Thus, the first danmaku comment is not displayed in the second interface.

[0008] It can be seen that when playing the first free viewpoint video, the terminal can display danmaku comments within a first preset angle range (i.e., the visible viewpoint range below) and will not display danmaku comments outside the first preset angle range. Since the content of the danmaku comments within the first preset angle range is usually strongly related to the content of the video image currently viewed by the user at the playback viewpoint, the user can focus on viewing the current video image and the content of the danmaku comments related to the current video image, improving the viewing experience.

[0009] In a possible implementation, the offset angle between the first viewpoint and the third viewpoint corresponding to the first barrage comment is smaller than the offset angle between the second viewpoint and the third viewpoint corresponding to the first barrage comment.

[0010] For example, the terminal may predefine the visible viewpoint range of the danmaku comment, or may receive the visible viewpoint range of the danmaku comment set by the user, as follows: The absolute value of the offset angle between the viewpoint of the danmaku comment and the playback viewpoint of the first free viewpoint video is less than the angle threshold E (e.g., 45 degrees). In this case, the terminal may determine that the visible viewpoint range of the danmaku comment is [playback viewpoint of the first free viewpoint video-angle threshold E, playback viewpoint of the first free viewpoint video+angle threshold E].

[0011] It can be seen that the offset angle between the first viewpoint and the third viewpoint corresponding to the first danmaku comment is within a first preset angle range, and the offset angle between the second viewpoint and the third viewpoint corresponding to the first danmaku comment is outside the first preset angle range. It may be inferred that the offset angle between the first viewpoint and the third viewpoint corresponding to the first danmaku comment is smaller than the offset angle between the second viewpoint and the third viewpoint corresponding to the first danmaku comment.

[0012] In a possible implementation manner, before the terminal receives the first operation, the method further includes: the terminal receives a second operation to pause the playback of the first free viewpoint video.

[0013] In one example, in the process of switching the viewpoint, the terminal automatically pauses the playback of the first free viewpoint video. Thus, before the user performs the first operation, the terminal plays the video image of the first free viewpoint video at the first viewpoint. When detecting that the user performs the first operation at the first playback time point, the terminal automatically pauses the playback of the first free viewpoint video. After the first operation is performed and the playback viewpoint of the first free viewpoint video is switched from the first viewpoint to the second viewpoint, the first free viewpoint video starts to be played from the first playback time point.

[0014] In another example, in the process of switching the viewpoint, the terminal continuously plays the first free viewpoint video. In this case, before performing the first operation, the user may first manually pause the playback of the first free viewpoint video. In this case, the first free viewpoint video stays at the first playback time point. Then, after the user performs the first operation and the terminal switches the playback viewpoint of the first free viewpoint video from the first viewpoint to the second viewpoint, the video image of the first free viewpoint video still stays at the first playback time point.

[0015] Alternatively, after the first operation is performed and the terminal switches the playback viewpoint of the first free viewpoint video from the first viewpoint to the second viewpoint, the playback starts from a playback point after the first playback point. Alternatively, the user may manually drag the playback progress bar to adjust the playback point back to the first playback point.

[0016] In a possible implementation, in a possible implementation, the second interface includes a second barrage comment, and an offset angle between the second viewpoint and a viewpoint corresponding to the second barrage comment is within a first preset angle range.

[0017] In a possible implementation, the sending time of the first barrage comment is relative to the first playback time, and the sending time of the second barrage comment is relative to the first playback time.

[0018] It can be understood that the first barrage comment is played back in the first interface and displayed on the video image at a first playback time point. In this case, the first playback time point is equal to the sending time point of the first barrage comment, or is after the sending time point of the first barrage comment and is within the continuous display duration of the first barrage comment. Similarly, the first playback time point is equal to the sending time point of the second barrage comment, or is after the sending time point of the second barrage comment and is within the continuous display duration of the second barrage comment.

[0019] In a possible implementation, the first interface displayed by the terminal further includes a first control, the first control being associated with a first number of danmaku comments, the first number of danmaku comments being the number of danmaku comments obtained when an offset angle between a viewpoint of the danmaku comments and a playback viewpoint of the terminal is within a second preset angle range, and the second preset angle range does not overlap with the first preset angle range.

[0020] In one example, the first control displays the number of danmaku comments obtained when the offset angle between the viewpoint of the danmaku comments and the playback viewpoint of the terminal is within a second preset angle range.

[0021] In a possible implementation manner, the method further includes: the terminal receives a third operation of switching a viewpoint; the terminal displays a fourth interface in the process of the third operation, and the number of barrage comments indicated by the first control changes in the fourth interface; after the process of the third operation, the terminal displays a second interface of the first free viewpoint video at the second playback point.

[0022] In a possible implementation, the terminal receives a third operation for the first control. In response to receiving the third operation, the terminal displays a third interface, where the third interface includes the first barrage comment.

[0023] It can be seen that when a user is interested in the content of the barrage comments outside the visible viewpoint range, the content of the barrage comments outside the visible viewpoint range may alternatively be expanded by using corresponding controls to meet the user's viewing requirements in multiple scenarios.

[0024] In a possible implementation manner, after the terminal displays the second interface of the first free viewpoint video at the first playback time point in response to receiving the first operation, the method further includes: the terminal receives a fourth operation, the fourth operation being an operation of sending a third barrage comment; the terminal sends data of the third barrage comment to a barrage comment server in response to receiving the fourth operation, and the data of the third barrage comment includes a fourth viewpoint.

[0025] It can be seen that when the user submits the third danmaku comment in the second interface, an association is established between the viewpoint of the third danmaku comment and the playback viewpoint of the first free viewpoint video in the second interface, i.e., the fourth viewpoint. Subsequently, when the third danmaku comment is displayed, it is determined whether or not to present the third danmaku comment based on the relationship between the playback viewpoint of the first free viewpoint video and the viewpoint of the third danmaku comment.

[0026] In a possible implementation, the first interface further includes a fourth barrage comment, the fourth barrage comment corresponding to the first viewpoint.

[0027] In a possible implementation, in the first interface, a first style of the first barrage comment is different from a second style of the fourth barrage comment.

[0028] It can be seen that the user may distinguish the danmaku comments of the first free viewpoint video at the playback viewpoint from the danmaku comments at other viewpoints based on the presentation style of the danmaku comments (e.g., the first style or the second style). In this way, the user will not have confusion or doubt when the video image at the playback viewpoint is inconsistent with the content of the danmaku comments at other viewpoints. This improves the viewing experience of the first free viewpoint video.

[0029] In a possible implementation manner, before the terminal displays a first interface of the first free viewpoint video at a first playback time point, the method includes: the terminal determines to display the first barrage comment in a first style on the first interface based on a first offset angle between the first viewpoint and a third viewpoint corresponding to the first barrage comment.

[0030] In a possible implementation, when the first offset angle is a positive value, the first style is specifically an orientation in which the text direction of the danmaku comments is obtained by rotating the first direction clockwise by the first angle, and the first direction is an orientation parallel to the first bezel of the terminal screen; alternatively, when the first offset angle is a negative value, the first style is specifically an orientation in which the text direction of the danmaku comments is obtained by rotating the first direction counterclockwise by the first angle.

[0031] In a possible implementation manner, the method further includes: the terminal receives a fifth operation of switching viewpoints; the terminal displays a fifth interface in the process of the fifth operation, and the fifth interface and the first interface are the same; after the process of the fifth operation, the terminal displays a sixth interface of the first free viewpoint video at the second playback time point.

[0032] In other words, in the fifth operation process, the video image of the first free viewpoint video displayed by the terminal remains unchanged, and the presentation manner of the barrage comments also remains unchanged.

[0033] In a possible implementation manner, the method further includes: the terminal receives a sixth operation of switching a viewpoint.

[0034] The terminal displays a seventh interface in the sixth operation process, and the seventh interface includes a video image of the first free viewpoint video at a third playback point in time. After the sixth operation process, the terminal displays an eighth interface of the first free viewpoint video at a second playback point in time, and the third playback point in time is after the first playback point in time and before the second playback point in time.

[0035] In other words, in the sixth operation process, the terminal plays the first free viewpoint video continuously, and therefore the playback time point of the first free viewpoint video changes.

[0036] In a possible implementation, the method further includes: the seventh interface further includes a fifth barrage comment, and the first interface does not include the fifth barrage comment.

[0037] In other words, in the sixth operation process, the terminal continuously plays the first free viewpoint video, and the barrage comments displayed by the terminal also change accordingly as the playback time point of the first free viewpoint video changes.

[0038] In a possible implementation, the method further includes: the seventh interface further includes a first barrage comment, and a display style of the first barrage comment in the seventh interface is different from a display style of the first barrage comment in the first interface.

[0039] In other words, in the sixth operation process, the terminal adjusts the display style of each barrage comment in real time according to changes in the playback viewpoint.

[0040] In a possible implementation, the seventh interface further includes a sixth barrage comment, and the appearance time of the sixth barrage comment is after the first playback time.

[0041] In other words, after the viewpoint of the first free viewpoint is switched, the terminal continuously plays the first free viewpoint video, and the content of the barrage comments changes accordingly as the playback time point of the first free viewpoint video changes.

[0042] According to a second aspect, there is provided a terminal including a processor, a memory and a touch screen. The memory and the touch screen are coupled to the processor, and the memory is configured to store computer program code, the computer program code including computer instructions. When the processor reads the computer instructions from the memory, the terminal is enabled to execute the method according to the first aspect and any one of the possible implementations of the first aspect.

[0043] According to a third aspect, there is provided a computer-readable storage medium comprising computer instructions which, when executed on a terminal, enable the terminal to perform a method according to the first aspect and any one of the possible implementations of the first aspect.

[0044] According to a fourth aspect, there is provided a computer program product, which, when executed on a computer, enables a terminal to execute the method according to the first aspect and any one of the possible implementations of the first aspect.

[0045] For technical effects that can be achieved by the terminal, computer-readable storage medium, and computer program product provided in the second to fourth aspects, please refer to the description of the technical effects in the first aspect and the possible implementation manner of the first aspect, and the details will not be described again here. [Brief description of the drawings]

[0046] [Figure 1] FIG. 1 is a schematic diagram of the structure of a system for playing free viewpoint video according to an embodiment of this application. [Diagram 2] 1 is a schematic diagram of the structure of a terminal according to an embodiment of this application; [Diagram 3] FIG. 2 is a schematic diagram of a server structure according to an embodiment of this application; [Figure 4] 1 is a schematic flowchart of a method for transmitting barrage comments of a free viewpoint video according to an embodiment of the present application; [Figure 5(1)] 2 is a schematic diagram of several terminal interfaces according to an embodiment of the present application; [Figure 5 (2)] 2 is a schematic diagram of several terminal interfaces according to an embodiment of the present application; [Figure 6(1)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 6 (2)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 6 (3)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 7] 1 is a schematic flowchart of a method for displaying barrage comments according to an embodiment of the present application; [Figure 8] 1 is a schematic diagram of a relationship between a viewpoint of a barrage comment and a playback viewpoint of a free viewpoint video according to an embodiment of the present application. [Figure 9(1)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 9 (2)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 9 (3)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 10A] 1 is a schematic flowchart of a method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application; [Figure 10B(1)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 10B(2)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 10B(3)]4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 11A] 11 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application. [Figure 11B(1)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 11B(2)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 11B(3)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 12A] 11 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application. [Figure 12B] 11 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application. [Figure 13] FIG. 13 is a schematic diagram of another relationship between the viewpoint of the barrage comments and the playback viewpoint of the free viewpoint video according to an embodiment of the present application. [Figure 14A(1)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 14A(2)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 14A(3)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 14B(1)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 14B(2)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 14B(3)] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 15A]11 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application. [Figure 15B] 4A-4C are schematic diagrams of some other terminal interfaces according to embodiments of the present application. [Figure 16] 11 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application. [Figure 17] 11 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints of a free viewpoint video according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] In order to facilitate understanding of the technical solutions provided in the embodiments of this application, technical terms in the embodiments of this application are first explained.

[0048] In free viewpoint video, a target object is shot simultaneously by multiple cameras, and during playback, this is equivalent to a user selecting the shooting angle of one of the cameras for viewing. Free viewpoint video may be generated in several ways:

[0049] Method 1: Multiple cameras arranged around a target object jointly shoot the target object by using the target object as the center of the shot, and a specific video encoding is performed on the jointly shot video stream to generate a special format video file. When a user views a free viewpoint video, this is equivalent to the viewer selecting a viewpoint around the shot target object for viewing.

[0050] Method 2: Multiple cameras are arranged around the viewer by using the viewer as the center to jointly shoot the target object, and specific video encoding is performed on the jointly shot video stream to generate a special format video file, for example, to generate 3-DoF (degrees of freedom) video (e.g., 360-degree panoramic video). When a user views a free viewpoint video, this is equivalent to the viewer not moving but rotating his / her head to view the surrounding scenery.

[0051] Method 3: The above two methods are combined. For example, a 6-DoF image is generated. When viewing a free viewpoint image, the user can move forward, backward, left or right to view the image, or rotate his / her head to view the surrounding scenery.

[0052] The camera position is the number of the position where each camera is located when multiple cameras around the target object or view capture the target object. Generally, one camera position corresponds to one shooting angle. At one shooting angle, an image of the target object within a certain angle range may be captured.

[0053] The viewpoint is also called a viewing viewpoint, a playback viewpoint, etc., and is a shooting angle at which a user views a free viewpoint video. Specifically, an image of a target object seen at a viewpoint may be considered as an image captured when a camera performs shooting at a position at the viewpoint (i.e., a camera position).

[0054] Video playback is the process of displaying a group of images in time order in a display window and playing the corresponding audio after a video file (including free viewpoint video) has been decoded, rendered, etc.

[0055] Danmaku comments are comments or annotations that are displayed on a video (including free viewpoint video) at a particular time or for a particular period of time according to certain rules, resulting in an interesting and unexpected experience in viewing the video.

[0056] 1 is a schematic diagram of a system architecture according to an embodiment of this application. The system architecture includes a camera array 500, a content distribution network (CDN) 200, a media server 300, a CDN 400, a terminal 100, and a barrage comment server 600.

[0057] Camera array 500 includes multiple (e.g., 18-36) cameras or other devices with camera lenses surrounding a target object (i.e., a shooting object / shooting scenario) and / or a viewer. These cameras correspond to camera position 0, camera position 1, ..., and camera position n, respectively. Camera array 500 separately shoots the target object from multiple angles and uploads the video streams shot from the multiple angles to media server 300. In some examples, synchronization of the videos shot by the multiple cameras in the camera array or other devices with camera lenses may be achieved by using a metronome.

[0058] By using functional modules such as load balancing, content delivery and scheduling modules on a central platform, the CDN 200 enables video streams collected by the camera array 500 from multiple angles to be uploaded to the media server 300 in an accelerated manner by using edge servers located at each location.

[0059] The media server 300 is configured to transcode the received video streams at multiple angles to generate free viewpoint videos. For example, the processing processes performed by the media server 300 on the video streams at multiple angles include, but are not limited to, focusing, frame alignment synchronization encoding, rotation auxiliary stream generation, virtual viewpoint synthesis, etc. Focusing may be used to ensure a smooth transition of the image of the shooting object in the process of switching the viewpoint in a manner such as cropping the image. Frame alignment synchronization encoding is used to align the video frames captured by multiple cameras at the same time. In virtual viewpoint synthesis, a virtual video stream is inserted between two real video streams to reduce the number of cameras that need to be used during shooting or to implement a smooth transition of the image of the shooting object in the process of switching the viewpoint. The rotation auxiliary stream is used to display in a rotation manner, which helps to reduce the total stream usage.

[0060] It should be noted that the above processing process of the media server 300 is merely an example, and the specific technical solutions used in the process of generating free viewpoint videos based on the video streams uploaded by the camera array 500 are not limited to the embodiments of this application.

[0061] It should be further noted that the media server 300 herein may be at least one of an independent physical server, a plurality of independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtual center.

[0062] It can be understood that the media server 300 stores and maintains a large amount of free viewpoint videos. When the media server 300 receives a request sent by the terminal 100 to request the free viewpoint video content, the media server 300 quickly returns the corresponding free viewpoint video content to the terminal 100 through the CDN 400.

[0063] It should be noted that the media server 300 here may be at least one of an independent physical server, a plurality of independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtual center.

[0064] The CDN 400 enables a terminal user (e.g., the terminal 100) to obtain nearby required content, such as free viewpoint video, by using functional modules such as a load balancing module, a content delivery module and a scheduling module on a central platform and by using edge servers located at each location, thereby reducing network congestion and improving user access response speed and hit rate.

[0065] The terminal 100 may request a free viewpoint video from the media server 300 by using a free viewpoint video client / application installed in the terminal 100, and then receive the free viewpoint video file returned by the media server 300 through the CDN 400. Alternatively, the terminal 100 downloads a free viewpoint video file from a corresponding server based on the playback address of the free viewpoint video returned by the media server 300. Finally, the terminal performs processing such as decoding and rendering on the free viewpoint video file, and then plays the free viewpoint video. If the terminal 100 further enables the barrage comment function when playing the free viewpoint video, the terminal 100 needs to further request barrage comment data corresponding to the current playback time or period from the barrage comment server 600.

[0066] In this embodiment of the application, the danmaku comment data returned by the danmaku comment server 600 includes the content of the danmaku comment, the time when the danmaku comment appeared, the viewpoint of the danmaku comment, etc. The viewpoint of the danmaku comment is the viewpoint of the free viewpoint video being viewed by the user when the danmaku comment was sent. It can be understood that the content of the danmaku comment sent by the user is usually related to the video image at the current viewing viewpoint. For example, the content of the danmaku comment sent by the user is usually a comment or explanation regarding the video image at the current viewing viewpoint. In other words, in this embodiment of the application, the danmaku comment sent by the user is related to the current playback viewpoint.

[0067] In this embodiment of the application, the terminal 100 uses different display methods for multiple danmaku comments at different viewpoints based on the viewpoints of the danmaku comments to help the user distinguish between multiple danmaku comments at different viewpoints. In other words, the danmaku comments at different viewpoints are displayed differently. For example, in a video image at a user's current playback viewpoint 1, danmaku comments at viewpoints 1 and 2 are currently displayed. In this case, the terminal 100 separately displays the danmaku comments at viewpoint 1 and the danmaku comments at viewpoint 2 in different display methods, thereby allowing the user to intuitively distinguish between the danmaku comments at viewpoint 1 and the danmaku comments at viewpoint 2. When the danmaku comments at viewpoint 2 do not match the video image at playback viewpoint 1, the user will not have confusion or bewilderment. This improves the video viewing experience.

[0068] Furthermore, when playing back the free viewpoint video, terminal 100 may further receive new content of the danmaku comment entered by the user, establish an association between the new content of the danmaku comment and the viewpoint of the video currently being viewed by the user, and upload the association to danmaku comment server 600 for storage.

[0069] The terminal 100 may further receive a viewpoint switching operation performed by a user (e.g., a user's finger sliding operation on a screen or a user's operation of pressing a button on a remote control), and may calculate a viewpoint switching direction (e.g., switching the viewpoint to the left or switching the viewpoint to the right) and a switching angle amplitude (e.g., switching by 10 degrees or 20 degrees) based on the viewpoint switching operation performed by the user to switch the viewing viewpoint of the current free viewpoint video. In some examples, the display of the terminal 100 is a touch screen. In this case, the terminal 100 may capture a screen event, the screen event including a detected finger slide event, including a finger slide direction and distance, and determine a viewing viewpoint switching direction and a switching angle based on the screen event. In some other examples, the terminal 100 is configured as a remote control. In this case, the terminal 100 may receive a button event from a remote control, where the button event includes a button identifier, the number of times the button is pressed or the duration of the press, etc., and then determine a switching direction and a switching angle of the viewing viewpoint based on the button event, for example, by identifying a button operated by a user, for example, tapping the left button or the right button, based on the button identifier, to determine the switching direction, and by identifying a switching angle amplitude based on the number of times the button is pressed or the duration of the press.

[0070] For example, the terminal 100 in this embodiment of the present application may be, for example, a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a netbook, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, an in-vehicle device, a smart screen, or an intelligent vehicle. The specific form of the terminal 100 is not particularly limited in this application.

[0071] The barrage comment server 600 is configured to receive barrage comment information transmitted by each terminal (including, for example, terminal 100), and the barrage comment information includes, but is not limited to, an identifier of the free viewpoint video, the time when the barrage comment appears, the duration of the barrage comment, the content of the barrage comment, the viewpoint of the barrage comment, etc.

[0072] It should be noted that the barrage comment server 600 here may be at least one of an independent physical server, a plurality of independent physical servers, a cloud server providing cloud computing, a cloud computing platform, and a virtual center.

[0073] It can be understood that the structure of the system shown in this embodiment of the present invention does not constitute a specific limitation on the system architecture to which the technical solution is applicable. In some other embodiments of this application, the system architecture to which the technical solution provided in the embodiments of this application is applicable may further include more or less devices than those shown in the drawings, and some devices are combined, or some devices are separated.

[0074] 2 is an exemplary diagram of the structure of a terminal 100 according to an embodiment of this application. The terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identification module (SIM) card interface 195, etc.

[0075] It may be understood that the structure shown in this embodiment of the invention does not constitute a specific limitation on the terminal 100. In some other embodiments of this application, the terminal 100 may include more or fewer components than those shown in the drawings, or some components may be combined, or some components may be split, or a different component layout may exist. The components shown in the drawings may be implemented in hardware, software, or a combination of software and hardware.

[0076] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), and / or the like. The different processing units may be separate components or may be integrated into one or more processors.

[0077] A video codec is configured to compress or decompress digital video. The terminal 100 may support one or more video codecs. Thus, the terminal 100 may play or record video in multiple encoding formats, such as moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4. In this embodiment of the present application, the video codec may decode free viewpoint video downloaded by the terminal 100 from the media server 300 to realize playback on the terminal 100.

[0078] The internal memory 121 may be configured to store computer executable program code. The executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., a free viewpoint video playback function), and the like. The data storage area may store data (e.g., audio data and contacts) and the like that are created when the terminal 100 is used. Furthermore, the internal memory 121 may include a high-speed random access memory, or may include a non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS). The processor 110 executes instructions stored in the internal memory 121 and / or instructions stored in a memory located in the processor to perform various functional applications and data processing of the terminal 100.

[0079] In some embodiments of this application, the internal memory 121 stores program code required by a free viewpoint video client / application. The free viewpoint video client / application may realize a function of downloading free viewpoint video from the media server 300 and playing the free viewpoint video, receive the contents of the danmaku comments input by the user, upload the contents of the danmaku comments input by the user and other danmaku comment information (e.g., danmaku comment viewpoint information) to the danmaku comment server 600, receive the danmaku comment data transmitted by the danmaku comment server 600, and display the danmaku comments at different viewpoints differently based on the viewpoint information carried in the danmaku comment data, etc.

[0080] The wireless communication function of the terminal 100 may be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, a baseband processor, and so on.

[0081] Mobile communication module 150 may provide wireless communication solutions including 2G / 3G / 4G / 5G / 6G / next generation wireless communication technologies, etc., applied to terminal 100. In some embodiments, at least some of the functional modules in mobile communication module 150 may be located in processor 110. In some embodiments, at least some of the functional modules of mobile communication module 150 may be located in the same device as at least some of the modules of processor 110.

[0082] The wireless communication module 160 may provide a solution for wireless communication applied to the terminal 100, including wireless local area network (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc.

[0083] The terminal 100 may implement display functions by using a GPU, a display 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0084] The terminal 100 may realize the photographing function by using an ISP, a camera 193, a video codec, a GPU, a display 194, an application processor, and the like.

[0085] The external memory interface 120 may be used to connect an external memory card, for example, a micro SD card, to expand the storage of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize a data storage function. For example, files such as music and video are stored in the external memory card.

[0086] The terminal 100 may implement audio functions such as music playback and recording through the use of an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, an application processor, and the like.

[0087] The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 may receive the charging input of the wired charger through the USB interface 130. In some embodiments in which wireless charging is performed, the charging management module 140 may receive the wireless charging input through a wireless charging coil of the terminal 100. When charging the battery 142, the charging management module 140 may further provide power to the terminal by using the power management module 141.

[0088] The power management module 141 is configured to connect to the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, etc. The power management module 141 may be further configured to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage or impedance). In some other embodiments, the power management module 141 may alternatively be located in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may alternatively be located in the same component.

[0089] 3 is a schematic diagram of the structure of the barrage comment server 600. The barrage comment server 600 includes one or more processors 610, one or more memories 620, and one or more communication interfaces 630.

[0090] The processor 610, the memory 620, and the communication interface 630 are connected through a bus. The processor 610 may include a general-purpose central processing unit (CPU) (e.g., CPU0 and CPU1), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a neural-network processing unit (NPU), an integrated circuit configured to control program execution of the solutions of this application, etc.

[0091] The memory 620 may be configured to store computer executable program code. The executable program code includes instructions. The memory 620 may include a program storage area and a data storage area. The program storage area may store an operating system, application code, and the like. In some examples, the data storage area stores barrage comment data for each free viewpoint video uploaded by a user, and the barrage comment data includes a viewpoint of each barrage comment. Furthermore, the memory 620 may include a high-speed random access memory, and may further include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory, or a universal flash storage (UFS). The processor 610 executes various functions, applications, and data processing of the barrage comment server 600 by executing instructions stored in the memory 620. In one example, the processor 610 may alternatively include multiple CPUs, and the processor 610 may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may be one or more devices, circuits, or processing cores configured to process data (eg, computer program instructions).

[0092] The communication interface 630 may be configured to communicate with other devices or communication networks, such as an Ethernet or a wireless local area network (WLAN).

[0093] It should be noted that the media server 300, the server in the CDN 200 or the CDN 400, etc. shown in Fig. 1 refer to the structure of the barrage comment server 600 shown in Fig. 3. The media server 300 and the server in the CDN 200 or the CDN 400 may include more or less components than the barrage comment server 600, or some components may be combined, or some components may be divided, or there may be a different component layout. This is not limited to the embodiment of this application.

[0094] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS With reference to the accompanying drawings and specific embodiments, the following describes in detail the technical solutions provided in the embodiments of this application.

[0095] 1. Solution for sending barrage comments

[0096] 4 is a schematic flowchart of a method for sending barrage comments according to an embodiment of the present application. The method includes the following steps:

[0097] S400: Terminal 100 plays back a video image of the first free viewpoint video at viewpoint A.

[0098] For example, a client (e.g., a web page) or an application (e.g., a video application of a mobile terminal) used to play a free viewpoint video is installed on the terminal 100. The client / application may be called a client / application of a free viewpoint video. A user may launch the client / application and select to play the first free viewpoint video in a display interface of the client / application. In some embodiments, the terminal 100 plays a video image of the first free viewpoint video at viewpoint A by default. When detecting an operation of selecting the playback of the first free viewpoint video by a user, the terminal 100 requests a video file 1 of the first free viewpoint video at viewpoint A from the media server 300 through the CDN 400. In response, the media server 300 returns the video file 1 to the terminal 100 through the CDN 400, and the terminal 100 displays the video image of the first free viewpoint video at viewpoint A based on the video file 1. In some other embodiments, the terminal 100 first plays a video image of the first free viewpoint video at a viewpoint other than viewpoint A. When the terminal 100 detects an operation by the user indicating switching to viewpoint A, the terminal 100 requests a video file 1 of a first free viewpoint video at viewpoint A from the media server 300 via the CDN 400. In response, the media server 300 returns the video file 1 to the terminal 100 via the CDN 400, and the terminal 100 displays a video image of the first free viewpoint video at viewpoint A based on the video file 1.

[0099] In some other examples, when the terminal 100 plays the first free viewpoint video, the danmaku comment function is enabled. In this case, the terminal 100 further requests danmaku comment data at the corresponding playback time point from the danmaku comment server 600 based on the playback progress. In this case, the video image played by the terminal 100 further includes the corresponding danmaku comment.

[0100] For example, an example in which the terminal 100 is a mobile phone is used for explanation. A video application installed on the mobile phone integrates a software development kit (SDK) for playing free viewpoint videos to support playing free viewpoint videos. In this case, a user can launch the video application on the mobile phone and select to play a first free viewpoint video in the interface of the video application. For example, FIG. 5(1) shows a playback interface 601 displayed by the mobile phone and used to play a first free viewpoint video. The playback interface 601 includes a video image 602 of the first free viewpoint video at a specific viewpoint, and danmaku comments (including, for example, danmaku comments 603) that appear on the video image 602 according to a specific rule. The playback interface 601 may further include some function controls, such as a playback progress bar 604, a playback control 609, a next episode control 610, a danmaku enable / disable control 605, a danmaku comment input box 606, a danmaku comment send control 607, and a viewpoint control 608.

[0101] The playback progress bar 604 is used to display the playback progress of the current video, and is used by the user to adjust the playback progress of the current video through dragging, etc. The playback control 609 is used to pause or play the current video continuously. The next episode control 610 is used to switch to the next episode or the next video. The danmaku enable / disable control 605 is used to enable or disable the danmaku comment function. The danmaku comment input box 606 is used to accept the content of the danmaku comment entered by the user. The danmaku comment send control 607 is used to send the content of the danmaku comment entered by the user or the default content of the danmaku comment.

[0102] The viewpoint control 608 is used to display a viewpoint (e.g., 60 degrees) corresponding to the video image 602 currently being played by the mobile phone, and is used to call up a viewpoint ruler used to switch viewpoints. When it is detected that the user has operated the viewpoint control 608, the mobile phone displays a playback interface 611 shown in FIG. 5(2), and a viewpoint ruler 612 is displayed on the playback interface 611. The user may switch the viewpoint of the free viewpoint video currently being played by the mobile phone by sliding left or right on the viewpoint ruler 612 by using a finger. For example, when it is detected that the user's finger slides right on the ruler, the mobile phone displays a playback interface 613 shown in FIG. 6(1). The playback interface 613 is a video image of a first free viewpoint video at viewpoint A (e.g., 70 degrees).

[0103] It should be noted that an example in which a user operates viewpoint control 608 to call viewpoint ruler 612 is used for explanation purposes here. In some other examples, the mobile phone may alternatively directly display viewpoint ruler 612 on free viewpoint video playback interface 601, call viewpoint ruler 612 in other manners, etc. The interface for displaying viewpoint ruler 612 is not limited in the embodiments of this application.

[0104] In some examples, after not receiving any operation performed by the user on the viewpoint ruler in the preset duration 1, the mobile phone may hide the viewpoint ruler, for example, as shown in the playback interface 614 in Fig. 6(2). In some other examples, when detecting a user tapping operation on the display in the playback interface 614, the mobile phone displays the playback interface 615 shown in Fig. 6(3), and the above-mentioned functional controls such as the playback progress bar 604, the playback control 609, the next episode control 610, the danmaku enable / disable control 605, the danmaku comment input box 606, the danmaku comment send control 607, and the viewpoint control 608 are redisplayed in the playback interface 615.

[0105] S401: Terminal 100 receives an operation by a user to input the content of a first barrage comment and send the first barrage comment, and terminal 100 generates data of the first barrage comment, and the data of the first barrage comment includes the content of the first barrage comment, the time when the first barrage comment appeared, and the viewpoint of the first barrage comment.

[0106] For example, a user may input and transmit a danmaku comment through the video playback interface 601 shown in Fig. 5(1) or the video playback interface 615 shown in Fig. 6(3). Here, an example in which a user inputs a first danmaku comment into the video playback interface 615 shown in Fig. 6(3) and transmits the first danmaku comment is used for explanation.

[0107] As shown in FIG. 6(3), when the mobile phone plays back the video of the first free viewpoint video at viewpoint A, the mobile phone detects that the user has input the content of the first danmaku comment into the danmaku comment input box 606 and operated the danmaku comment transmission control 607. The mobile phone generates data of the first danmaku comment. In an embodiment of this application, the data of the first danmaku comment generated by the mobile phone includes the content of the first danmaku comment (e.g., "Amazing"), the appearance time of the first danmaku comment (e.g., 20:10), and the viewpoint of the first danmaku comment (i.e., viewpoint A, e.g., 70 degrees). Furthermore, the data of the first danmaku comment further includes one or more of an identifier of the first free viewpoint video, an identifier of the first danmaku comment, a duration ΔT of the first danmaku comment, a format of the first danmaku comment, a transmission position of the first danmaku comment (i.e., a playback time point at which the first danmaku comment is transmitted), and the like. Table 1 below shows an example of a data format used for the data of the first barrage comment. [Table 1]

[0108] The value of the movie identifier field is an identifier of the first free viewpoint video, for example, movie 1. The value of the danmaku comment identifier field is an identifier of the first danmaku comment, for example, danmaku comment 1. The value of the danmaku comment content field is the content of the first danmaku comment entered by the user, for example, "Amazing."

[0109] The value of the appearance time field is the playback time when the display starts to display the first danmaku comment, for example, the playback time T1. In some embodiments, the playback time T1 is equal to the transmission time of the first danmaku comment. For example, if the user of terminal 1 transmits the first danmaku comment at 20:10 of the free viewpoint video, the appearance time of the first danmaku comment, i.e., the playback time T1, may be 20:10. In some other embodiments, it may be understood that the appearance time of the first danmaku comment is different from the transmission time of the first danmaku comment. For example, in some scenarios, the duration of a highlight in a video is long, or the user is accustomed to transmitting a danmaku comment at the beginning of a highlight. In this case, the transmission time of the first danmaku comment is determined as the playback time T1, reminding other users to pay attention to the highlight in the video. In some other embodiments, the playback time T1 is equal to the transmission time of the first danmaku comment minus the preset time offset value 1. The preset time offset value 1 may be predefined or may be set by the user. This is not limited in the embodiment of this application. The setting manner and calculation method of other preset time offset values ​​(for example, preset time offset value 2) in this specification are similar to those of preset time offset value 1, and may be the same as or different from those of preset time offset value 1. Details will not be described below. For example, if the sending time of the first barrage comment is 20:10 and the preset time offset value 1 is 5 seconds, the appearance time of the first barrage comment, that is, the playback time T1 is 20:05. It can be understood that in some other scenarios, the duration of the highlight in the video is short, or the user is accustomed to sending the barrage comment at the end of the highlight. In this case, the sending time of the first barrage comment-preset time offset value 1 is determined as the playback time T1, to remind other users in advance to pay attention to the highlight in the video. In some other embodiments, the playback time T1 is equal to the sending time of the first barrage comment+preset time offset value 2.For example, if the sending time of the first barrage comment is 20:10 and the preset time offset value 2 is 3 seconds, the appearance time of the first barrage comment, i.e., the playback time T1, is 20:08. The preset time offset value 1 and the preset time offset value 2 may be the same or different. In some other scenarios, it may be understood that the user is accustomed to having a discussion or the like before the highlight. In this case, the sending time of the first barrage comment + the preset time offset value 2 is determined as the playback time T1, reminding other users to pay attention to the highlight in the video in a timely manner.

[0110] The value of the duration field is the duration ΔT (e.g., 15 seconds) during which the first barrage comment is continuously displayed on the display. In some examples, the first barrage comment starts to appear from the rightmost side of the display of the terminal, and then the first barrage comment moves from the rightmost side to the leftmost side of the display until the first barrage comment disappears from the display. The entire process from the time when the first barrage comment is displayed from the rightmost side of the display to the time when the first barrage comment disappears at the leftmost side of the display takes ΔT, e.g., 15 seconds. The duration ΔT may be a preset value or may be set by the user. For example, the user may set a barrage comment mode, and different barrage comment modes correspond to different durations. The value of the font or color field, etc. is set by the user when the user inputs the content of the first barrage comment, or is set by default on the terminal 100.

[0111] The value of the viewpoint field is the viewpoint corresponding to the video image currently being displayed by terminal 100 when the first danmaku comment was sent, i.e., viewpoint A. In other words, a correspondence is established between the first danmaku comment and viewpoint A.

[0112] S402: The terminal 100 transmits the data of the first barrage comment to the barrage server 600.

[0113] In some embodiments, after receiving the data of the first barrage comment transmitted by the terminal 100, the barrage comment server reviews the data of the first barrage comment, for example, reviews the content and reviews the format. After the review is successful, the barrage comment server stores the data of the first barrage comment.

[0114] In this way, the danmaku comment server 600 stores the danmaku comment data of the first free viewpoint video transmitted from each viewpoint and uploaded by each user.

[0115] It can be understood that the danmaku comment server 600 stores a large amount of danmaku comment data of a free viewpoint video at each viewpoint. Subsequently, other terminals (including the terminal 100) may request the danmaku comment data of a specific free viewpoint video at a specific playback time point from the danmaku comment server 600. Furthermore, the danmaku comment data returned by the danmaku comment server 600 includes the contents of the danmaku comment, the time point at which the danmaku comment appears, the viewpoint of the danmaku comment, etc. Subsequently, when displaying the danmaku comments, the other terminals may determine different display methods for the danmaku comments at different viewpoints.

[0116] 2. Solution for displaying barrage comments

[0117] For example, the method for displaying barrage comments provided in the embodiments of this application is described by using an example in which a terminal 100 (here, the terminal 100 may be the same as or different from the terminal 100 that sends the first barrage comment) plays a first free viewpoint video.

[0118] 7 is a schematic flowchart of a method for displaying barrage comments according to an embodiment of the present application. The method includes the following steps:

[0119] S701: The terminal 100 requests a video file of a first free viewpoint video in viewpoint B at playback time point T2 from the media server 300. In response, the media server 300 returns the video file of the first free viewpoint video in viewpoint B at playback time point T2 to the terminal 100.

[0120] For example, a user may select to play a video at any viewpoint by using a free viewpoint video client / application of the terminal 100. Based on the playback progress of the first free viewpoint video, the terminal 100 requests a video file at a corresponding playback time point from the media server 300. For example, the terminal 100 requests a video file of the first free viewpoint video at viewpoint B at playback time point T2 from the media server 300.

[0121] S702: The terminal 100 further requests danmaku comment data of the first free viewpoint video at the playback time point T2 from the danmaku comment server 600. In response, the danmaku comment server 600 returns the danmaku comment data of the first free viewpoint video at the playback time point T2 to the terminal 100.

[0122] When the terminal 100 plays back a first free viewpoint video and the danmaku comment function is enabled, the terminal 100 further needs to request danmaku comment data at a corresponding playback time point from the danmaku comment server 600 based on the playback progress of the first free viewpoint video. For example, the terminal 100 requests danmaku comment data of the first free viewpoint video at playback time point T2 from the media server 300. In this case, it should be noted that the danmaku comment data of the first free viewpoint video at playback time point T2 returned to the terminal 100 by the danmaku comment server 600 includes danmaku comments at each viewpoint, for example, a danmaku comment at viewpoint B, a danmaku comment at viewpoint A, and danmaku comments at other viewpoints.

[0123] It should be noted that step S702 may be performed after, before, or simultaneously with step S701, which is not particularly limited in the embodiment of this application.

[0124] S703: The terminal 100 calculates the offset angle between viewpoint B and the viewpoint of each barrage comment.

[0125] In some embodiments, the offset angle includes an offset direction and / or a value of the offset angle. The offset direction includes a left offset direction and a right offset direction. If the offset direction is a left offset direction, this means that when viewpoint B is switched to the viewpoint of the danmaku comment, viewpoint B needs to be switched to the left. If the offset direction is a right offset direction, this means that when viewpoint B is switched to the viewpoint of the danmaku comment, viewpoint B needs to be switched to the right. The value of the offset angle is the absolute value of the difference between viewpoint B and viewpoint of the danmaku comment. In other embodiments, offset angle = viewpoint of the danmaku comment - viewpoint B. When the offset angle is a positive number, this indicates that the danmaku comment is a right offset danmaku comment. When the offset angle is a negative number, this indicates that the danmaku comment is a left offset danmaku comment. The definition and calculation of the offset angle are not particularly limited in the embodiments of this application. This specification is described by using an example where offset angle = viewpoint of the danmaku comment - viewpoint B. Details will not be described later.

[0126] For example, in the playback interface 613 shown in Fig. 6(1), the current playback viewpoint (i.e., viewpoint B) is 70 degrees, and the viewpoint of danmaku comment 1 is 60 degrees. In this case, the offset angle between viewpoint B and the viewpoint of danmaku comment 1 is a left offset of 10 degrees. When the viewpoint of danmaku comment 2 is 100 degrees, the offset angle between viewpoint B and the viewpoint of danmaku comment 2 is a right offset of 30 degrees.

[0127] In another example, as described above, the free viewpoint video is a video captured by a camera around the shooting target object, and thus the free viewpoint video includes images of the shooting target object at multiple viewpoints. As shown in FIG. 8, the current viewer angle (the playback viewpoint of the free viewpoint video, i.e., also called viewpoint B) is 0 degrees. When the transmission viewpoint of danmaku comment 1 (also called the viewpoint of danmaku comment 1) is -30 degrees, the offset angle between viewpoint B and the viewpoint of danmaku comment 1 is a left offset of 30 degrees. When the transmission viewpoint of danmaku comment 2 (also called the viewpoint of danmaku comment 2) is +25 degrees, the offset angle between viewpoint B and the viewpoint of danmaku comment 2 is a right offset of 25 degrees. When the transmission viewpoint of danmaku comment 3 (also called the viewpoint of danmaku comment 3) is -100 degrees, the offset angle between viewpoint B and the viewpoint of danmaku comment 3 is a left offset of 100 degrees. When the sending viewpoint of danmaku comment 4 (also called the viewpoint of danmaku comment 4) is +90 degrees, the offset angle between viewpoint B and the viewpoint of danmaku comment 4 is a right offset of 90 degrees.

[0128] It should be noted that FIG. 9 shows a free viewpoint image captured in a circle (i.e., 360 degrees) around the shooting target object. In other words, the free viewpoint image is a 360-degree surrounding image. Obviously, in some other examples, the free viewpoint image may not be captured in a circle around the shooting target object. For example, the free viewpoint image is a 180-degree surrounding image, a 130-degree surrounding image, or a 270-degree surrounding image.

[0129] S704: The terminal 100 displays a video image of the first free viewpoint video at viewpoint B, and presents each danmaku comment based on an offset angle between viewpoint B and the viewpoint of each danmaku comment.

[0130] It can be understood that when the value of the offset angle between viewpoint B and the viewpoint of the barrage comments is different, the barrage comments are presented in different manners. For example, the differences in the presentation manners include different tilt manners (including tilt direction and / or tilt degree) of the text direction of the barrage comments, different rotation manners (including rotation direction and / or rotation degree) of the text of the barrage comments, different fonts, different three-dimensional effects, different movement manners, etc.

[0131] In the embodiment, the barrage comment of the first free viewpoint video at the playback viewpoint (i.e., viewpoint B) is presented in a different manner from the barrage comment at other viewpoints. In this embodiment example, the different barrage comment may be presented by using the following policy. When |Barrage comment viewpoint-viewpoint B|≦threshold A (for example, threshold A is 0 degrees or 5 degrees), the barrage comment is displayed in a first presentation manner. When |Barrage comment viewpoint-viewpoint B|>threshold A, the barrage comment is displayed in a second presentation manner, which is different from the first presentation manner. For example, the first presentation manner is that the text of the barrage comment is displayed normally (for example, the text of the barrage comment is not tilted or rotated, the font is not transformed, and there is a flat effect). In this case, the text direction of the barrage comment is parallel to the first direction. For example, the first direction is a direction parallel to the upper bezel / lower bezel of the screen of the terminal 100. The second presentation manner performs display in one or more manners of tilt, rotation, font transformation, and three-dimensional effect, i.e., the text direction of the barrage comments is a direction obtained by rotating the first direction (e.g., parallel to the upper bezel / lower bezel of the screen of the terminal 100) clockwise by a specific angle. In this case, a specific included angle is formed between the text direction of the barrage comments and the bezel on one side of the screen of the terminal 100. In another example, in the first presentation manner, the barrage comments move in a first direction (e.g., left), and in the second presentation manner, the barrage comments move in a second direction (e.g., right). The first direction is different from the second direction.

[0132] For example, in the playback interface 901 of the first free viewpoint video shown in FIG. 9(1), the viewpoint of the danmaku comment 902 (the viewpoint is 60 degrees) is the same as the playback viewpoint of the first free viewpoint video (i.e., viewpoint B, which is 60 degrees), and the text of the danmaku comment 902 is displayed normally without tilting. The viewpoint of the danmaku comment 903 (the viewpoint is 90 degrees), the viewpoint of the danmaku comment 904 (the viewpoint is 30 degrees), the viewpoint of the danmaku comment 905 (the viewpoint is 120 degrees), and the viewpoint of the danmaku comment 906 (the viewpoint is 10 degrees) are all different from the playback viewpoint of the first free viewpoint video, and the text of the four danmaku comments are all displayed in a tilted manner. It may be understood that in some other examples, when displaying the danmaku comments, the terminal 100 may display viewpoint information of the danmaku comments or may not display viewpoint information of the danmaku comments.

[0133] It can be seen that the user may distinguish the danmaku comments of the first free viewpoint video at the playback viewpoint (i.e., viewpoint B) from the danmaku comments at other viewpoints based on the presentation manner of the danmaku comments. In this way, the user will not have confusion or doubt when the video at the playback viewpoint is inconsistent with the content of the danmaku comments at other viewpoints. This improves the viewing experience of the first free viewpoint video.

[0134] In other embodiments, when the offset direction between viewpoint B and the viewpoint of the danmaku comments is different, the danmaku comments are presented in different manners. In one example, when the offset angle between viewpoint B and the viewpoint of the danmaku comments is a left offset, the danmaku comments are displayed in a third presentation manner. When the offset angle between viewpoint B and the viewpoint of the danmaku comments is a right offset, the danmaku comments are displayed in a fourth presentation manner. The third presentation manner is different from the fourth presentation manner.

[0135] For example, an example in which the first direction is a direction parallel to the top bezel of the screen of the terminal 100 is used for explanation. In the third presentation method, the text direction of the barrage comment is a direction obtained by rotating the first direction counterclockwise by a specific angle. In the fourth presentation method, the text direction of the barrage comment is a direction obtained by rotating the first direction clockwise by a specific angle. For example, in the playback interface 907 of the first free viewpoint video shown in FIG. 9(2), the viewpoint of the barrage comment 902 (the viewpoint is 60 degrees) is the same as the playback viewpoint of the first free viewpoint video (i.e., viewpoint B, which is 60 degrees), and the text of the barrage comment 902 is displayed normally without tilt. The offset angle between the viewpoint of the barrage comment 903 (the viewpoint is 90 degrees) and the playback viewpoint of the first free viewpoint video is a right offset, and the text direction of the barrage comment 903 is a direction obtained by rotating the first direction clockwise by a specific angle. Similarly, the offset angle between the viewpoint of the danmaku comment 905 (the viewpoint is 120 degrees) and the playback viewpoint of the first free viewpoint video is also a right offset, and the text direction of the danmaku comment 905 is a direction obtained by rotating the first direction clockwise by a specific angle. The offset angle between the viewpoint of the danmaku comment 904 (the viewpoint is 30 degrees) and the playback viewpoint of the first free viewpoint video is a left offset, and the text direction of the danmaku comment 904 is a direction obtained by rotating the first direction counterclockwise by a specific angle. Similarly, the offset angle between the viewpoint of the danmaku comment 906 (the viewpoint is 10 degrees) and the playback viewpoint of the first free viewpoint video is also a left offset, and the text direction of the danmaku comment 906 is a direction obtained by rotating the first direction counterclockwise by a specific angle.

[0136] In another example, the third presentation scheme includes a first mark (eg, a left arrow) and the fourth presentation scheme includes a second mark (eg, a right arrow).

[0137] It can be seen that when the offset direction between the viewpoint of the danmaku comment and the playback viewpoint of the first free viewpoint video is different and the presentation manner of the danmaku comment is also different, the user may distinguish the offset direction of the viewpoint corresponding to the danmaku comment based on the presentation manner of the danmaku comment. In this case, when the user is interested in some danmaku comments (called target danmaku comments), the user may determine the switching direction of the viewpoint based on the presentation manner of the target danmaku comment to switch to the viewpoint corresponding to the target danmaku comment. In other words, the different presentation methods of the danmaku comments may inform the user of the direction that needs to be switched when the user switches the viewpoint, thereby enabling the user to quickly switch the viewpoint.

[0138] For example, in the playback interface 907 shown in FIG. 9(2), if a user is interested in danmaku comment 903 and wishes to switch to a viewpoint corresponding to danmaku comment 903, the user knows that the viewpoint offset between playback viewpoint B and the viewpoint of danmaku comment 903 is a right offset based on the presentation method in which danmaku comment 903 tilts to the right. In this way, the user may call up a viewpoint ruler and slide to the right on the viewpoint ruler until the viewpoint corresponding to danmaku comment 903 can be switched to. If a user is interested in danmaku comment 904 and wishes to switch to a viewpoint corresponding to danmaku comment 904, the user knows that the viewpoint offset between playback viewpoint B and the viewpoint of danmaku comment 904 is a left offset based on the presentation method in which danmaku comment 904 tilts to the left. In this way, the user may call up a viewpoint ruler and slide to the left on the viewpoint ruler until the viewpoint corresponding to danmaku comment 904 can be switched to.

[0139] In yet another embodiment, when the offset direction between viewpoint B and the viewpoint of the danmaku comment is the same but the value of the offset angle is different, the danmaku comment is presented in a different manner. For example, the larger the offset angle between viewpoint B and the viewpoint of the danmaku comment, the greater the degree of tilt present when the danmaku comment is presented, i.e., the greater the tilt angle of the danmaku comment. For example, in the playback interface 908 shown in FIG. 9(3), both the danmaku comment 903 and the danmaku comment 905 are right offset with respect to the playback viewpoint of the first free viewpoint video. When |viewpoint of danmaku comment 903-viewpoint B|<{viewpoint of danmaku comment 905-viewpoint B|, the tilt angle a of the danmaku comment 903<the tilt angle b of the danmaku comment 905. Both the danmaku comment 904 and the danmaku comment 906 are right offset with respect to the playback viewpoint of the first free viewpoint video. When |viewpoint of danmaku comment 904-viewpoint B|<|viewpoint of danmaku comment 906-viewpoint B|, then the inclination angle c of danmaku comment 904<the inclination angle d of danmaku comment 906.

[0140] Specifically, in one example, the slant angle of the text direction of the barrage comment is directly proportional to the value of the offset angle of the barrage comment. In other words, the slant angle of the text direction of the barrage comment is in a linear increasing relationship based on the value of the offset angle of the barrage comment. In another example, the slant angle of the text direction of the barrage comment may alternatively be in a stepwise increasing relationship based on the value of the offset angle of the barrage comment. For example, the following policies may be used to present different barrage comments:

[0141] When |viewpoint of danmaku comment-viewpoint B|≦threshold B (for example, threshold B is 10 degrees or 15 degrees), the danmaku comment is displayed as usual.

[0142] When threshold B<|viewpoint of danmaku comment-viewpoint B|≦threshold C (for example, threshold C is 30 degrees or 45 degrees), the text direction of the danmaku comment is the direction obtained by rotating the first direction counterclockwise / clockwise by a first angle.

[0143] When threshold C<|viewpoint of danmaku comment-viewpoint B|≦threshold D (e.g., threshold D is 60 degrees), the text direction of the danmaku comment is a direction obtained by rotating the first direction counterclockwise / clockwise by a second angle, where the second angle is greater than the first angle.

[0144] ...

[0145] Table 2 below provides an example of the correspondence between the offset angle of the barrage comment and the tilt angle of the barrage comment. [Table 2]

[0146] In conclusion, when the offset angle between viewpoint B and the viewpoint of the danmaku comment is different and the presentation manner of the danmaku comment is different, the user may distinguish the value of the offset angle between viewpoint B and the viewpoint of the danmaku comment based on the presentation manner of the danmaku comment. In this case, when the user is interested in some danmaku comments (called target danmaku comments), the user may determine the viewpoint switching amplitude (e.g., the sliding distance of the finger, or the pressing duration or the number of pressings of the button of the remote control) based on the presentation manner of the target danmaku comment to switch to the viewpoint corresponding to the target danmaku comment. In other words, the different presentation manners of the danmaku comments may inform the user of the viewpoint switching amplitude (e.g., the sliding distance of the finger, or the pressing duration or the number of pressings of the button of the remote control), thereby enabling the user to quickly switch the viewpoint.

[0147] It can be understood that in some other embodiments, the danmaku comments in the playback interface in all viewpoints, such as danmaku comment 902, danmaku comment 903, danmaku comment 904, danmaku comment 905, and danmaku comment 906 in the playback interface 908, may be presented in the same manner, e.g., all are displayed normally. It can be understood that the user may distinguish the danmaku comments of the first free viewpoint video in the playback viewpoint (i.e., viewpoint B) from the danmaku comments in the other viewpoints based on the viewpoint carried in each danmaku comment. In this way, the user will not have confusion or doubt when the video in the playback viewpoint does not match the content of the danmaku comments in the other viewpoints. This improves the viewing experience of the first free viewpoint video.

[0148] As described above, the user may randomly switch the playback viewpoint of the first free viewpoint video. In some embodiments, in the process of switching the viewpoint of the first free viewpoint video, the terminal 100 keeps the content and presentation manner of the danmaku comments unchanged. After the viewpoint of the first free viewpoint video is switched, the terminal 100 reacquires the danmaku comment data acquired after the viewpoint is switched, and determines the display content of the danmaku comments and the specific presentation method of each danmaku comment based on the reacquired danmaku comment data.

[0149] For example, Figure 10A shows a method for displaying barrage comments in a process of switching viewpoints according to an embodiment of the present application. The method includes the following steps:

[0150] S1001: At time t0, the terminal 100 detects that the user has started to perform a viewpoint switching operation, and the viewpoint switching operation indicates switching of the viewpoint of the first free viewpoint video.

[0151] It should be noted that before step S1001, the terminal 100 may display a video image of the first free viewpoint video at any viewpoint and a corresponding barrage comment by using the method in steps S701 to S704. In one example, the terminal 100 displays a video image of the first free viewpoint video at viewpoint B at playback time point T2 and a barrage comment corresponding to playback time point T2.

[0152] The viewpoint switching operation may be, for example, a slide operation performed by the user on the screen of the terminal 100, an operation of tilting the terminal 100 by the user, or an operation of operating a button on a remote control by the user.

[0153] For example, when the terminal 100 detects that the user's finger stays on the viewpoint ruler for longer than a preset duration 2 (e.g., 3 seconds), the terminal 100 determines that the user has started to perform a viewpoint switching operation and records the current time as time t0. In another example, when the terminal 100 is configured with a device such as a gravity acceleration or IMU device, the attitude of the terminal 100 can be detected. When it is detected that the tilt angle of the terminal 100 with respect to the horizontal plane is equal to or greater than threshold 1, it is determined that the user has started to perform a viewpoint switching operation and records the current time as time t0. In another example, when the terminal 100 receives a signal corresponding to a specific button transmitted by a remote control, it determines that the user has started to perform a viewpoint switching operation and records the current time as time t0.

[0154] S1002: From time t0 to time t1, the terminal 100 keeps the presentation method of each barrage comment unchanged.

[0155] S1003: At time t1, the terminal 100 detects that the viewpoint switching operation is completed.

[0156] For example, after time t0, the terminal continuously detects that the user's finger slides on the screen. When it detects that the user's finger is removed from the screen, the terminal determines that the user has completed the viewpoint switching operation, and records the current time as time t1. In another example, after time t0, the terminal continuously detects the tilt angle of the terminal 100 with respect to the horizontal plane. When it detects that the tilt angle of the terminal 100 with respect to the horizontal angle is less than threshold 1, it is determined that the user has completed the viewpoint switching operation, and records the current time as time t1. In another example, after time t0, the terminal continuously detects a signal corresponding to a specific button transmitted by the remote control. When the signal corresponding to the specific button is not detected for a preset duration 3 (e.g., 5 seconds) after the signal corresponding to the specific button was last detected, it is determined that the user has completed the viewpoint switching operation, and records the current time as time t1.

[0157] For example, in the process of a viewpoint switching operation being performed by a user, i.e., between time t0 and time t1 (excluding time t1), terminal 100 may leave the presentation method of each danmaku comment unchanged, for example, may leave the tilt direction and tilt angle of each danmaku comment unchanged. In other words, regardless of whether the playback viewpoint of the first free viewpoint video changes between time t0 and time t1, terminal 100 always maintains the presentation method of the danmaku comments displayed by terminal 100 at time t0.

[0158] It should be noted that in some examples, the playback viewpoint of the video image of the first free viewpoint video changes between time t0 and time t1. For example, the terminal 100 may alternatively play the video image of the first free viewpoint video at another viewpoint (e.g., any one or more viewpoints between viewpoint B and viewpoint C) based on a viewpoint switching operation performed by a user (e.g., a finger slide distance, or a number of times or duration of pressing a button on a remote control). In other words, from time t0 to time t1, the playback viewpoint of the first free viewpoint video is switched from viewpoint B to viewpoint C, and the playback viewpoint of the first free viewpoint video passes through all or some of the viewpoints from viewpoint B to viewpoint C. For example, when the viewpoint of the first free viewpoint video is switched from 60 degrees to 90 degrees, the viewpoint of the first free viewpoint video undergoes all angle changes between 60 degrees and 90 degrees, or undergoes some angle changes between 60 degrees and 90 degrees. For example, the angles that are received include 60 degrees, 70 degrees, 80 degrees, and 90 degrees.

[0159] In some other examples, between time t0 and time t1, the playback viewpoint of the video image of the first free viewpoint video does not change, i.e., the terminal 100 continues to play the video image of the first free viewpoint video at angle B.

[0160] It should be further noted that the terminal 100 may determine the barrage comments to be displayed and the display positions of these barrage comments based on the playback progress (i.e., the playback time point) of the first free viewpoint video. In some examples, in the process of performing a viewpoint switching operation by the user, i.e., between time t0 and time t1 (excluding time t1), the terminal 100 plays the first free viewpoint video continuously. In this case, the playback time point of the video image of the first free viewpoint video changes. In this case, the terminal 100 determines in real time the barrage comments to be displayed by the terminal 100 and the display positions of the barrage comments based on the change in the playback time point. However, the presentation manner of each barrage comment (e.g., the tilt direction and tilt angle of the barrage comment) remains unchanged. In some other examples, in the process of performing a viewpoint switching operation by the user, i.e., between time t0 and time t1 (excluding time t1), the terminal 100 pauses the playback of the first free viewpoint video. During this period, the playback time point of the video image of the first free viewpoint video does not change. In this case, both the barrage comment displayed by the terminal 100 and the display position of the barrage comment remain unchanged.

[0161] S1004: After time t1, the terminal 100 requests a video file of the first free viewpoint video from viewpoint C at playback time T3 from the media server.

[0162] In response, the media server returns to the terminal 100 the video file of the first free viewpoint video from viewpoint C at the playback time point T3.

[0163] In some embodiments, after detecting that the user has completed the viewpoint switching operation, the terminal 100 sends a request 1 to the media server, where the request 1 is used to request a video file corresponding to the viewpoint acquired after switching. In one example, the request 1 includes an identifier of the first free viewpoint video, a viewpoint C, and a playback time T3. The playback time T3 is related to the playback time T2. For example, the playback time T3=the playback time T2, the playback time T3=the playback time T2-preset time offset value 1, or the playback time T3=the playback time T2+preset time offset value 2. The preset time offset value 1 and the preset time offset value 2 are the same or different.

[0164] S1005: After time t1, the terminal 100 further requests, from the barrage comment server, barrage comment data corresponding to playback time T3 of the first free viewpoint video.

[0165] In response, the barrage comment server returns to the terminal 100 the barrage comment data corresponding to the playback time point T3 of the first free viewpoint video.

[0166] In some embodiments, after detecting that the user has completed the viewpoint switching operation, the terminal 100 sends a request 2 to the media server, and the request 2 is used to request barrage comment data corresponding to the playback time T3 of the first free viewpoint video. In one example, the request 2 includes an identifier of the first free viewpoint video and a playback time T3. The playback time T3 is related to the playback time T2. For example, the playback time T3=playback time T2, the playback time T3=playback time T2-preset time offset value 1, or the playback time T3=playback time T2+preset time offset value 2. In some other embodiments, the terminal 100 plays the first free viewpoint video continuously in the process of performing the viewpoint switching operation by the user. In this case, the playback time T3=playback time T2+(time t1-time t0).

[0167] It should be noted that step S1005 may be performed after, before, or simultaneously with step S1004.

[0168] It should further be noted that this step is an optional step. In some other embodiments, playback time point T3=playback time point T2, or playback time point T3=playback time point T2-preset time offset value 1. The terminal 100 stores the danmaku comment data corresponding to the playback time point T3. In this case, the terminal 100 does not need to request the danmaku comment data at the playback time point T3 again from the danmaku comment server. As playback of the first free viewpoint video progresses, the terminal 100 requests, from the danmaku comment server, danmaku comment data that exists after the playback time point T3 or the playback time point T2.

[0169] S1006: The terminal 100 recalculates the offset angle between the viewpoint C and the viewpoint of each barrage comment after the viewpoint is switched.

[0170] It should be noted that before the viewpoint is switched, the terminal 100 calculates the offset angle between viewpoint B and the viewpoint of each danmaku comment. After the viewpoint is switched, the terminal 100 calculates the offset angle between viewpoint C and the viewpoint of each danmaku comment. For a specific method for calculating the offset angle, see the related content in step S703.

[0171] S1007: The terminal 100 displays the video image of the first free viewpoint video at viewpoint C at playback time point T3, and displays each barrage comment based on the offset angle between viewpoint C and the viewpoint of each barrage comment.

[0172] For this step, please refer to the relevant content in step S704, and the details will not be described again here.

[0173] For example, as shown in FIG. 10B(1), the terminal 100 plays a video image 1001 of a first free viewpoint video at a viewpoint of 60 degrees (i.e., viewpoint B). The playback interface 1001 further displays barrage comments at different viewpoints, such as barrage comment 1002, barrage comment 1003, barrage comment 1004, barrage comment 1005, and barrage comment 1006. At time t0, the terminal 100 detects that the user has dragged the viewpoint ruler in the playback interface 1001 (the playback time in this case is 20:05, i.e., playback time T2), and at time t1, the terminal 100 detects that the user's finger has been removed from the display of the terminal 100. In response to information indicating that the user has performed a viewpoint switching operation (i.e., an operation of dragging the viewpoint ruler by the user) at time t1, the terminal 100 switches the playback viewpoint of the first free viewpoint video, and displays the playback interface 1007 shown in FIG. 10B(2). It may be noted that in this case, the playback viewpoint of the first free viewpoint video is switched to a viewpoint of 70 degrees (i.e., viewpoint C). Furthermore, the presentation manner of each danmaku comment in the playback interface 1007 is changed. For example, compared with the presentation manner of each danmaku comment in the playback interface 1001, the danmaku comment 1002 is changed from a normal display to a tilted display. The tilt angles of the danmaku comments 1003 and 1005 are decreased (a is changed to a', and b is changed to b'). The tilt angles of the danmaku comments 1004 and 1006 are increased (c is changed to c', and d is changed to d'). In some examples, the terminal 100 pauses the playback of the first free viewpoint video between time t0 and time t1 (excluding time t1), i.e., the video image displayed by the terminal 100 remains unchanged. In this case, the playback time point of the first free viewpoint video in the playback interface 1007 is 20:05 (that is, the playback time point T3 is the same as the playback time point T2).

[0174] In some other examples, the terminal 100 continuously plays the first free viewpoint video between time t0 and time t1 (excluding time t1). In this case, in response to information indicating that the user has performed a viewpoint switching operation (i.e., an operation of dragging the viewpoint ruler by the user) at time t1, the terminal 100 switches the playback viewpoint of the first free viewpoint video and displays the playback interface 1008 shown in FIG. 10B(3). The playback time of the first free viewpoint video in the playback interface 1008 is 20:10 (specifically, the playback time T3=playback time T3=playback time T2+(time t1-time t0)).

[0175] In conclusion, in this embodiment, in the process of switching the viewpoint by the user, i.e., between time t0 and viewpoint t1, the terminal 100 does not update the specific presentation manner of the barrage comments. This helps the user to compare the change in the presentation manner of the barrage comments before and after the viewpoint is switched, reduces the processing load of the terminal 100, and reduces the power consumption of the terminal 100.

[0176] In some other embodiments, in the process of switching the viewpoint of the first free viewpoint video, the presentation manner of each barrage comment displayed by the terminal 100 changes in real time based on the current playback viewpoint of the first free viewpoint video. In other words, step S1002 in Fig. 10A may be replaced by step S1101 in Fig. 11A.

[0177] 11A shows another method for displaying barrage comments in a process of switching viewpoints according to an embodiment of the present application, the method includes step S1001, step S1101, and step S1003 to step S1007.

[0178] For step S1001 and steps S1003 to S1007, please refer to the explanation of the corresponding steps in Fig. 10A, and the details will not be described again here.

[0179] S1101: From time t0 to time t1, terminal 100 calculates in real time a viewpoint offset between the viewpoint of each danmaku comment and the playback viewpoint of the first free viewpoint video, and displays each danmaku comment based on the viewpoint offset calculated in real time.

[0180] In some embodiments, the playback viewpoint of the video image of the first free viewpoint video changes between time t0 and time t1. For example, the terminal 100 may alternatively play the video image of the first free viewpoint video at another viewpoint (e.g., any one or more viewpoints between viewpoint B and viewpoint C) based on a viewpoint switching operation performed by a user (e.g., a finger slide distance, or a number of times or duration of pressing a button on a remote control). In other words, from time t0 to time t1, the playback viewpoint of the first free viewpoint video is switched from viewpoint B to viewpoint C, and the playback viewpoint of the first free viewpoint video passes through all or some of the viewpoints from viewpoint B to viewpoint C. For example, when the viewpoint of the first free viewpoint video is switched from 60 degrees to 90 degrees, the viewpoint of the first free viewpoint video passes through all angle changes between 60 degrees and 90 degrees, or passes through some angle changes between 60 degrees and 90 degrees, for example, passing through angles including 60 degrees, 70 degrees, 80 degrees, and 90 degrees.

[0181] From time t0 to time t1, the terminal 100 determines that as the playback viewpoint of the first free viewpoint video changes, the presentation manner of each danmaku comment changes accordingly. Specifically, when the playback viewpoint of the first free viewpoint video changes, the terminal 100 calculates in real time an angle offset between the viewpoint of each danmaku comment and the changed playback viewpoint of the first free viewpoint video, and determines the presentation manner of each danmaku comment based on the angle offset calculated in real time.

[0182] It should be further noted that the terminal 100 may determine the barrage comments to be displayed and the display positions of these barrage comments based on the playback progress (i.e., the playback time point) of the first free viewpoint video. In some examples, in the process of performing a viewpoint switching operation by the user, that is, between time t0 and time t1 (excluding time t1), the terminal 100 plays the first free viewpoint video continuously. In this case, the playback time point of the video image of the first free viewpoint video changes. In this case, the terminal 100 determines the barrage comments to be displayed by the terminal 100 and the display positions of the barrage comments in real time based on the change in the playback time point. In other words, the contents of the barrage comments displayed by the terminal 100 change with the playback progress of the first free viewpoint video. Optionally, in this scenario, between time t0 and time t1 (excluding time t1), the terminal 100 may further request barrage comment data corresponding to each playback time point in real time from the barrage comment server based on the playback progress of the first free viewpoint video.

[0183] In some other examples, in the process of performing a viewpoint switching operation by the user, that is, between time t0 and time t1 (excluding time t1), the terminal 100 pauses the playback of the first free viewpoint video. During this period, the playback time of the video image of the first free viewpoint video does not change. In this case, both the barrage comment displayed by the terminal 100 and the display position of the barrage comment remain unchanged.

[0184] 11B(1), for example, terminal 100 plays video image 1101 of a first free viewpoint video at a viewpoint of 60 degrees (i.e., viewpoint B). Playback interface 1101 further displays danmaku comments at different viewpoints, such as danmaku comment 1102, danmaku comment 1103, danmaku comment 1104, danmaku comment 1105, and danmaku comment 1106. At time t0, terminal 100 detects that the user has dragged the viewpoint ruler in playback interface 1101 (the playback time in this case is 20:05, i.e., playback time T2), and at time t1, terminal 100 detects that the user's finger has been lifted off the display of terminal 100.

[0185] When the terminal 100 detects that the user has performed a viewpoint switching operation between time t0 and time t1 (excluding time t0 and time t1), the playback viewpoint of the first free viewpoint video displayed by the terminal 100 changes continuously, for example, from 60 degrees to 65 degrees, as shown in the playback interface 1107 shown in FIG. 11B(2). It may be noted that the presentation method of each danmaku comment in the playback interface 1107 is changed. For example, compared with the presentation method of each danmaku comment in the playback interface 1101, the danmaku comment 1102 changes from a normal display to a tilted display (tilt angle is e'). The tilt angles of the danmaku comment 1003 and the danmaku comment 1005 decrease (a is changed to a'' and b is changed to b''). The tilt angles of the danmaku comment 1004 and the danmaku comment 1006 increase (c is changed to c'' and d is changed to d'').

[0186] At time t1, the terminal 100 further switches the playback viewpoint of the first free viewpoint video, and displays the playback interface 1108 shown in FIG. 11B(3). In this case, it may be noted that the playback viewpoint of the first free viewpoint video is switched to a viewpoint of 70 degrees (i.e., viewpoint C). Furthermore, the presentation manner of each danmaku comment in the playback interface 1108 is changed again. For example, compared with the presentation manner of each danmaku comment in the playback interface 1107, the tilt angle of the danmaku comment 1102 further increases (e' is changed to e). The tilt angles of the danmaku comment 1003 and the danmaku comment 1005 decrease (a'' is changed to a', and b'' is changed to b'). The tilt angles of the danmaku comment 1004 and the danmaku comment 1006 increase (c'' is changed to c', and d'' is changed to d').

[0187] In conclusion, in this embodiment, in the process of switching the viewpoint by the user, the terminal 100 adjusts the presentation manner of the barrage comments in real time based on the change of the playback viewpoint of the first free viewpoint video, which helps the user know the distribution status of the barrage comments of the first free viewpoint video at each viewpoint, and finally indicates to the user to select the switched playback viewpoint.

[0188] In some other embodiments, in the process of switching the viewpoint of the first free viewpoint video, execution of the viewpoint switching operation may be paused. In this case, when the viewpoint switching operation is paused, the terminal 100 may re-request the barrage comment data corresponding to the current playback time point from the barrage comment server, and may determine the display contents of the barrage comments and the specific presentation method of each barrage comment based on the re-acquired barrage comment data.

[0189] 12A and 12B are schematic flowcharts of another method for displaying barrage comments in a process of switching viewpoints according to an embodiment of the present application. The method includes the following steps:

[0190] S1201: At time t0, the terminal 100 detects that the user has started to perform a viewpoint switching operation, where the viewpoint switching operation indicates switching of the viewpoint of the first free viewpoint video.

[0191] In one example, the terminal 100 displays a video image of the first free viewpoint video at viewpoint B at the playback time point T2 and barrage comment data of the first free viewpoint video at the playback time point T2.

[0192] The viewpoint switching operation may be, for example, a slide operation performed by the user on the screen of the terminal 100, an operation of tilting the terminal 100 by the user, or an operation of a button on a remote control by the user. For example, if the terminal 100 detects that the user's finger remains on the viewpoint ruler for longer than a preset duration 2 (e.g., 3 seconds), the terminal 100 determines that the user has started to perform a viewpoint switching operation and records the current time as time t0.

[0193] For other contents of this step, please refer to the relevant contents in step S1001 in Fig. 10A, and the details will not be described again here.

[0194] S1202: From time t0 to time t2, the terminal 100 keeps the presentation method of the displayed barrage comments unchanged.

[0195] Time point t2 is between time points t0 and t1.

[0196] For other contents of this step, please refer to the relevant contents of step S1002 in Fig. 10A, and the details will not be described again here.

[0197] In some other embodiments, between time t0 and time t2, the terminal 100 may alternatively adjust the presentation manner of each barrage comment in real time based on a change in the playback viewpoint of the first free viewpoint video. For related content, refer to the description of the content of step S1101 in FIG. 11A. Details will not be described again here.

[0198] S1203: At time t2, the terminal 100 detects that the user has temporarily halted the execution of the viewpoint switching operation.

[0199] For example, after time t0, the terminal continuously detects that the user's finger slides on the screen. When the terminal detects that the duration during which the user's finger does not slide and does not leave the screen exceeds a preset duration 4 (e.g., 3 seconds), the terminal determines that the user has paused the execution of the viewpoint switching operation and records the current time as time t2.

[0200] S1204: After time point t2, the terminal 100 requests a video file of the first free viewpoint video at viewpoint D at playback time point T4 from the media server.

[0201] In other words, after detecting that the user pauses the execution of the viewpoint switching operation, the terminal 100 determines the current playback viewpoint of the first free viewpoint video, for example, viewpoint D. The playback time T4 is related to the playback time T2. In some examples, when the terminal 100 pauses the playback of the first free viewpoint video from time t0 to time t2, the playback time T4=playback time T2, or the playback time T4=playback time T2-preset time offset value 1. For example, when the playback time T2 is 20:10, the playback time T4 is 20:10 or 20:05. In some other examples, when the terminal 100 continuously plays the first free viewpoint video from time t0 to time T2, the playback time T4=playback time T2+(time t2-time t0). In other examples, when the playback time T2 is 20:10, the playback time T4 is 20:15.

[0202] S1205: After time point t2, the terminal 100 further requests, from the barrage comment server, barrage comment data corresponding to playback time point T4 of the first free viewpoint video.

[0203] S1206: The terminal 100 recalculates the offset angle between viewpoint D and the viewpoint of each barrage comment after the viewpoint is switched.

[0204] S1207: The terminal 100 displays the video image of the first free viewpoint video at viewpoint D at playback time point T4, and displays each barrage comment based on the offset angle between viewpoint D and the viewpoint of each barrage comment.

[0205] For other details of steps S1204 to S1207, refer to the relevant details of steps S1004 to S1007 in Fig. 10A, and the details will not be described again here.

[0206] S1208: At time t1, the terminal 100 detects that the viewpoint switching operation is completed.

[0207] For example, after time t2, the terminal continuously detects that the user's finger slides on the screen. When the terminal detects that the user's finger is removed from the screen, the terminal determines that the user has completed the viewpoint switching operation and records the current time as time t1.

[0208] S1209: After time point t1, the terminal 100 requests a video file of the first free viewpoint video from viewpoint C at playback time point T3 from the media server.

[0209] S1210: After time point t1, the terminal 100 further requests, from the barrage comment server, barrage comment data corresponding to playback time point T3 of the first free viewpoint video.

[0210] In some examples, the playback time T3 is relative to the playback time T4, for example, playback time T3=playback time T4, playback time T3=playback time T4-preset time offset value 1, or playback time T3=playback time T4+preset time offset value 2. In other examples, playback time T3=playback time T4+(time t1-time t0).

[0211] S1211: The terminal 100 recalculates the offset angle between the viewpoint C and the viewpoint of each barrage comment after the viewpoint is switched.

[0212] S1212: The terminal 100 displays the video image of the first free viewpoint video at viewpoint C at playback time point T3, and displays each barrage comment based on the offset angle between viewpoint C and the viewpoint of each barrage comment.

[0213] It should be noted that the embodiments described in Figures 12A and 12B are described by using an example in which the viewpoint switching operation is paused only once. It can be understood that the technical solutions provided in the embodiments of this application are also applicable to the case in which the viewpoint switching operation is paused more than once.

[0214] For other details of steps S1208 to S1212, refer to the relevant details of steps S1003 to S1007 in Fig. 10A, and the details will not be described again here.

[0215] In this embodiment, in the process of switching the viewpoint by the user, after the user pauses the operation of switching the viewpoint, the terminal 100 may adjust the display manner of the barrage comments in real time based on the current playback viewpoint of the first free viewpoint video. This helps the user know the distribution status of the barrage comments of the first free viewpoint video at the current playback viewpoint, and indicates to the user to finally select the switched playback viewpoint.

[0216] In another embodiment, the terminal 100 may further predefine a visible viewpoint range of the barrage comments, or the terminal 100 may provide an option for the user to set the visible viewpoint range of the barrage comments. The visible viewpoint range of the barrage comments means that the terminal 100 displays the contents of the barrage comments within the viewpoint range and does not display the contents of the barrage comments outside the viewpoint range.

[0217] For example, terminal 100 may predefine the visible viewpoint range of the barrage comment, or may receive the visible viewpoint range of the barrage comment set by the user, as follows: The absolute value of the offset angle between the viewpoint of the barrage comment and the playback viewpoint of the first free viewpoint video is less than the angle threshold E (e.g., 45 degrees). In this case, terminal 100 may determine that the visible viewpoint range of the barrage comment is [playback viewpoint of the first free viewpoint video-angle threshold E, playback viewpoint of the first free viewpoint video+angle threshold E].

[0218] In another example, the terminal 100 may predefine the visible viewpoint range of the danmaku comment, or may receive the visible viewpoint range of the danmaku comment set by the user, as follows: The absolute value of the offset angle between the viewpoint of the danmaku comment and the playback viewpoint of the first free viewpoint video is less than the angle threshold E, and the offset angle is equal to or less than 0 (i.e., the offset direction is a left offset). In this case, the terminal 100 may determine that the visible viewpoint range of the danmaku comment is [playback viewpoint of the first free viewpoint video - angle threshold E, playback viewpoint of the first free viewpoint video].

[0219] In another example, the terminal 100 may predefine the visible viewpoint range of the danmaku comment, or may receive the visible viewpoint range of the danmaku comment set by the user, as follows: The absolute value of the offset angle between the viewpoint of the danmaku comment and the playback viewpoint of the first free viewpoint video is less than the angle threshold E, and the offset angle is equal to or greater than 0 (i.e., the offset direction is a right offset). In this case, the terminal 100 may determine that the visible viewpoint range of the danmaku comment is [playback viewpoint of the first free viewpoint video, playback viewpoint of the first free viewpoint video + angle threshold E].

[0220] In some other embodiments, terminal 100 may further collect and count the number of danmaku comments outside the visible viewpoint range of the danmaku comments, and display the number of danmaku comments outside the visible viewpoint range to notify the user of the state of the danmaku comments outside the visible viewpoint range. Furthermore, terminal 100 may further count the number of left offset danmaku comments and the number of right offset danmaku comments separately for the danmaku comments outside the visible viewpoint range of the danmaku comments.

[0221] Alternatively, the terminal 100 may further display a control related to the number of barrage comments outside the visible viewpoint range. For example, the control related to the number of barrage comments outside the visible viewpoint range may indicate the enthusiasm of users participating in discussions about video images outside the visible viewpoint range. In a specific implementation, when the number of barrage comments outside the visible viewpoint range is equal to or greater than a certain number, the control may indicate that the viewpoint outside the visible viewpoint range is a hot area, and may be used to prompt users to pay attention to video images outside the visible viewpoint range. In another specific implementation, the terminal 100 displays a control 1, which is related to the number of left offset barrage comments. When the number of left offset barrage comments is equal to or greater than a certain number, the control 1 may indicate that the left offset viewpoint outside the visible viewpoint range is a hot area. The terminal 100 displays a control 2, which is related to the number of right offset barrage comments. When the number of right offset barrage comments is equal to or greater than a certain number, the control 2 may indicate that the right offset viewpoint outside the visible viewpoint range is a hot area.

[0222] For example, the visible viewpoint range of the danmaku comments is set as follows: the absolute value of the offset angle between the viewpoint of the danmaku comment and the playback viewpoint of the first free viewpoint video is less than an angle threshold E, the first free viewpoint video is a 360-degree surrounding video, the number of left offset danmaku comments may be calculated by using formula (1), and the number of right offset danmaku comments may be calculated by using formula (2).

[0223] Number of left offset barrage comments = Σ(-180 degrees < offset angle < - angle threshold E) Equation (1)

[0224] Number of right offset barrage comments = Σ(angle threshold E<offset angle<180 degrees) Equation (2)

[0225] It can be understood that if the first free viewpoint image is a 180-degree ambient image, the 180 degrees in equations (1) and (2) are replaced with 90 degrees. If the first free viewpoint image is a 270-degree ambient image, the 180 degrees in equations (1) and (2) are replaced with 135 degrees.

[0226] As shown in FIG. 13, when the angle threshold E is set to 45 degrees, the visible viewpoint range of the barrage comments in the first free viewpoint video is [playback viewpoint of the first free viewpoint video -45 degrees, playback viewpoint of the first free viewpoint video +45 degrees].

[0227] When the playback viewpoint of the first free viewpoint video is 0 degrees, the visible viewpoint range of the danmaku comments is [-45 degrees, +45 degrees]. Danmaku comment 1 and danmaku comment 2 are located within the visible viewpoint range of the danmaku comments, and terminal 100 displays the contents of danmaku comment 1 and danmaku comment 2. Danmaku comment 3 and danmaku comment 5 are located at [-180 degrees, -45 degrees), outside the visible viewpoint range of the danmaku comments, and are left offset danmaku comments. In this case, terminal 100 may display prompt information, for example, +2, on the left side of the screen to prompt the user that the number of left offset danmaku comments located outside the visible viewpoint range of the danmaku comments is 2. Danmaku comment 4 and danmaku comment 6 are located at (+45 degrees, +180 degrees), outside the visible viewpoint range of the danmaku comments, and are right offset danmaku comments. In this case, the terminal 100 may display prompt information, for example, +2, on the right side of the screen to prompt the user that the number of right-offset barrage comments located outside the visible viewpoint range of the barrage comments is 2.

[0228] As shown in the playback interface 1401 illustrated in FIG. 14A(1), the playback interface 1401 includes danmaku comments within a visible viewpoint range, for example, danmaku comment 1 and danmaku comment 2. The playback interface 1401 further includes a control 1402 and a control 1403. The control 1402 is used to display the number of danmaku comments that are located outside the visible viewpoint range of the danmaku comments and have a left offset. The control 1403 is used to display the number of danmaku comments that are located outside the visible viewpoint range of the danmaku comments and have a right offset. In some examples, the user may operate the control 1402 to display the danmaku comments that are located outside the visible viewpoint range of the danmaku comments and have a left offset. For example, in response to detecting that the user has operated the control 1402 in the playback interface 1401, the terminal 100 displays the playback interface 1404 illustrated in FIG. 14A(2). The playback interface 1404 displays the danmaku comments that are located outside the visible viewpoint range of the danmaku comments and have a left offset, for example, the danmaku comment 3 and the danmaku comment 5. It should be noted that the position of each danmaku comment on the display is related to the current playback time point of the first free viewpoint video and the appearance time point of each danmaku comment, and is not related to the angle offset between the viewpoint of each danmaku comment and the current playback viewpoint of the first free viewpoint video. The playback interface 1404 may further include a control 1405 used to hide danmaku comments outside the visible viewpoint range. Optionally, in response to detecting that the user has operated the control 1402 in the playback interface 1404, the terminal 100 displays the playback interface 1406 shown in FIG. 14A(3). The playback interface 1406 displays danmaku comments that are located outside the visible viewpoint range of the danmaku comments and are offset to the right, for example, danmaku comment 4 and danmaku comment 6. The playback interface 1406 may further include a control 1405 used to hide danmaku comments outside the visible viewpoint range.

[0229] It should be noted that the barrage comments in the playback interface 1401, the playback interface 1404, and the playback interface 1406 may be presented in the same manner, e.g., may be displayed normally. In some other embodiments, the barrage comments in the playback interface 1401, the playback interface 1404, and the playback interface 1406 may alternatively be different. For example, any one of the above-mentioned barrage comment display methods may be used for presentation. The specific presentation manner of the barrage comments is not limited herein.

[0230] In another example, as shown in a playback interface 1407 shown in Fig. 14B(1), a playback interface 1408 shown in Fig. 14B(2), and a playback interface 1409 shown in Fig. 14B(3), the presentation method of each danmaku comment is determined based on an offset angle between the viewpoint of each danmaku comment and the playback viewpoint of the first free viewpoint video. For a specific method of determination, refer to the above description. Furthermore, for functional control in each playback interface, refer to the description of the related contents of the playback interface 1401 shown in Fig. 14A(1), the playback interface 1404 shown in Fig. 14A(2), and the playback interface 1406 shown in Fig. 14A(2). Details will not be described again here.

[0231] In some other examples, the terminal 100 may automatically further display hidden danmaku comments outside the visible viewpoint range based on a viewpoint switching operation performed by the user. For example, when the terminal 100 detects that the user has performed an operation to switch the viewpoint to the left in the interface 1401 shown in FIG. 14A(1), when switching the viewpoint, the terminal 100 may also display danmaku comments that are located outside the visible viewpoint range of the danmaku comments and that are offset to the left. In another example, when the terminal 100 detects that the user has performed an operation to switch the viewpoint to the right in the interface 1401 shown in FIG. 14A(1), when switching the viewpoint, the terminal 100 may also display danmaku comments that are located outside the visible viewpoint range of the danmaku comments and that are offset to the right.

[0232] Since the contents of the danmaku comments within the visible viewpoint range are usually strongly related to the content of the video image currently viewed by the user at the playback viewpoint, it can be seen that when the terminal 100 displays only the contents of the danmaku comments within the visible viewpoint range and does not display the contents of the danmaku comments outside the visible viewpoint range, the user can concentrate on viewing the current video image and the contents of the danmaku comments related to the current video image, improving the viewing experience. When the user is interested in the contents of the danmaku comments outside the visible viewpoint range, the contents of the danmaku comments outside the visible viewpoint range may alternatively be expanded by using corresponding controls to meet the viewing requirements of the user in multiple scenarios.

[0233] As described above, the user may randomly switch the playback viewpoint of the first free viewpoint video. In some embodiments, in the process of switching the viewpoint of the first free viewpoint video, the terminal 100 leaves the number of danmaku comments outside the visible viewpoint range unchanged. After the viewpoint of the first free viewpoint video is switched, the terminal 100 reacquires the danmaku comment data acquired after the viewpoint is switched, and determines the contents of the danmaku comments within the visible viewpoint range and the number of danmaku comments outside the visible viewpoint range based on the reacquired danmaku comment data.

[0234] For example, Figure 15A shows a method for displaying barrage comments in a process of switching viewpoints according to an embodiment of the present application. The method includes the following steps:

[0235] S1501: At time t0, the terminal 100 detects that the user has started to perform a viewpoint switching operation, and the viewpoint switching operation indicates switching of the viewpoint of the first free viewpoint video.

[0236] It should be noted that before step S1501, terminal 100 may display a video image of the first free viewpoint video at any viewpoint and the corresponding danmaku comments by using the method in steps S701 to S704. In one example, terminal 100 displays a video image of the first free viewpoint video at viewpoint B at playback time point T2 and the danmaku comments corresponding to playback time point T2. The danmaku comments corresponding to playback time point T2 include the contents of the danmaku comments within the visible viewpoint range and the number of danmaku comments outside the visible viewpoint range.

[0237] S1502: From time t0 to time t1, the terminal 100 displays the content that the number of barrage comments outside the visible viewpoint range remains unchanged.

[0238] In some embodiments, between time t0 and time t1, terminal 100 retains the number of barrage comments outside the visible viewpoint range displayed by terminal 100 at time t0. In other words, between time t0 and time t1, the number of barrage comments outside the visible viewpoint range displayed by terminal 100 remains unchanged.

[0239] It should be noted that the image of the first free viewpoint video displayed by the terminal 100 may or may not change between time t0 and time t1. This is not limited in the embodiment of this application. For the change of the image of the first free viewpoint video displayed by the terminal 100 between time t0 and time t1, please refer to the description of the related content above. The details will not be described again here.

[0240] Between time t0 and time t1, the presentation manner of the barrage comments within the visible viewing range displayed by the terminal 100 may or may not change. For specific contents, please refer to the above description of the related contents. Details will not be described again here.

[0241] S1503: At time t1, the terminal 100 detects that the viewpoint switching operation is completed.

[0242] S1504: After time point t1, the terminal 100 requests a video file of the first free viewpoint video from viewpoint C at playback time point T3 from the media server.

[0243] In response, the media server returns to the terminal 100 the video file of the first free viewpoint video from viewpoint C at the playback time point T3.

[0244] The playback time point T3 is relative to the playback time point T2, for example, playback time point T3=playback time point T2, playback time point T3=playback time point T2-preset time offset value 1, or playback time point T3=playback time point T2+preset time offset value 2.

[0245] S1505: After time point t1, the terminal 100 further requests, from the barrage comment server, barrage comment data corresponding to playback time point T3 of the first free viewpoint video.

[0246] In response, the barrage comment server returns to the terminal 100 the barrage comment data corresponding to the playback time point T3 of the first free viewpoint video.

[0247] The playback time point T3 is relative to the playback time point T2, for example, playback time point T3=playback time point T2, playback time point T3=playback time point T2-preset time offset value 1, or playback time point T3=playback time point T2+preset time offset value 2.

[0248] It should be noted that this step is an optional step. In some other embodiments, playback time point T3=playback time point T2, or playback time point T3=playback time point T2-preset time offset value 1. The terminal 100 stores the danmaku comment data corresponding to the playback time point T3. In this case, the terminal 100 does not need to request the danmaku comment data at the playback time point T3 again from the danmaku comment server. Instead, as playback of the first free viewpoint video progresses, the terminal 100 requests, from the danmaku comment server, danmaku comment data that exists after the playback time point T3 or the playback time point T2.

[0249] It should further be noted that step S1505 may be performed after, before, or simultaneously with step S1504.

[0250] S1506: The terminal 100 recalculates the number of barrage comments outside the visible viewpoint range after the viewpoint has been switched.

[0251] In some embodiments, the terminal 100 further recalculates the offset angle between the viewpoint C and the viewpoint of each barrage comment within the visible viewpoint range, and determines the presentation manner of each barrage comment within the visible viewpoint range based on the recalculated offset angle.

[0252] S1507: The terminal 100 displays the video image of the first free viewpoint video at viewpoint C at playback time point T3, the barrage comments within the visible viewpoint range, and the number of barrage comments outside the visible viewpoint range.

[0253] For other contents of this embodiment, please refer to the relevant contents of steps S1001 to S1007 in Fig. 10A, and the details will not be described again here.

[0254] In some other embodiments, in step 1502, the number of danmaku comments outside the visible viewpoint range displayed by the terminal 100 may alternatively be changed. For example, when the terminal 100 continuously plays the first free viewpoint video between time t0 and time t1, the playback time point of the first free viewpoint video changes continuously. Correspondingly, the number of danmaku comments corresponding to the playback time point of the first free viewpoint video also changes, and the number of danmaku comments outside the visible viewpoint range changes accordingly.

[0255] For example, as shown in (1) of FIG. 15B, the terminal 100 plays a video image 1501 of a first free viewpoint video at a 60-degree viewpoint (i.e., viewpoint B). The playback interface 1501 includes barrage comments within the visible viewpoint range, such as barrage comment 1, barrage comment 2, and barrage comment 3. The playback interface 1501 further displays a control 1502 and a control 1503. The control 1502 is used to indicate the number of barrage comments located outside the visible viewpoint range and offset to the left (e.g., 2). The control 1503 is used to indicate the number of barrage comments located outside the visible viewpoint range and offset to the right (e.g., 2). When the first free viewpoint video is played up to 20:05, the terminal 100 detects a viewpoint switching operation performed by the user (e.g., sliding the viewpoint ruler to the right), and the terminal 100 continuously plays the first free viewpoint video and switches the playback viewpoint of the first free viewpoint video in real time based on the progress of the user performing the viewpoint switching operation. For example, in the process of a user performing a viewpoint switching operation, the terminal 100 displays a playback interface 1504 shown in (2) of FIG. 15B. In this case, the playback viewpoint of the first free viewpoint video is 65 degrees, and the playback time is 20:15. It can be understood that when the playback time changes to 20:15, the danmaku comments corresponding to the playback time also change accordingly. As shown in the playback interface 1504, the danmaku comments within the current visible viewpoint range change to danmaku comment 2, danmaku comment 3, and danmaku comment 4. The number of danmaku comments that are outside the visible viewpoint range and have a left offset changes to 3. The number of danmaku comments that are outside the visible viewpoint range and have a right offset changes to 5. In this case, it should be noted that the danmaku comments in the playback interface 1504 change accordingly as the playback time of the first free viewpoint video changes. In this case, the terminal 100 does not request new danmaku comment data from the danmaku comment server.

[0256] Furthermore, after terminal 100 detects that the user has completed the viewpoint switching operation, terminal 100 requests danmaku comment data corresponding to the current playback time point from the danmaku comment server. The content of the danmaku comments changes. As shown in playback interface 1505 of (3) in FIG. 15B, in this case, the playback viewpoint of the first free viewpoint video is 70 degrees, the playback time point is 20:15, and the danmaku comments within the visible viewpoint range change from danmaku comment 3 to danmaku comment 7. The number of danmaku comments that are outside the visible viewpoint range and offset to the left changes to 12. The number of danmaku comments that are outside the visible viewpoint range and offset to the right changes to 9.

[0257] In some other embodiments, in the process of performing a viewpoint switching operation by a user, the execution of the viewpoint switching operation may be paused. In this case, when the execution of the viewpoint switching operation is paused, the terminal 100 may re-request the barrage comment data corresponding to the current playback time point from the barrage comment server, and then update the contents of the barrage comments within the visible viewpoint range and the number of barrage comments outside the visible viewpoint range based on the reacquired barrage comment data.

[0258] For example, Figure 16 is a schematic flowchart of another method for displaying barrage comments in a process of switching viewpoints according to an embodiment of the present application. The method includes the following steps.

[0259] S1601: At time t0, the terminal 100 detects that the user has started to perform a viewpoint switching operation on the first free viewpoint video, where the viewpoint switching operation indicates switching the viewpoint of the first free viewpoint video.

[0260] In one example, the terminal 100 displays a video image of a first free viewpoint video at viewpoint B at playback time point T2 and danmaku comment data of the first free viewpoint video at playback time point T2. The danmaku comment corresponding to playback time point T2 includes the content of the danmaku comment within the visible viewpoint range and the number of danmaku comments outside the visible viewpoint range.

[0261] For example, if the terminal 100 detects that the user's finger remains on the viewpoint ruler for longer than a preset duration 2 (e.g., 3 seconds), the terminal 100 determines that the user has begun to perform a viewpoint switching operation and records the current time as time t0.

[0262] S1602: From time t0 to time t2, the terminal 100 keeps the number of displayed barrage comments outside the visible viewpoint range unchanged.

[0263] In some embodiments, between time t0 and time t1, the terminal 100 holds the number of danmaku comments outside the visible viewpoint range displayed by the terminal 100 at time t0. In other words, between time t0 and time t1, the number of danmaku comments outside the visible viewpoint range displayed by the terminal 100 remains unchanged. In some other embodiments, the number of danmaku comments outside the visible viewpoint range displayed by the terminal 100 may alternatively change. For example, when the terminal 100 continuously plays the first free viewpoint video between time t0 and time t1, the playback time point of the first free viewpoint video changes continuously. Correspondingly, the number of danmaku comments corresponding to the playback time point of the first free viewpoint video also changes, and the number of danmaku comments outside the visible viewpoint range changes accordingly.

[0264] It should be noted that between time t0 and time t1, the video image of the first free viewpoint video displayed by the terminal 100 may or may not change. This is not limited in the embodiment of this application. Between time t0 and time t1, the presentation manner of the danmaku comments within the visible viewpoint range displayed by the terminal 100 may not change. Obviously, the presentation manner of the danmaku comments within the visible viewpoint range displayed by the terminal 100 may change as an alternative. This is not limited in the embodiment of this application.

[0265] S1603: At time t2, the terminal 100 detects that the user has temporarily halted the execution of the viewpoint switching operation.

[0266] For example, after time t0, the terminal continuously detects that the user's finger has slid on the screen. When the terminal detects that the user's finger has not slid and has not been removed from the screen, the terminal determines that the user has paused the execution of the viewpoint switching operation and records the current time as time t2.

[0267] S1604: After time point t2, the terminal 100 requests a video file of the first free viewpoint video at viewpoint D at playback time point T4 from the media server.

[0268] In other words, after detecting that the user has paused the execution of the viewpoint switching operation, the terminal 100 determines the current playback viewpoint of the first free viewpoint video, for example, viewpoint D. The playback time point T4 is relative to the playback time point T2. For example, the playback time point T4=the playback time point T2, the playback time point T4=the playback time point T2-the preset time offset value 1, or the playback time point T4=the playback time point T2+the preset time offset value 2.

[0269] S1605: After time point t2, the terminal 100 further requests, from the barrage comment server, barrage comment data corresponding to playback time point T4 of the first free viewpoint video.

[0270] S1606: The terminal 100 recalculates the number of barrage comments outside the visible viewpoint range after the viewpoint has been switched.

[0271] In some embodiments, the terminal 100 further recalculates the offset angle between the viewpoint C and the viewpoint of each barrage comment within the visible viewpoint range, and determines the presentation manner of each barrage comment within the visible viewpoint range based on the recalculated offset angle.

[0272] S1607: Terminal 100 displays a video image of the first free viewpoint video from viewpoint D at playback time T4, and terminal 100 displays a video image of the first free viewpoint video from viewpoint C at playback time T3, the number of barrage comments within the visible viewpoint range, and the number of barrage comments outside the visible viewpoint.

[0273] S1608: At time t1, the terminal 100 detects that the viewpoint switching operation is completed.

[0274] For example, after time t2, the terminal continuously detects that the user's finger continuously slides on the screen. When the terminal detects that the user's finger is removed from the screen, the terminal determines that the user has completed the viewpoint switching operation and records the current time as time t1.

[0275] S1609: After time point t1, the terminal 100 requests a video file of the first free viewpoint video from viewpoint C at playback time point T3 from the media server.

[0276] S1610: After time t1, the terminal 100 further requests, from the barrage comment server, barrage comment data corresponding to playback time T3 of the first free viewpoint video.

[0277] In some examples, the playback time point T3 is relative to the playback time point T4, for example, playback time point T3=playback time point T4, playback time point T3=playback time point T4-preset time offset value 1, or playback time point T3=playback time point T4+preset time offset value 2.

[0278] S1611: The terminal 100 recalculates the number of barrage comments outside the visible viewpoint range after the viewpoint is switched.

[0279] S1612: Terminal 100 displays a video image of the first free viewpoint video from viewpoint C at playback time T3, and terminal 100 displays the video image of the first free viewpoint video from viewpoint C at playback time T3, the barrage comments within the visible viewpoint range, and the number of barrage comments outside the visible viewpoint.

[0280] For other details of steps S1601 to S1612, refer to the relevant details of steps S1001 to S1007 in Fig. 10A and the relevant details of steps S1501 to S1507 in Fig. 15A. Details will not be described again here.

[0281] It should be noted that the embodiment described in Fig. 16 is described by using an example in which the viewpoint switching operation is paused only once. It can be understood that the technical solution provided in the embodiment of this application is also applicable to the case in which the viewpoint switching operation is paused more than once.

[0282] 17 is a schematic flowchart of another method for displaying barrage comments of a free viewpoint video according to an embodiment of the present application. The procedure includes the following steps.

[0283] S1701: The terminal displays a first interface of a first free viewpoint video at a first playback point in time.

[0284] The first interface includes a video image of a first free viewpoint video at a first viewpoint and a first barrage comment.

[0285] For example, the first playback time point is related to the appearance time of the first danmaku comment. For example, the first playback time point is the appearance time of the first danmaku comment. Alternatively, the first playback time point is after the appearance time of the first danmaku comment and within the duration during which the first danmaku comment is continuously displayed.

[0286] A viewpoint offset between the third viewpoint corresponding to the first barrage comment and the first viewpoint is within a first preset angle range. The first preset angle range is a preset visible viewpoint range.

[0287] S1702: The terminal receives a first operation, and the first operation is an operation of switching the playback viewpoint of the first free viewpoint video from the first viewpoint to the second viewpoint.

[0288] S1073: In response to receiving the first operation, the terminal displays a second interface of the first free viewpoint video at the first playback point in time.

[0289] The second interface includes a video image of the first free viewpoint video at a second viewpoint, and the second interface does not include the first barrage comment.

[0290] The viewpoint offset between the third viewpoint corresponding to the first danmaku comment and the second viewpoint is outside the first preset angle range. In other words, the viewpoint offset between the third viewpoint corresponding to the first danmaku comment and the second viewpoint is not within the preset visible viewpoint range. It can be understood that the offset angle between the first viewpoint and the third viewpoint corresponding to the first danmaku comment is smaller than the offset angle between the second viewpoint and the third viewpoint corresponding to the first danmaku comment.

[0291] In one example, in the process of switching the viewpoint, the terminal automatically pauses the playback of the first free viewpoint video. Thus, before the user performs the first operation, the terminal plays the video image of the first free viewpoint video at the first viewpoint. When detecting that the user performs the first operation at the first playback time point, the terminal automatically pauses the playback of the first free viewpoint video. After the first operation is performed and the playback viewpoint of the first free viewpoint video is switched from the first viewpoint to the second viewpoint, the first free viewpoint video starts to be played from the first playback time point.

[0292] In another example, in the process of switching the viewpoint, the terminal continuously plays the first free viewpoint video. In this case, before performing the first operation, the user may first manually pause the playback of the first free viewpoint video. In this case, the first free viewpoint video stays at the first playback time point. Then, after the user performs the first operation and the terminal switches the playback viewpoint of the first free viewpoint video from the first viewpoint to the second viewpoint, the video image of the first free viewpoint video still stays at the first playback time point.

[0293] Alternatively, after the first operation is performed and the terminal switches the playback viewpoint of the first free viewpoint video from the first viewpoint to the second viewpoint, the playback starts from a playback point after the first playback point. Alternatively, the user may manually drag the playback progress bar to adjust the playback point back to the first playback point.

[0294] Optionally, the second interface includes a second barrage comment, and an offset angle between the second viewpoint and a viewpoint corresponding to the second barrage comment is within a first preset angle range. It may be understood that a playback time point of the second barrage comment is relative to the first playback time point.

[0295] In some other embodiments, the first interface displayed by the terminal further includes a first control, the first control being associated with a number of first danmaku comments, the number of first danmaku comments being the number of danmaku comments acquired when an offset angle between a viewpoint of the danmaku comments and a playback viewpoint of the terminal is within a second preset angle range, the second preset angle range not overlapping with the first preset angle range. The terminal displays a third interface upon receiving a third operation on the first control, the third interface including the first danmaku comment. In other words, the user may expand the danmaku comment hidden by the terminal by using the first control.

[0296] An embodiment of this application further provides an apparatus. The apparatus is included in a terminal, and the apparatus has a function of implementing the behavior of the terminal in any of the methods in the above embodiments. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function, such as a detection module or unit, a display module or unit, a determination module or unit, and a calculation module or unit.

[0297] An embodiment of this application further provides a computer storage medium containing computer instructions, which, when executed on a terminal, enable the terminal to perform any of the methods in the above embodiments.

[0298] An embodiment of this application further provides a computer program product, which, when executed on a computer, enables the computer to perform any of the methods in the above embodiments.

[0299] An embodiment of the present application further provides a graphical user interface on a terminal, the terminal including a display, a camera, a memory, and one or more processors, the one or more processors being configured to execute one or more computer programs stored in the memory, the graphical user interface including a graphical user interface displayed when the terminal executes any of the methods in the above embodiments.

[0300] It can be understood that in order to realize the above functions, the terminal etc. includes corresponding hardware structures and / or software modules for performing the functions. In combination with the examples described in the embodiments disclosed in this specification, those skilled in the art should easily recognize that the units, algorithms and steps may be realized by hardware or a combination of hardware and computer software in the embodiments of this application. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the functions described for each specific application, but the realization manner should not be considered to go beyond the scope of the embodiments of the present invention.

[0301] In the embodiment of this application, the terminal or the like may be divided into functional modules based on the above method examples. For example, each functional module may be obtained through division based on the corresponding function, or two or more functions may be integrated into one processing module. The integrated module may be realized in the form of hardware, or may be realized in the form of a software functional module. It should be noted that in the embodiment of the present invention, the division into modules is an example, and is merely a logical functional division, and other divisions may be used in the actual implementation.

[0302] The above description of the implementation manner allows those skilled in the art to clearly understand that for the purpose of convenient and concise description, the above division of functional modules is taken as an example for illustration. In practical application, the above functions can be allocated to different modules and realized as needed, that is, the internal structure of the device is divided into different functional modules to realize all or part of the above functions. For the detailed operation process of the above system, device and unit, please refer to the corresponding process in the above method embodiment, and the details will not be described again here.

[0303] The functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated units may be realized in the form of hardware or in the form of software functional units.

[0304] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of this application may be essentially realized in the form of a software product, or the part contributing to the prior art or all or part of the technical solutions. The computer software product is stored in a storage medium and includes some instructions for instructing a computer device (which may be a personal computer, a server or a network device) or a processor to execute all or part of the steps of the method according to the embodiments of this application. The above storage medium includes any medium that can store program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0305] The above description is merely a specific implementation of this application, and is not intended to limit the scope of protection of this application. Any variation or replacement within the technical scope disclosed in this application shall be included in the scope of protection of this application. Therefore, the scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A method for displaying barrage comments of a free viewpoint video, which is applied to a terminal, comprising: A step of displaying a first interface of a first free viewpoint video at a first playback time point by the terminal, the first interface including a video image of the first free viewpoint video at a first viewpoint and a first barrage comment; receiving a first operation by the terminal, the first operation being an operation of switching a playback viewpoint of the first free viewpoint video from the first viewpoint to a second viewpoint; a step of displaying, by the terminal, a second interface of the first free viewpoint video at the first playback time point in response to receiving the first operation, the second interface including a video image of the first free viewpoint video at the second viewpoint, and the second interface including no first barrage comment; The method includes:

2. 2. The method of claim 1, wherein an offset angle between the first viewpoint and a third viewpoint corresponding to the first barrage comment is smaller than an offset angle between the second viewpoint and the third viewpoint corresponding to the first barrage comment.

3. Prior to receiving the first operation by the terminal, The method of claim 1 , further comprising receiving, by the terminal, a second operation to pause playback of the first free viewpoint video.

4. 4. The method of claim 1, wherein an offset angle between the first viewpoint and a third viewpoint corresponding to the first danmaku comment is within a first preset angle range, and an offset angle between the second viewpoint and the third viewpoint corresponding to the first danmaku comment is outside the first preset angle range.

5. 5. The method of claim 4, wherein the second interface includes a second barrage comment, and an offset angle between the second viewpoint and a viewpoint corresponding to the second barrage comment is within the first preset angle range.

6. The method of claim 4 , wherein a sending time of the first barrage comment is relative to the first playback time, and a sending time of the second barrage comment is relative to the first playback time.

7. 7. The method according to claim 4, wherein the first interface displayed by the terminal further includes a first control, the first control being associated with a first number of danmaku comments, the first number of danmaku comments being the number of danmaku comments obtained when an offset angle between a viewpoint of the danmaku comments and a playback viewpoint of the terminal is within a second preset angle range, the second preset angle range not overlapping with the first preset angle range.

8. receiving, by the terminal, a third operation for the first control; a step of displaying a third interface by the terminal in response to receiving the third operation, the third interface including the first barrage comment; The method of claim 7 further comprising:

9. by the terminal, in response to receiving the first operation, after displaying a second interface of the first free viewpoint video at the first playback point in time; receiving a fourth operation by the terminal, the fourth operation being an operation of sending a third barrage comment; a step of transmitting data of the third barrage comment to a barrage comment server by the terminal in response to receiving the fourth operation, the data of the third barrage comment including a fourth viewpoint; The method of claim 1 , further comprising:

10. The method of claim 1 , wherein the first interface further comprises a fourth barrage of comments, the fourth barrage of comments corresponding to the first viewpoint.

11. The method of claim 10 , wherein in the first interface, a first style of the first barrage comments is different from a second style of the fourth barrage comments.

12. Before displaying a first interface of a first free viewpoint video at a first playback point in time by the terminal, 12. The method of claim 11, further comprising: determining, by the terminal, to display the first barrage comment in the first style in the first interface based on a first offset angle between the first viewpoint and a third viewpoint corresponding to the first barrage comment.

13. The method of claim 12, wherein when the first offset angle is a positive value, the first style is specifically an orientation in which the text direction of a barrage comment is obtained by rotating a first direction clockwise by a first angle, and the first direction is an orientation parallel to a first bezel of a screen of the terminal, or when the first offset angle is a negative value, the first style is specifically an orientation in which the text direction of a barrage comment is obtained by rotating a first direction counterclockwise by a first angle.

14. receiving a fifth operation for switching a viewpoint by the terminal; displaying a fifth interface by the terminal in the process of the fifth operation, the fifth interface and the first interface being the same; After the process of the fifth operation, displaying a sixth interface of the first free viewpoint video at a second playback point in time by the terminal. The method of any one of claims 1 to 13, further comprising:

15. receiving a sixth operation for switching a viewpoint by the terminal; displaying a seventh interface by the terminal in the sixth operation process, the seventh interface including a video image of the first free viewpoint video at a third playback point in time; after the process of the sixth operation, displaying an eighth interface of the first free viewpoint video at a second playback point in time by the terminal, where the third playback point in time is after the first playback point in time and before the second playback point in time; The method of any one of claims 1 to 13, further comprising:

16. 16. The method of claim 15, wherein the seventh interface further comprises a fifth barrage comment, and the first interface further comprises not including the fifth barrage comment.

17. 16. The method of claim 15, further comprising: the seventh interface further including the first barrage comment; and a display style of the first barrage comment in the seventh interface differing from a display style of the first barrage comment in the first interface.

18. 16. The method of claim 15, wherein the seventh interface further includes a sixth barrage comment, the sixth barrage comment having an appearance time after the first playback time.

19. A terminal including a processor, a memory, and a touch screen, A terminal, wherein the memory and the touch screen are coupled to the processor, the memory is configured to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, the terminal is enabled to execute the method for displaying barrage comments of free viewpoint video described in any one of claims 1 to 18.

20. A computer-readable storage medium containing computer instructions, comprising: A computer-readable storage medium, the computer instructions being executed on a terminal to enable the terminal to execute a method for displaying barrage comments in a free viewpoint video as described in any one of claims 1 to 18.

Images