Panoramic video display control method and device, computer equipment and storage medium

Abstract

The invention relates to a panoramic video display control method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a panoramic video and position information matched with each frame of panoramic image in the panoramic video; receiving a control instruction for performing first view angle control on the panoramic video, and determining direction control information and moving speed information indicated by the control instruction; calculating a first camera displacement corresponding to each frame of panoramic image at least according to the position information matched with each frame of panoramic image; an initial frame panoramic image in the panoramic video is determined, a target frame panoramic image is determined based on the first camera displacement, the initial frame panoramic image and the moving speed information corresponding to each frame panoramic image, and the target panoramic image is the next frame image after the initial frame panoramic image is displayed in the first view angle; and based on the initial frame panoramic image and the target frame panoramic image, adjusting a panoramic video displayed during the first view angle control. By adopting the method, the fluency can be improved, and the watching experience is ensured.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video image processing, and in particular to a display control method, device, computer equipment, and storage medium for panoramic videos. Background Art

[0002] With the development of panoramic video technology, the application scenarios of panoramic video are becoming more and more extensive. In order to enhance the viewing experience of panoramic videos, users usually use VR devices or other devices to experience panoramic videos immersively. For example, in the FPV (First Person View) application scenario, users can get an immersive experience.

[0003] However, when watching panoramic videos or in FPV scenarios, users usually use a controller to control the panoramic video to achieve the FPV driving experience. However, when using the controller to make adjustments, the panoramic video may be shot quickly or slowly during the shooting process, resulting in insufficient viewing smoothness, freezes, or unclear video changes, which affects the viewing experience. Summary of the invention

[0004] Based on this, it is necessary to provide a display control method, device, computer equipment, and storage medium for panoramic video to address the above technical problems.

[0005] In a first aspect, the present disclosure provides a method for controlling the display of a panoramic video. The method comprises:

[0006] Acquire a panoramic video and position information matching each panoramic image frame in the panoramic video;

[0007] receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction;

[0008] Calculate a first camera displacement corresponding to each panoramic image frame at least according to the position information matched by each panoramic image frame;

[0009] Determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image, and the moving speed information, wherein the target panoramic image is a next frame image after the initial frame panoramic image is displayed in the first viewing angle;

[0010] Based on the initial frame panoramic image and the target frame panoramic image, the panoramic video displayed during the first viewing angle control is adjusted.

[0011] In one of the embodiments, determining the target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information includes:

[0012] Determining, based on the movement speed information, a second camera displacement corresponding to a next panoramic image frame of the initial panoramic image frame;

[0013] A target frame panoramic image is determined according to a camera position corresponding to the initial frame panoramic image, a first camera displacement corresponding to each frame panoramic image, and the second camera displacement.

[0014] In one of the embodiments, determining the target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame panoramic image, and the second camera displacement includes:

[0015] Subtract the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference;

[0016] Determine, according to each panoramic image frame in the panoramic video and the initial panoramic image frame, a preselected panoramic image frame whose shooting time is after the initial panoramic image frame;

[0017] Determine a preselected frame panoramic image that satisfies the displacement difference according to the displacement difference and a first camera displacement corresponding to the preselected frame panoramic image;

[0018] A target frame panoramic image is determined according to the preselected frame panoramic image that satisfies the displacement difference.

[0019] In one embodiment, the method further comprises:

[0020] In response to the direction indicated by the direction control information being different from the initial viewing angle of the panoramic video, determining a target viewing angle corresponding to a next panoramic image frame of the initial panoramic image frame based on the direction control information, the movement speed information, and the initial viewing angle of the panoramic video;

[0021] Determine a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information;

[0022] Based on the initial frame panoramic image, the initial viewing angle, the target frame panoramic image and the target viewing angle, the panoramic video displayed during the first viewing angle control is adjusted.

[0023] In one embodiment, the position information includes: IMU information; the calculating the first camera displacement corresponding to each frame of panoramic image at least according to the position information matched by each frame of panoramic image includes:

[0024] Obtaining acceleration information from the IMU information that matches each frame of the panoramic image;

[0025] Adjusting the acceleration information to remove gravity from the acceleration information;

[0026] The acceleration data after removing gravity is integrated to calculate the speed when each frame of the panoramic image is taken;

[0027] Based on the speed when shooting each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is calculated.

[0028] In one embodiment, the location information includes: GPS information; and the calculating of the first camera displacement corresponding to each panoramic image frame at least according to the location information matched with each panoramic image frame includes:

[0029] According to the GPS information matched with each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is determined and calculated.

[0030] In one embodiment, the position information includes: IMU information, the IMU information includes a roll angle, a pitch angle, and a yaw angle; before matching the position information at least according to each frame of the panoramic image, the method further includes:

[0031] Construct a rotation matrix based on the roll angle, pitch angle and yaw angle of each frame of the panoramic image;

[0032] Using the rotation matrix to adjust the posture of each frame of the panoramic image;

[0033] In response to the presence of blank areas in the panoramic image after the posture adjustment, the blank areas are filled by using an interpolation method.

[0034] In a second aspect, the present disclosure further provides a display control device for panoramic video. The device comprises:

[0035] An information acquisition module, used to acquire a panoramic video and position information matching each panoramic image frame in the panoramic video;

[0036] An instruction parsing module, used for receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction;

[0037] A displacement calculation module, used to calculate a first camera displacement corresponding to each frame of the panoramic image at least according to the position information matched with each frame of the panoramic image;

[0038] an image determination module, configured to determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the movement speed information;

[0039] The video adjustment module is used to adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image and the target frame panoramic image.

[0040] In a third aspect, the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0041] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0042] In a fifth aspect, the present disclosure further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0043] In the above embodiments, when it is necessary to immersively watch a panoramic video from a first perspective, the panoramic video and the position information matching each panoramic image frame in the panoramic video can be obtained. Since the user needs to simulate the uniform or variable speed viewing of the panoramic image, a control instruction for the first perspective control of the panoramic video can also be received, the control instruction is parsed, and the direction control information and the moving speed information indicated by the control instruction are determined. At least according to the position information matching each panoramic image frame, the first camera displacement corresponding to each panoramic image frame is calculated. The initial panoramic image frame in the panoramic video is determined, and in response to the direction indicated by the direction control information being the same as the initial perspective of the panoramic video, the panoramic video may be stuck due to the variable speed processing of the panoramic video. Therefore, the target panoramic image frame can be determined based on the first camera displacement corresponding to each panoramic image frame, the initial panoramic image frame and the moving speed information. The target panoramic image frame is the next frame image after the initial panoramic image frame is displayed in the first perspective; based on the initial panoramic image frame and the target panoramic image frame, the panoramic video displayed during the first perspective control is adjusted to ensure smoothness and improve the viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of an application environment of a panoramic video display control method in one embodiment;

[0046] Figure 2 1 is a flow chart of a method for controlling the display of a panoramic video in one embodiment;

[0047] Figure 3 This is a schematic diagram of the process of step S208 in one embodiment;

[0048] Figure 4 Schematic diagram of the process of step S304 in one embodiment;

[0049] Figure 5 A schematic diagram of a process of controlling a panoramic video when viewing angles are different in one embodiment;

[0050] Figure 6 This is a schematic diagram of the process of step S206 in one embodiment;

[0051] Figure 7 A schematic diagram of a flow chart of a step of adjusting a panoramic image in one embodiment;

[0052] Figure 8 is a schematic block diagram of the structure of a display control device for panoramic video in one embodiment;

[0053] Fig. 9 Schematic diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0057] As described in the background technology, during the shooting of a panoramic video, due to the uneven speed of the camera during the shooting process, the photographer may stop moving, resulting in a still panoramic video, or the photographer may move at different speeds, resulting in the speed of the panoramic video being fast and slow. In FPV, since it simulates the first-person perspective, users are usually required to control the uniform speed or variable speed to watch the panoramic video. Therefore, it is necessary to remove the interference caused by uneven speed in the panoramic video. Since the speed of the camera is generally low during shooting, in FPV, the speed of simulating the first-person perspective will be much greater than the speed of the camera during shooting, and the frame rate and resolution of the panoramic video are generally large. If it is adjusted by fast forward or fast rewind playback, ordinary electronic devices cannot process high frame rates (mobile phones generally support decoding of a maximum frame rate of 60fps, but the required frame rate may even reach 6000fps), which will cause display freezes and affect the viewing experience.

[0058] The present disclosure provides a method for controlling the display of a panoramic video, which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can obtain the panoramic video and the position information matching each panoramic image frame in the panoramic video. The terminal 102 receives the control instruction for the first perspective control of the panoramic video, parses the control instruction, and determines the direction control information and the moving speed information indicated by the control instruction. The terminal 102 calculates the first camera displacement corresponding to each panoramic image frame at least according to the position information matching each panoramic image frame. The terminal 102 determines the initial frame panoramic image in the panoramic video. In response to the direction indicated by the direction control information in the terminal 102 being the same as the initial perspective of the panoramic video. The terminal 102 determines the target frame panoramic image based on the first camera displacement corresponding to each panoramic image frame, the initial frame panoramic image and the moving speed information. The terminal 102 adjusts the panoramic video displayed during the first perspective control based on the initial frame panoramic image and the target frame panoramic image. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The portable wearable devices may be smart watches, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0059] In one embodiment, Figure 2 As shown, a method for controlling the display of a panoramic video is provided, and the method is applied to Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0060] S202: Obtain a panoramic video and position information matching each panoramic image frame in the panoramic video.

[0061] Among them, the position information can be recorded by the IMU in the camera that shoots the panoramic video. The position information may include: IMU information, GPS information, etc. It can be understood that in some embodiments of the present disclosure, the device for shooting the panoramic video can be a panoramic camera. In addition, it should be noted that, since it is a simulated first-person perspective, the panoramic video involved in some embodiments of the present disclosure is usually a panoramic video that only has a forward direction.

[0062] Specifically, the terminal can be connected to a server or a panoramic camera to obtain the panoramic video stored in the server and the location information matching each panoramic image frame in the panoramic video, or obtain the panoramic video captured by the panoramic camera and the location information matching each panoramic image frame in the panoramic video.

[0063] S204, receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction.

[0064] The direction control information may generally be information indicating the display direction of the panoramic video. The moving speed information may generally be information indicating the viewing speed of the first perspective.

[0065] Specifically, due to the different types of terminals, the implementation methods of control instructions are also different. When the terminal is a head-mounted FPV device, the user can use a remote controller that matches the FPV to output control instructions, or use a virtual remote controller on the camera / mobile phone screen to output control instructions. When the terminal is a mobile phone or a device with an electronic screen, the user can click a virtual button on the screen to output control instructions. In some embodiments of the present disclosure, the implementation methods of control instructions are not limited. The control instructions are then parsed to determine the direction control information and the movement speed information.

[0066] S206: Calculate a first camera displacement corresponding to each frame of the panoramic image at least according to the position information matched by each frame of the panoramic image.

[0067] Specifically, the first camera displacement of the camera when shooting each panoramic image frame can be calculated according to the position information corresponding to each panoramic image frame. It should be noted that, since the movement speed of the camera is different during the shooting process, the first camera displacement corresponding to each panoramic image frame is also different, and the first camera displacement corresponding to each panoramic image frame can be calculated according to the position information matched to each panoramic image frame.

[0068] S208, determining an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determining a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image, and the movement speed information. The target panoramic image is a next frame image after the initial frame panoramic image is displayed in the first viewing angle.

[0069] The initial frame panoramic image may be a panoramic image of a certain frame in the panoramic video that is first viewed when the panoramic video is viewed from a first-person perspective. The initial viewing angle may be the direction in which the camera moves when the panoramic video is shot.

[0070] Specifically, the initial frame panoramic image in the panoramic video can be determined according to a preset number of frames. If there is no preset number of frames, the first frame image in the panoramic video can be determined as the initial frame panoramic image. In some embodiments of the present disclosure, there is no restriction on how the initial frame panoramic image is determined. When the direction indicated by the direction control information is the same as the initial viewing angle of the panoramic video, there is no need to adjust the viewing angle when watching the panoramic video from a first-person perspective, so the subsequent viewing angle can default to the initial viewing angle. The target frame panoramic image to be displayed in the next frame can be determined according to the first camera displacement, the initial frame panoramic image and the speed information corresponding to each frame of the panoramic image.

[0071] S210: Adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image and the target frame panoramic image.

[0072] Specifically, here, taking the display of two frames of images as an example, the initial viewing angle, the initial frame panoramic image and the target frame panoramic image can be displayed in sequence, thereby displaying a panoramic video. In addition, it should be noted that when more frames of images need to be displayed, the target frame panoramic image can be used as the initial frame panoramic image, and the next frame panoramic image of the target frame panoramic image can be determined according to the above steps, and the next frame panoramic image is continuously determined and displayed in this way, and the panoramic video displayed during the first viewing angle control is adjusted so that the speed of the panoramic video meets the moving speed information.

[0073] In addition, it should be noted that in the process of simulating ascent or descent, since the shooting position of the camera is usually fixed when shooting panoramic videos, the camera cannot rise or fall. The perspective can be rotated up and down, combined with the perspective conversion, to simulate the effect of moving up and down.

[0074] In the above-mentioned panoramic video display control method, when it is necessary to immersively watch the panoramic video from the first perspective, the panoramic video and the position information matching each panoramic image frame in the panoramic video can be obtained. Since the user needs to simulate the uniform or variable speed viewing of the panoramic image, a control instruction for the first perspective control of the panoramic video can also be received, the control instruction is parsed, and the direction control information and the moving speed information indicated by the control instruction are determined. At least according to the position information matching each frame of the panoramic image, the first camera displacement corresponding to each frame of the panoramic image is calculated. The initial frame panoramic image in the panoramic video is determined, and in response to the direction indicated by the direction control information being the same as the initial perspective of the panoramic video, since the speed change processing of the panoramic video may cause the panoramic video to freeze, the target frame panoramic image can be determined based on the first camera displacement corresponding to each frame of the panoramic image, the initial frame panoramic image and the moving speed information, and the target frame panoramic image is the next frame image after the initial frame panoramic image is displayed in the first perspective; based on the initial frame panoramic image and the target frame panoramic image, the panoramic video displayed during the first perspective control is adjusted to ensure smoothness and improve the viewing experience.

[0075] In one embodiment, Figure 3 As shown, the determining of the target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information includes:

[0076] S302: Determine, based on the movement speed information, a second camera displacement corresponding to a next panoramic image frame of the initial panoramic image frame.

[0077] Specifically, due to the different moving speed information, and in FPV, it is the first driving flight field of the drone that is simulated, and the user needs to simulate the uniform or variable speed viewing of the panoramic image, so it is necessary to eliminate the interference of uneven forward speed during shooting. If the panoramic video is simply adjusted in speed, there may be a problem of display freeze after the frame rate is high, so it is necessary to perform frame extraction to determine the image to be displayed for each frame, so as to achieve the effect of uniform speed or variable speed. Due to the different moving speed information, the panoramic image to be displayed for each frame is also different. Therefore, the second camera displacement of the initial frame panoramic image compared to the next frame panoramic image in the current first perspective can be calculated according to the moving speed information and the time to display each frame of the panoramic image.

[0078] S304: Determine a target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame panoramic image, and the second camera displacement.

[0079] The camera position corresponding to the initial frame panoramic image can be calculated through position information.

[0080] Specifically, since the second camera displacement has been determined, a panoramic image satisfying the second camera displacement may be determined according to the first camera displacement corresponding to each panoramic image frame after the initial panoramic image frame, and the panoramic image may be a target panoramic image frame.

[0081] In this embodiment, the second camera displacement corresponding to the next panoramic image of the initial panoramic image is determined by the moving speed information, and based on the first camera displacement of each panoramic image, the next frame of the image to be displayed can be determined without playing the video at double the speed, thereby ensuring the smoothness of the display.

[0082] In one embodiment, Figure 4 As shown, determining the target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame panoramic image, and the second camera displacement includes:

[0083] S402: Subtract the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference.

[0084] S404: Determine, according to each panoramic image frame in the panoramic video and the initial panoramic image frame, a preselected panoramic image frame whose shooting time is after the initial panoramic image frame.

[0085] S406 , determining a preselected frame panoramic image that satisfies the displacement difference according to the displacement difference and a first camera displacement corresponding to the preselected frame panoramic image.

[0086] S408: Determine a target frame panoramic image according to the preselected frame panoramic image satisfying the displacement difference.

[0087] Specifically, after the second camera displacement is calculated, since the direction indicated by the direction control information is the same as the initial viewing angle of the panoramic video, the second camera displacement can be subtracted from the camera position corresponding to the initial panoramic image to calculate the displacement difference, and the distance to be moved within one frame time is determined according to the displacement difference. Since the first-person perspective needs to be simulated, and the panoramic video is usually played continuously, it can be determined that in the panoramic video, the preselected frame panoramic image is shot after the initial frame panoramic image. Then, according to the first camera displacement and displacement difference corresponding to each preselected frame panoramic image, the preselected frame panoramic image that meets the displacement difference is determined. Then the preselected frame panoramic image that meets the displacement difference is determined as the target frame panoramic image.

[0088] In some exemplary embodiments, for example, there are 5 panoramic images in a panoramic video, the first camera displacement corresponding to the first panoramic image is 0.2m, the first camera displacement corresponding to the second panoramic image is 0m, the first camera displacement corresponding to the third panoramic image is 0.3m, the first camera displacement corresponding to the fourth panoramic image is 0.6m, and the first camera displacement corresponding to the fifth panoramic image is 0.5m. If the initial panoramic image is the third panoramic image and the second camera displacement is 0.8m, the preselected panoramic images may be the fourth and fifth panoramic images. The final target panoramic image may be the fourth panoramic image, and if the second camera displacement is 1.3, the target panoramic image may be the fifth panoramic image.

[0089] In this embodiment, the displacement difference is calculated by the displacement of the second camera, and then the target frame panoramic image can be accurately determined in the pre-selected frame panoramic image according to the displacement difference, thereby ensuring the smoothness of the first viewing angle.

[0090] In one embodiment, Figure 5 As shown, the method also includes:

[0091] S502, in response to the direction indicated by the direction control information being different from the initial viewing angle of the panoramic video, determining a target viewing angle corresponding to a next panoramic image frame of the initial panoramic image frame based on the direction control information, the moving speed information and the initial viewing angle of the panoramic video.

[0092] Specifically, when the direction indicated by the direction control information is different from the initial viewing angle of the panoramic video, since it is necessary to simulate the first-person perspective to watch the panoramic video, when the direction indicated by the direction control information is different from the initial viewing angle, it is necessary to determine the target viewing angle corresponding to the next frame of the panoramic image according to the direction control information, the moving speed information and the initial viewing angle. It should be noted that the moving speed information here generally refers to the moving speed information that matches the direction indicated by the direction control information.

[0093] S504: Determine a target panoramic image frame based on the first camera displacement corresponding to each panoramic image frame, the initial panoramic image frame and the moving speed information.

[0094] Specifically, since the panoramic video moves in the direction of the initial viewing angle during the shooting process, and the directions indicated by the initial viewing angle and the control information are different, in order to accurately simulate an image that matches the moving speed information in different directions, it is necessary to determine the speed of the moving speed information in the direction of the initial viewing angle according to the angle between the initial viewing angle of the initial panoramic image and the direction indicated by the direction control information, as well as the moving speed information, and determine the third camera displacement of the next frame of the panoramic image according to the speed in the direction of the initial viewing angle and the time of one frame, and then determine the target frame of the panoramic image according to the camera position corresponding to the initial frame of the panoramic image, the first camera displacement corresponding to each frame of the panoramic image, and the second camera displacement. (For the specific implementation here, please refer to the above embodiment, which will not be repeated here).

[0095] S506: Adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image, the initial viewing angle, the target frame panoramic image, and the target viewing angle.

[0096] Specifically, here, taking the display of two frames of images as an example, the initial frame panoramic image can be displayed according to the initial viewing angle, and then the target frame panoramic image can be displayed according to the target viewing angle, so as to display the panoramic video. In addition, it should be noted that when more frames of images need to be displayed, the target frame panoramic image can be used as the initial frame panoramic image, and the next frame panoramic image and the target viewing angle of the target frame panoramic image can be continuously determined according to the above steps S502-S504, and the next frame panoramic image is continuously determined and displayed.

[0097] In this embodiment, when the direction indicated by the direction control information is different from the initial viewing angle, the target viewing angle can be determined and adjusted according to the target viewing angle, thereby ensuring that the panoramic video can accurately display a viewing angle that matches the first viewing angle when the first viewing angle is controlled, thereby improving the viewing experience.

[0098] In one embodiment, Figure 6 As shown, the position information includes: IMU information. Wherein, IMU (Inertial Measurement Unit) is a device for measuring and reporting the acceleration, angular velocity and sometimes magnetic field of an object. IMU is usually composed of multiple sensors, including accelerometers, gyroscopes and magnetometers. The first camera displacement corresponding to each panoramic image frame is calculated at least based on the position information matched by each panoramic image frame, including:

[0099] S602: Acquire acceleration information in the IMU information that matches each frame of the panoramic image.

[0100] S604: Adjust the acceleration information to remove gravity from the acceleration information.

[0101] S606: Integrate the acceleration data after removing gravity to calculate the speed when taking each frame of the panoramic image.

[0102] S608: Calculate a first camera displacement corresponding to each panoramic image frame based on the speed at which each panoramic image frame is captured and the time of one frame.

[0103] Specifically, acceleration (usually a three-dimensional vector) and angular velocity (also usually a three-dimensional vector) data can be obtained from the IMU information. Acceleration data is usually in g (gravitational acceleration) units and needs to be converted to meters per second squared (m / s²). When calculating the speed of the camera, the effect of gravity needs to be removed. The acceleration vector can be rotated by attitude information (for example, using quaternions or Euler angles) so that it can be expressed in the world coordinate system. Integrate the acceleration data after removing gravity to calculate the speed. After obtaining the speed when each frame of the panoramic image is taken, the speed when each frame of the image is taken can be multiplied by the time of one frame to determine the first camera displacement corresponding to each frame of the image.

[0104] In this embodiment, the speed of shooting each panoramic image frame is calculated using the acceleration information in the IMU information, and the first camera displacement can be accurately calculated based on the speed, thereby ensuring the accuracy of subsequently determining the target frame panoramic image.

[0105] In one embodiment, the location information further includes: GPS information. GPS (Global Positioning System) information is location information provided by a satellite system. The main content of GPS information is location coordinates, including:

[0106] Longitude: indicates the position in the east-west direction, ranging from -180° to +180°. Latitude: indicates the position in the north-south direction, ranging from -90° to +90°. Altitude: the height relative to the sea level, usually in meters. The first camera displacement corresponding to each panoramic image frame is calculated based on at least the position information matched by each panoramic image frame, including:

[0107] According to the GPS information matched with each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is determined and calculated.

[0108] Specifically, the GPS information matching each panoramic image frame can be determined, and then the change of the GPS information of each panoramic image frame within one frame time can be determined according to the time of one frame, and the first camera displacement corresponding to each panoramic image frame can be determined according to the change of the GPS information.

[0109] In this embodiment, the first camera displacement corresponding to each frame of the panoramic image can be quickly determined through GPS information, thereby improving the speed of determining the target frame of the panoramic image.

[0110] In one embodiment, Figure 7 As shown, the IMU information includes roll angle, pitch angle and yaw angle. Since in the process of shooting panoramic video, the panoramic video may not be level with the horizontal plane. Therefore, the posture can be adjusted according to the IMU information when shooting each frame, so that the horizontal coordinate of each frame of the panoramic image is parallel to the horizontal line. Thus, the horizontal posture feature of the first perspective is simulated. Before at least matching the position information of each frame of the panoramic image, the method also includes:

[0111] S702: construct a rotation matrix according to the roll angle, pitch angle and yaw angle of each frame of the panoramic image.

[0112] S704: Use the rotation matrix to adjust the posture of each frame of the panoramic image.

[0113] S706: In response to a blank area existing in the panoramic image after the posture adjustment, fill the blank area by using an interpolation method.

[0114] Specifically, obtain IMU data: obtain the attitude information corresponding to each frame of the panoramic image from the IMU sensor, which usually includes roll angle (roll), pitch angle (pitch) and yaw angle (yaw). Calculate the rotation matrix based on the IMU data. For a given roll angle and pitch angle, the following formula can be used to construct the rotation matrix: ,in, , , are the rotation matrices around the x, y, and z axes, respectively. For each panoramic image, apply the calculated rotation matrix to adjust the image's pose. You can use an image processing library such as OpenCV to achieve this. When adjusting the panoramic image, blank areas may appear. You can use an interpolation method such as bilinear interpolation to fill these areas.

[0115] In this embodiment, each frame of the panoramic image is adjusted through IMU information to ensure that the posture of the panoramic video is accurate when watching the panoramic video from a first-person perspective. In addition, blank areas may appear during the adjustment process, and interpolation methods (such as bilinear interpolation) can be used to fill these areas to ensure the integrity of the panoramic image.

[0116] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0117] Based on the same inventive concept, the embodiment of the present disclosure also provides a panoramic video display control device for implementing the panoramic video display control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more panoramic video display control devices provided below can refer to the limitations of the panoramic video display control method above, and will not be repeated here.

[0118] In one embodiment, Figure 8 As shown, a panoramic video display control device 800 is provided, comprising: an information acquisition module 802, an instruction parsing module 804, a displacement calculation module 806, an image determination module 808 and a video adjustment module 810, wherein:

[0119] An information acquisition module 802 is used to acquire a panoramic video and position information matching each panoramic image frame in the panoramic video;

[0120] The instruction parsing module 804 is used to receive a control instruction for controlling the first viewing angle of the panoramic video, parse the control instruction, and determine the direction control information and the moving speed information indicated by the control instruction;

[0121] A displacement calculation module 806, configured to calculate a first camera displacement corresponding to each panoramic image frame at least according to the position information matched with each panoramic image frame;

[0122] An image determination module 808 is configured to determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the movement speed information;

[0123] The video adjustment module 810 is used to adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image and the target frame panoramic image.

[0124] In one embodiment of the apparatus, the image determination module 808 includes:

[0125] A displacement determination module, configured to determine, based on the movement speed information, a second camera displacement corresponding to a next panoramic image frame of the initial panoramic image frame;

[0126] The determination submodule is used to determine the target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame panoramic image, and the second camera displacement.

[0127] In one embodiment of the device, the determination submodule is further used to subtract the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference; determine a preselected frame panoramic image whose shooting time is after the initial frame panoramic image based on each frame panoramic image in the panoramic video and the initial frame panoramic image; determine a preselected frame panoramic image that satisfies the displacement difference based on the displacement difference and the first camera displacement corresponding to the preselected frame panoramic image; and determine a target frame panoramic image based on the preselected frame panoramic image that satisfies the displacement difference.

[0128] In one embodiment of the device, the device further comprises:

[0129] A target viewing angle determination module is used to determine a target viewing angle corresponding to a next panoramic image of the initial frame panoramic image based on the direction control information, the moving speed information and the initial viewing angle of the panoramic video in response to the direction indicated by the direction control information being different from the initial viewing angle of the panoramic video.

[0130] The image determination module 808 is further configured to determine a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information.

[0131] The video adjustment module 810 is further used to adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image, the initial viewing angle, the target frame panoramic image and the target viewing angle.

[0132] In one embodiment of the device, the position information includes: IMU information; the displacement calculation module 806 is also used to obtain acceleration information in the IMU information matching each frame of the panoramic image; adjust the acceleration information to remove gravity from the acceleration information; integrate the acceleration data after removing gravity to calculate the speed when taking each frame of the panoramic image; based on the speed when taking each frame of the panoramic image and the time of one frame, calculate the first camera displacement corresponding to each frame of the panoramic image.

[0133] In one embodiment of the device, the location information includes: GPS information; the displacement calculation module 806 is further used to determine and calculate the first camera displacement corresponding to each frame of the panoramic image based on the GPS information matched with each frame of the panoramic image and a frame time.

[0134] In one embodiment of the device, the position information includes: IMU information, the IMU information includes roll angle, pitch angle and yaw angle. The device also includes: an image adjustment module, which is used to construct a rotation matrix according to the roll angle, pitch angle and yaw angle of each frame of the panoramic image; use the rotation matrix to adjust the posture of each frame of the panoramic image; in response to the presence of blank areas in the panoramic image after the posture is adjusted, fill the blank areas using interpolation.

[0135] Each module in the above-mentioned panoramic video display control device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0136] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a display control method for a panoramic video is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse.

[0137] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0140] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0141] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.

Claims

1. A method for controlling the display of a panoramic video, characterized in that: The method comprises: Acquire a panoramic video and position information matching each panoramic image frame in the panoramic video; receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction; Calculate a first camera displacement corresponding to each frame of the panoramic image at least according to the position information matched by each frame of the panoramic image; Determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image, and the moving speed information, wherein the target panoramic image is a next frame image after the initial frame panoramic image is displayed in the first viewing angle; Based on the initial frame panoramic image and the target frame panoramic image, the panoramic video displayed during the first viewing angle control is adjusted.

2. The method according to claim 1, characterized in that: The step of determining a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information comprises: Determining, based on the movement speed information, a second camera displacement corresponding to a next panoramic image frame of the initial panoramic image frame; A target frame panoramic image is determined according to a camera position corresponding to the initial frame panoramic image, a first camera displacement corresponding to each frame panoramic image, and the second camera displacement.

3. The method according to claim 2, characterized in that The step of determining a target frame panoramic image according to a camera position corresponding to the initial frame panoramic image, a first camera displacement corresponding to each frame panoramic image, and a second camera displacement includes: Subtract the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference; Determine, according to each panoramic image frame in the panoramic video and the initial panoramic image frame, a preselected panoramic image frame whose shooting time is after the initial panoramic image frame; Determine a preselected frame panoramic image that satisfies the displacement difference according to the displacement difference and a first camera displacement corresponding to the preselected frame panoramic image; A target frame panoramic image is determined according to the preselected frame panoramic image that satisfies the displacement difference.

4. The method according to claim 1, characterized in that: The method further comprises: In response to the direction indicated by the direction control information being different from the initial viewing angle of the panoramic video, determining a target viewing angle corresponding to a next panoramic image frame of the initial panoramic image frame based on the direction control information, the movement speed information, and the initial viewing angle of the panoramic video; Determine a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information; Based on the initial frame panoramic image, the initial viewing angle, the target frame panoramic image and the target viewing angle, the panoramic video displayed during the first viewing angle control is adjusted.

5. The method according to claim 1, characterized in that The position information includes: IMU information; the calculating the first camera displacement corresponding to each frame of panoramic image at least according to the position information matched by each frame of panoramic image includes: Obtaining acceleration information from the IMU information that matches each frame of the panoramic image; Adjusting the acceleration information to remove gravity from the acceleration information; The acceleration data after removing gravity is integrated to calculate the speed when each frame of the panoramic image is taken; Based on the speed when shooting each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is calculated.

6. The method according to claim 1, characterized in that The position information includes: GPS information; the calculating the first camera displacement corresponding to each panoramic image frame at least according to the position information matched by each panoramic image frame includes: According to the GPS information matched with each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is determined and calculated.

7. The method according to any one of claims 1 to 6, characterized in that The position information includes: IMU information, and the IMU information includes a roll angle, a pitch angle, and a yaw angle; before the position information matched at least according to each frame of the panoramic image, the method further includes: Construct a rotation matrix based on the roll angle, pitch angle and yaw angle of each frame of the panoramic image; Using the rotation matrix to adjust the posture of each frame of the panoramic image; In response to the presence of blank areas in the panoramic image after the posture adjustment, the blank areas are filled by using an interpolation method.

8. A panoramic video display control device, characterized in that: The device comprises: An information acquisition module, used to acquire a panoramic video and position information matching each panoramic image frame in the panoramic video; An instruction parsing module, used for receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction; A displacement calculation module, used to calculate a first camera displacement corresponding to each frame of the panoramic image at least according to the position information matched with each frame of the panoramic image; an image determination module, configured to determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the movement speed information; The video adjustment module is used to adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image and the target frame panoramic image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images