Display control method, device, computer device, and storage medium for panoramic video

Abstract

The present disclosure relates to a display control method, device, computer device, and storage medium for panoramic videos. The method includes: obtaining a panoramic video and position information matching each frame of panoramic image in the panoramic video; receiving a control instruction for first perspective control of the panoramic video, and determining direction control information and moving speed information indicated by the control instruction; calculating a first camera displacement corresponding to each frame of panoramic image at least according to the position information matching each frame of panoramic image; determining an initial frame panoramic image in the panoramic video, and determining a target frame panoramic image based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the moving speed information, where the target panoramic image is the next frame image after the initial frame panoramic image is displayed in the first perspective; and adjusting the panoramic video displayed during the first perspective control based on the initial frame panoramic image and the target frame panoramic image. Using this method can improve fluency and ensure the viewing experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of video image processing, and particularly to a method, an apparatus, a computer device, and a storage medium for displaying and controlling a panoramic video. Background Art

[0002] With the development of panoramic video technology, the application scenarios of panoramic videos are becoming more and more extensive. To enhance the viewing experience of panoramic videos, users usually use VR devices or other devices to immerse themselves in the panoramic videos. For example, in the FPV (First Person View) application scenario, users can obtain an immersive experience.

[0003] However, during the process of watching a panoramic video or in the FPV scenario, users usually use a controller to control the panoramic video to achieve the experience of FPV driving. However, when using the controller to make adjustments, due to the fact that the panoramic video may be shot quickly or slowly during the shooting process, the viewing fluency is insufficient, and there may be stuttering or the video change is not obvious, thus affecting the viewing experience. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method, an apparatus, a computer device, and a storage medium for displaying and controlling a panoramic video.

[0005] In a first aspect, the present disclosure provides a method for displaying and controlling a panoramic video. The method includes:

[0006] Obtain a panoramic video and position information matching each frame of panoramic image in the panoramic video;

[0007] Receive a control instruction for controlling the first perspective of the panoramic video, parse the control instruction, and determine the direction control information and moving speed information indicated by the control instruction;

[0008] Calculate the first camera displacement corresponding to each frame of panoramic image at least according to the position information matching each frame of panoramic image;

[0009] Determine the initial frame panoramic image in the panoramic video. In response to the direction indicated by the direction control information being the same as the initial perspective of the panoramic video, based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the moving speed information, determine the target frame panoramic image, where the target panoramic image is the next frame image after displaying the initial frame panoramic image in the first perspective;

[0010] Adjust the panoramic video displayed during the first perspective control based on the initial frame panoramic image and the target frame panoramic image.

[0011] In one embodiment, determining the target panoramic image based on the first camera displacement corresponding to each frame of the panoramic image, the initial frame panoramic image, and the moving speed information includes:

[0012] Based on the moving speed information, determining the second camera displacement corresponding to the next frame panoramic image of the initial frame panoramic image;

[0013] According to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame of the panoramic image, and the second camera displacement, determining the target frame panoramic image.

[0014] In one embodiment, the determining the target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame of the panoramic image, and the second camera displacement includes:

[0015] Subtracting the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference;

[0016] According to each frame of the panoramic image in the panoramic video and the initial frame panoramic image, determining a preselected frame panoramic image whose shooting time is after the initial frame panoramic image;

[0017] According to the displacement difference and the first camera displacement corresponding to the preselected frame panoramic image, determining the preselected frame panoramic image that satisfies the displacement difference;

[0018] According to the preselected frame panoramic image that satisfies the displacement difference, determining the target frame panoramic image.

[0019] In one embodiment, the method further includes:

[0020] In response to the direction indicated by the direction control information being different from the initial viewing angle of the panoramic video, based on the direction control information, the moving speed information, and the initial viewing angle of the panoramic video, determining the target viewing angle corresponding to the next frame panoramic image of the initial frame panoramic image;

[0021] Based on the first camera displacement corresponding to each frame of the panoramic image, the initial frame panoramic image, and the moving speed information, determining the target frame panoramic image;

[0022] Based on the initial frame panoramic image, the initial viewing angle, the target frame panoramic image, and the target viewing angle, adjusting the panoramic video displayed during the first viewing angle control.

[0023] In one embodiment, the position information includes: IMU information; calculating a first camera displacement corresponding to each frame of panoramic image based on at least the position information matched with each frame of panoramic image includes:

[0024] Obtain the acceleration information in the IMU information matched with each frame of panoramic image;

[0025] Adjust the acceleration information to remove the gravity in the acceleration information;

[0026] Integrate the acceleration data after removing gravity to calculate the speed when each frame of panoramic image is captured;

[0027] Calculate a first camera displacement corresponding to each frame of panoramic image based on the speed when each frame of panoramic image is captured and the time of one frame.

[0028] In one embodiment, the position information includes: GPS information; calculating a first camera displacement corresponding to each frame of panoramic image based on at least the position information matched with each frame of panoramic image includes:

[0029] Determine a first camera displacement corresponding to each frame of panoramic image according to the GPS information matched with each frame of panoramic image and the time of one frame.

[0030] In one embodiment, the position information includes: IMU information, and the IMU information includes roll angle, pitch angle and yaw angle; before calculating at least based on the position information matched with each frame of panoramic image, the method further includes:

[0031] Construct a rotation matrix according to the roll angle, pitch angle and yaw angle of each frame of panoramic image;

[0032] Adjust the pose of each frame of panoramic image by using the rotation matrix;

[0033] In response to the existence of blank areas in the panoramic image after pose adjustment, fill the blank areas by using the interpolation method.

[0034] In a second aspect, the present disclosure also provides a display control device for panoramic video. The device includes:

[0035] An information acquisition module, configured to acquire a panoramic video and position information matched with each frame of panoramic image in the panoramic video;

[0036] An instruction parsing module, configured to receive a control instruction for first perspective control of the panoramic video, parse the control instruction, and determine direction control information and moving speed information indicated by the control instruction;

[0037] A displacement calculation module, configured to calculate a first camera displacement corresponding to each panoramic image at least according to the position information matched with each frame of panoramic image.

[0038] An image determination module, configured to determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the movement speed information.

[0039] A video adjustment module, configured to adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image and the target frame panoramic image.

[0040] In a third aspect, the present disclosure further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0041] In a fourth aspect, the present disclosure further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0042] In a fifth aspect, the present disclosure further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0043] In the above embodiments, when immersive viewing of a panoramic video from a first viewing angle is required, a panoramic video and position information matched with each frame of panoramic image in the panoramic video can be obtained. Since the user needs to simulate a uniform or variable-speed viewing of the panoramic image by themselves, a control instruction for first viewing angle control of the panoramic video can also be received, the control instruction can be parsed, and the direction control information and movement speed information indicated by the control instruction can be determined. At least according to the position information matched with each frame of panoramic image, a first camera displacement corresponding to each frame of panoramic image is calculated. The initial frame panoramic image in the panoramic video is determined. In response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, since variable-speed processing of the panoramic video may cause the panoramic video to freeze, a target frame panoramic image can be determined based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the movement speed information. The target panoramic image is the next frame image after the initial frame panoramic image is displayed in the first viewing angle; based on the initial frame panoramic image and the target frame panoramic image, the panoramic video displayed during the first viewing angle control is adjusted, so as to ensure smoothness and improve the viewing experience. Brief Description of the Drawings

[0044] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 Schematic diagram of the application environment of the panoramic video display control method in one embodiment;

[0046] Figure 2 Schematic flowchart of the panoramic video display control method in one embodiment;

[0047] Figure 3 Schematic flowchart of step S208 in one embodiment;

[0048] Figure 4 Schematic flowchart of step S304 in one embodiment;

[0049] Figure 5 Schematic flowchart of controlling the panoramic video when the viewing angles are different in one embodiment;

[0050] Figure 6 Schematic flowchart of step S206 in one embodiment;

[0051] Figure 7 Schematic flowchart of the step of adjusting the panoramic image in one embodiment;

[0052] Figure 8 Schematic block diagram of the structure of the panoramic video display control device in one embodiment;

[0053] Figure 9 Schematic internal structure diagram of a computer device in one embodiment. Specific Embodiments

[0054] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the following further elaborates on the present disclosure in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0055] It should be noted that the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of this article are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or equipment.

[0056] In this article, the term "and / or" is merely a relational description of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0057] As described in the background art, during the shooting of panoramic videos, due to the uneven speed during the camera shooting process, the shooter may stop moving, resulting in the panoramic video being stationary, or the shooter may move at different speeds, resulting in the speed of the panoramic video being sometimes fast and sometimes slow. In FPV, since it is an analog first-person perspective, it is usually necessary for the user to control the viewing of the panoramic video at a constant speed or variable speed. Therefore, it is necessary to remove the interference caused by uneven speed in the panoramic video. Since the speed of the camera during shooting is generally low, and in FPV, the speed of the analog first-person perspective is much greater than the speed of the camera during shooting, and the frame rate and resolution of the panoramic video are generally large. If adjusted by fast-forwarding or rewinding the playback method, ordinary electronic devices cannot process high frame rates (mobile phones generally support a maximum decoding frame rate of 60fps, but the required frame rate may even reach 6000fps), which will cause display stuttering and affect the viewing experience.

[0058] Embodiments of the present disclosure provide a method for displaying and controlling panoramic videos, which can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed in the cloud or other network servers. The terminal 102 can obtain the panoramic video and the position information matching each panoramic image in the panoramic video. The terminal 102 receives a control instruction for first-person perspective control of the panoramic video, parses the control instruction, and determines the direction control information and the moving speed information indicated by the control instruction. The terminal 102 calculates at least the first camera displacement corresponding to each frame of the panoramic image according to the position information matching each frame of the panoramic image. The terminal 102 determines the initial frame panoramic image in the panoramic video. In response to the direction indicated by the direction control information in the terminal 102 being the same as the initial perspective of the panoramic video. The terminal 102 determines the target frame panoramic image based on the first camera displacement corresponding to each frame of the panoramic image, the initial frame panoramic image, and the moving speed information. The terminal 102 adjusts the panoramic video displayed during the first-person perspective control based on the initial frame panoramic image and the target frame panoramic image. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The portable wearable device can be a smart watch, a head-mounted device, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0059] In one embodiment, as Figure 2 shown, a method for displaying and controlling a panoramic video is provided. Taking the terminal 102 in Figure 1 as an example, the method includes the following steps:

[0060] S202, obtaining a panoramic video and position information matching each panoramic image in the panoramic video.

[0061] Among them, the position information can be recorded by the IMU in the camera that shoots the panoramic video. The position information can include: IMU information, GPS information, etc. It can be understood that in some embodiments of the present disclosure, the device for shooting the panoramic video can be a panoramic camera. In addition, it should be noted that since it is simulating the first-person perspective, the panoramic videos involved in some embodiments of the present disclosure usually only have a panoramic video in the forward direction.

[0062] Specifically, the terminal can be connected to the server or the panoramic camera to obtain the panoramic video stored in the server and the position information matching each panoramic image in the panoramic video. Or obtain the panoramic video captured by the panoramic camera and the position information matching each panoramic image in the panoramic video.

[0063] S204, receive a control instruction for first - person view control of the panoramic video, parse the control instruction, and determine the direction control information and the moving speed information indicated by the control instruction.

[0064] Among them, the direction control information can generally be information indicating the display direction of the panoramic video. The moving speed information is generally information indicating the viewing speed of the first - person view.

[0065] Specifically, due to different types of terminals, the implementation methods of control instructions are also different. When the terminal is a head - mounted FPV device, the user can use a remote controller matching the FPV to output control instructions, or use a virtual remote controller on the camera / phone screen to output control instructions. When the terminal is a mobile phone or a device with an electronic screen, the user can click on the virtual buttons on the screen to output control instructions. In some embodiments of the present disclosure, the implementation method of the control instruction is not limited. Then, the control instruction is parsed to determine the direction control information and the moving speed information.

[0066] S206, calculate the first camera displacement corresponding to each frame of the panoramic image at least according to the position information matching each frame of the panoramic image.

[0067] Specifically, the first camera displacement of the camera when shooting a certain frame of the panoramic image can be calculated according to the position information corresponding to each frame of the panoramic image. It should be noted that since the moving speed of the camera is different during the shooting process, the first camera displacement corresponding to each frame of the panoramic image is also different, and the first camera displacement corresponding to each frame of the panoramic image can be calculated according to the position information matching each frame of the panoramic image.

[0068] S208, determine the initial - frame panoramic image in the panoramic video. In response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, based on the first camera displacement corresponding to each frame of the panoramic image, the initial - frame panoramic image, and the moving speed information, determine the target - frame panoramic image. The target panoramic image is the next frame after the initial - frame panoramic image is displayed in the first - person view.

[0069] Among them, the initial - frame panoramic image can generally be a certain frame of the panoramic image in the panoramic video that is first viewed when viewing the panoramic video from the first - person perspective. The initial viewing angle can generally be the moving direction of the camera when shooting the panoramic video.

[0070] Specifically, the initial frame panoramic image in the panoramic video can be determined according to a preset number of frames. If there is no preset number of frames, the first frame image in the panoramic video can be determined as the initial frame panoramic image. In some embodiments of the present disclosure, there is no limitation on how the initial frame panoramic image is determined. When the direction indicated by the direction control information is the same as the initial viewing angle of the panoramic video, there is no need to adjust the viewing angle when viewing the panoramic video from the first-person perspective later. Therefore, the subsequent viewing angle can be defaulted to the initial viewing angle. The target frame panoramic image to be displayed in the next frame can be determined according to the first camera displacement, the initial frame panoramic image, and the speed information corresponding to each frame of panoramic image.

[0071] S210. Based on the initial frame panoramic image and the target frame panoramic image, adjust the panoramic video displayed during the first perspective control.

[0072] Specifically, here, taking the display of two frames of images as an example, the panoramic video can be displayed in sequence according to the initial viewing angle, the initial frame panoramic image, and the target frame panoramic image. In addition, it should be noted that when more frames of images need to be displayed, the target frame panoramic image can be used as the initial frame panoramic image, and the next frame of the target frame panoramic image can be continuously determined according to the above steps, and so on, to continuously determine and display the next frame of the panoramic image, and adjust the panoramic video displayed during the first perspective control to make the speed of the panoramic video meet the movement speed information.

[0073] In addition, it should be noted that during the simulation of ascending or descending, since the shooting position of the camera is usually fixed when shooting the panoramic video and the camera cannot actually ascend or descend, the effect of moving up and down can be simulated by rotating the viewing angle up and down and combining it with the viewing angle conversion.

[0074] In the above display control method of panoramic video, when immersive viewing of the panoramic video from the first perspective is required, the panoramic video and position information matching each frame of panoramic image in the panoramic video can be obtained. Since the user needs to simulate uniform or variable-speed viewing of the panoramic image by himself / herself, a control instruction for first-perspective control of the panoramic video can also be received, the control instruction can be parsed, and the direction control information and moving speed information indicated by the control instruction can be determined. At least based on the position information matching each frame of panoramic image, the first camera displacement corresponding to each frame of panoramic image is calculated. The initial frame panoramic image in the panoramic video is determined. In response to the direction indicated by the direction control information being the same as the initial perspective of the panoramic video, since variable-speed processing of the panoramic video may cause the panoramic video to freeze, the target frame panoramic image can be determined based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the moving speed information. The target panoramic image is the next frame image after displaying the initial frame panoramic image in the first perspective; based on the initial frame panoramic image and the target frame panoramic image, the panoramic video displayed during first-perspective control is adjusted, thereby ensuring smoothness and improving the viewing experience.

[0075] In one embodiment, as Figure 3 shown, determining the target frame panoramic image based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the moving speed information includes:

[0076] S302, based on the moving speed information, determine the second camera displacement corresponding to the next frame panoramic image of the initial frame panoramic image.

[0077] Specifically, due to different moving speed information, and in FPV, it is to simulate the first-person flight vision of a drone, and the user needs to simulate uniform or variable-speed viewing of the panoramic image by himself / herself. Therefore, it is necessary to exclude the interference of uneven forward speed during shooting. If the panoramic video is simply speed-adjusted, there may be a problem that the frame rate is high and the display freezes after speed adjustment. Therefore, frame extraction processing needs to be performed to determine each frame of image to be displayed, so as to achieve the effect of uniform or variable speed. Due to different moving speed information, the panoramic images to be displayed for each frame determined are also different. Therefore, the second camera displacement of the initial frame panoramic image compared to the next frame panoramic image in the current first perspective can be calculated according to the moving speed information and the time for displaying each frame of panoramic image.

[0078] S304, according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame of panoramic image, and the second camera displacement, determine the target frame panoramic image.

[0079] Among them, the camera position corresponding to the initial frame panoramic image can be calculated through position information.

[0080] Specifically, since the second camera displacement has been determined, therefore, according to the first camera displacement corresponding to each panoramic image after the initial frame panoramic image, the panoramic image that satisfies the second camera displacement can be determined, and this panoramic image can be the target frame panoramic image.

[0081] In this embodiment, through the moving speed information, the second camera displacement corresponding to the next frame panoramic image of the initial frame panoramic image is determined, and according to the first camera displacement of each frame panoramic image, the image to be displayed in the next frame can be determined without video speed-up playback, thereby ensuring the smoothness of the display.

[0082] In one embodiment, as Figure 4 shown, determining the target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame panoramic image, and the second camera displacement includes:

[0083] S402, subtracting the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference.

[0084] S404, according to each frame panoramic image in the panoramic video and the initial frame panoramic image, determine the preselected frame panoramic image whose shooting time is after the initial frame panoramic image.

[0085] S406, according to the displacement difference and the first camera displacement corresponding to the preselected frame panoramic image, determine the preselected frame panoramic image that satisfies the displacement difference.

[0086] S408, according to the preselected frame panoramic image that satisfies the displacement difference, determine the target frame panoramic image.

[0087] Specifically, after the second camera displacement is calculated, since the direction indicated by the direction control information is the same as the initial view angle of the panoramic video, therefore, the second camera displacement can be subtracted from the camera position corresponding to the initial panoramic image, so as to calculate the displacement difference, and the distance to be moved within one frame time can be determined according to the displacement difference. Since the first-person view needs to be simulated and the panoramic video is usually played continuously, therefore, in the panoramic video, the preselected frame panoramic image whose shooting time is after the initial frame panoramic image can be determined. Then, according to the first camera displacement and the displacement difference corresponding to each preselected frame panoramic image, the preselected frame panoramic image that satisfies the displacement difference is determined. Then, the preselected frame panoramic image that satisfies the displacement difference is determined as the target frame panoramic image.

[0088] In some exemplary embodiments, for example, there are 5 panoramic images in a panoramic video. The first camera displacement corresponding to the first panoramic image is 0.2 m, the first camera displacement corresponding to the second panoramic image is 0 m, the first camera displacement corresponding to the third panoramic image is 0.3 m, the first camera displacement corresponding to the fourth panoramic image is 0.6 m, and the first camera displacement corresponding to the fifth panoramic image is 0.5 m. If the initial frame panoramic image is the third panoramic image and the second camera displacement is 0.8 m, then the preselected frame panoramic images can be the fourth panoramic image and the fifth panoramic image. The finally determined target frame panoramic image can be the fourth panoramic image. If the second camera displacement is 1.3, the determined target frame panoramic image can be the fifth panoramic image.

[0089] In this embodiment, the displacement difference is calculated through the second camera displacement, and then the target frame panoramic image can be accurately determined from the preselected frame panoramic images according to the displacement difference, thereby ensuring the smoothness of the first-person perspective viewing.

[0090] In one embodiment, as Figure 5 shown, the method further includes:

[0091] S502, in response to the direction indicated by the direction control information being different from the initial perspective of the panoramic video, based on the direction control information, the movement speed information, and the initial perspective of the panoramic video, determine the target perspective corresponding to the next frame panoramic image of the initial frame panoramic image.

[0092] Specifically, when the direction indicated by the direction control information is different from the initial perspective of the panoramic video, since it is necessary to simulate the first-person perspective to view the panoramic video, when the direction indicated by the direction control information is different from the initial perspective, it is necessary to determine the target perspective corresponding to the next frame panoramic image according to the direction control information, the movement speed information, and the initial perspective. It should be noted that the movement speed information here generally refers to the movement speed information matching the direction indicated by the direction control information.

[0093] S504, based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image, and the movement speed information, determine the target frame panoramic image.

[0094] Specifically, since the panoramic video moves in the direction of the initial perspective during shooting, and the initial perspective is different from the direction indicated by the control information, in order to accurately simulate an image that matches the movement speed information when the directions are different, it is necessary to determine the speed of the movement speed information in the direction of the initial perspective according to the angle between the initial perspective of the initial panoramic image and the direction indicated by the direction control information, as well as the movement speed information. Then, according to the speed in the direction of the initial perspective and the time of one frame, determine the third camera displacement of the next frame of the panoramic image. Then, according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame of the panoramic image, and the second camera displacement, determine the target frame panoramic image. (For the specific implementation here, reference can be made to the above embodiments, and no repeated description will be given here).

[0095] S506. Based on the initial frame panoramic image, the initial perspective, the target frame panoramic image, and the target perspective, adjust the panoramic video displayed during the first perspective control.

[0096] Specifically, here, taking the display of two frames of images as an example, the initial frame panoramic image can be displayed according to the initial perspective, and then the target frame panoramic image can be displayed according to the target perspective, so as to display the panoramic video. In addition, it should be noted that when more frames of images need to be displayed, the target frame panoramic image can be used as the initial frame panoramic image, and the next frame of the panoramic image and the target perspective of the target frame panoramic image can be continuously determined according to the above steps S502 - S504, and so on, continuously determining the next frame of the panoramic image and displaying it.

[0097] In this embodiment, when the direction indicated by the direction control information is different from the initial perspective, the target perspective can be determined and adjusted accordingly, so as to ensure that the panoramic video during the first perspective control can accurately display the perspective that matches the first perspective, and improve the viewing experience.

[0098] In one embodiment, as Figure 6 shown, the position information includes: IMU information. Among them, IMU (Inertial Measurement Unit) is a device used to measure and report the acceleration, angular velocity, and sometimes also the magnetic field of an object. An IMU usually consists of multiple sensors, including an accelerometer, a gyroscope, and a magnetometer. The calculating the first camera displacement corresponding to each frame of the panoramic image according to at least the position information matching each frame of the panoramic image includes:

[0099] S602. Obtain the acceleration information in the IMU information matching each frame of the panoramic image.

[0100] S604. Adjust the acceleration information to remove the gravity in the acceleration information.

[0101] S606, Integrate the acceleration data after removing gravity to calculate the velocity at the time of capturing each panoramic image.

[0102] S608, Based on the velocity at the time of capturing each panoramic image and the time of one frame, calculate the first camera displacement corresponding to each panoramic image.

[0103] Specifically, acceleration (usually a three-dimensional vector) and angular velocity (also usually a three-dimensional vector) data can be obtained from the IMU information. The acceleration data is usually in units of g (acceleration due to gravity) and needs to be converted to meters per second squared (m / s²). When calculating the velocity of the camera, the influence of gravity needs to be removed. The acceleration vector can be rotated through attitude information (for example, using quaternions or Euler angles) so that it is represented in the world coordinate system. Integrate the acceleration data after removing gravity to calculate the velocity. After obtaining the velocity at the time of capturing each panoramic image, the first camera displacement corresponding to each frame of the image can be determined by multiplying the time of one frame by the velocity at the time of capturing each frame of the image.

[0104] In this embodiment, the velocity at the time of capturing each panoramic image is calculated through the acceleration information in the IMU information, and then the first camera displacement can be accurately calculated based on this velocity, thereby ensuring the accuracy of determining the target frame panoramic image.

[0105] In one embodiment, the position information further includes: GPS information. GPS (Global Positioning System) information is the position information provided by the satellite system. The main content of GPS information is position coordinates, including:

[0106] Longitude: Represents the position in the east-west direction, ranging from -180° to +180°. Latitude: Represents the position in the north-south direction, ranging from -90° to +90°. Altitude: The height relative to sea level, usually in meters. The calculating the first camera displacement corresponding to each panoramic image at least according to the position information matched with each panoramic image includes:

[0107] Determine the first camera displacement corresponding to each panoramic image according to the GPS information matched with each panoramic image and the time of one frame.

[0108] Specifically, the GPS information matched with each panoramic image can be determined, and then according to the time of one frame, the change of the GPS information of each panoramic image within one frame time can be determined, and the first camera displacement corresponding to each panoramic image can be determined according to the change of the GPS information.

[0109] In this embodiment, the first camera displacement corresponding to each panoramic image can be quickly determined through the GPS information, improving the speed of determining the target frame panoramic image.

[0110] In one embodiment, as Figure 7 shown, the IMU information includes roll angle, pitch angle, and yaw angle. Since during the process of shooting a panoramic video, the panoramic video may not be horizontal with respect to the horizontal plane. Therefore, according to the IMU information at the time of shooting for each frame of the picture, the attitude can be adjusted so that the abscissa of the picture of each frame of the panoramic image is parallel to the horizontal line. Thus, the feature of maintaining a horizontal attitude when simulating the first-person view is achieved. Before at least according to the position information matching each frame of the panoramic image, the method further includes:

[0111] S702, constructing a rotation matrix according to the roll angle, pitch angle, and yaw angle of each frame of the panoramic image.

[0112] S704, adjusting the attitude of each frame of the panoramic image by using the rotation matrix.

[0113] S706, in response to there being blank areas in the panoramic image after adjusting the attitude, filling the blank areas by using the interpolation method.

[0114] Specifically, obtain IMU data: Obtain the attitude information corresponding to each frame of the panoramic image from the IMU sensor, which usually includes roll angle (roll), pitch angle (pitch), and yaw angle (yaw). Calculate the rotation matrix according to the IMU data. For a given roll angle and pitch angle, the following formula can be used to construct the rotation matrix: , where , , are the rotation matrices around the x, y, and z axes respectively. For each frame of the panoramic image, apply the calculated rotation matrix to adjust the attitude of the image. This can be achieved by using an image processing library (such as OpenCV). When adjusting the panoramic image, blank areas may occur. The interpolation method (such as bilinear interpolation) can be used to fill these areas.

[0115] In this embodiment, by adjusting each frame of the panoramic image through the IMU information, it can be ensured that when viewing the panoramic video from the first-person perspective, the attitude of the panoramic video is accurate. Additionally, blank areas may occur during the adjustment process, and the interpolation method (such as bilinear interpolation) can be used to fill these areas, thereby ensuring the integrity of the panoramic image.

[0116] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0117] Based on the same inventive concept, an embodiment of the present disclosure also provides a panoramic video display control device for implementing the panoramic video display control method described above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the panoramic video display control device provided below can refer to the limitations on the panoramic video display control method in the above text, and will not be repeated here.

[0118] In one embodiment, as Figure 8 shown, a panoramic video display control device 800 is provided, including: an information acquisition module 802, an instruction parsing module 804, a displacement calculation module 806, an image determination module 808, and a video adjustment module 810, where:

[0119] The information acquisition module 802 is configured to acquire a panoramic video and position information matching each frame of panoramic image in the panoramic video;

[0120] The instruction parsing module 804 is configured to receive a control instruction for first-view control of the panoramic video, parse the control instruction, and determine the direction control information and moving speed information indicated by the control instruction;

[0121] The displacement calculation module 806 is configured to calculate a first camera displacement corresponding to each frame of panoramic image at least according to the position information matching each frame of panoramic image;

[0122] The image determination module 808 is configured to determine an initial frame panoramic image in the panoramic video. In response to the direction indicated by the direction control information being the same as the initial view angle of the panoramic video, based on the first camera displacement corresponding to each frame of panoramic image, the initial frame panoramic image, and the moving speed information, determine a target frame panoramic image;

[0123] A video adjustment module 810, configured to adjust a panoramic video displayed during a first perspective control based on the initial frame panoramic image and the target frame panoramic image.

[0124] In an embodiment of the device, the image determination module 808 includes:

[0125] A displacement determination module, configured to determine a second camera displacement corresponding to a next-frame panoramic image of the initial frame panoramic image based on the movement speed information;

[0126] A determination sub-module, configured to determine a target frame panoramic image according to the camera position corresponding to the initial frame panoramic image, the first camera displacement corresponding to each frame panoramic image, and the second camera displacement.

[0127] In an embodiment of the device, the determination sub-module is further configured to subtract the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference; determine a preselected frame panoramic image whose shooting time is after the initial frame panoramic image according to each frame panoramic image in the panoramic video and the initial frame panoramic image; determine a preselected frame panoramic image that satisfies the displacement difference according to the displacement difference and the first camera displacement corresponding to the preselected frame panoramic image; and determine a target frame panoramic image according to the preselected frame panoramic image that satisfies the displacement difference.

[0128] In an embodiment of the device, the device further includes:

[0129] A target perspective determination module, configured to, in response to the direction indicated by the direction control information being different from the initial perspective of the panoramic video, determine a target perspective corresponding to a next-frame panoramic image of the initial frame panoramic image based on the direction control information, the movement speed information, and the initial perspective of the panoramic video.

[0130] The image determination module 808 is further configured to determine a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image, and the movement speed information.

[0131] The video adjustment module 810 is further configured to adjust a panoramic video displayed during a first perspective control based on the initial frame panoramic image, the initial perspective, the target frame panoramic image, and the target perspective.

[0132] In one embodiment of the device, the position information includes: IMU information; the displacement calculation module 806 is further configured to obtain the acceleration information in the IMU information matched with each frame of panoramic image; adjust the acceleration information to remove the gravity in the acceleration information; integrate the acceleration data after removing the gravity to calculate the speed when each frame of panoramic image is captured; and calculate the first camera displacement corresponding to each frame of panoramic image based on the speed when each frame of panoramic image is captured and the time of one frame.

[0133] In one embodiment of the device, the position information includes: GPS information; the displacement calculation module 806 is further configured to determine and calculate the first camera displacement corresponding to each frame of panoramic image according to the GPS information matched with each frame of panoramic image and the time of one frame.

[0134] In one embodiment of the device, the position information includes: IMU information, and the IMU information includes roll angle, pitch angle and yaw angle. The device further includes: an image adjustment module, configured to construct a rotation matrix according to the roll angle, pitch angle and yaw angle of each frame of panoramic image; use the rotation matrix to adjust the pose of each frame of panoramic image; and in response to the existence of blank areas in the panoramic image after adjusting the pose, fill the blank areas by using the interpolation method.

[0135] Each module in the above panoramic video display control device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in the form of hardware or be independent of the processor, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0136] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 9As shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a display control method for panoramic videos. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0137] Those skilled in the art can understand that Figure 9 the structure shown in is only a block diagram of some structures related to the present disclosure solution, and does not constitute a limitation on the computer device to which the present disclosure solution is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0138] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the steps in the above method embodiments.

[0139] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0140] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0141] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided by the present disclosure can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided by the present disclosure can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0142] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0143] The above-described embodiments merely represent several implementation manners of the present disclosure. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patents of the present disclosure. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present disclosure, several modifications and improvements can still be made, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the appended claims.

Claims

1. A method for controlling the display of a panoramic video, characterized in that: The method comprises: Acquire a panoramic video and position information matching each panoramic image frame in the panoramic video; receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction; Calculate a first camera displacement corresponding to each frame of the panoramic image at least according to the position information matched by each frame of the panoramic image; Determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image, and the moving speed information, wherein the target frame panoramic image is a next frame image after the initial frame panoramic image is displayed in the first viewing angle; The step of determining a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information comprises: Based on the movement speed information, determine a second camera displacement corresponding to a next panoramic image frame of the initial panoramic image frame; determine a target panoramic image frame according to a camera position corresponding to the initial panoramic image frame, a first camera displacement corresponding to each panoramic image frame, and the second camera displacement; Based on the initial frame panoramic image and the target frame panoramic image, the panoramic video displayed during the first viewing angle control is adjusted.

2. The method according to claim 1, characterized in that The step of determining a target frame panoramic image according to a camera position corresponding to the initial frame panoramic image, a first camera displacement corresponding to each frame panoramic image, and a second camera displacement includes: Subtract the camera position corresponding to the initial frame panoramic image from the second camera displacement to obtain a displacement difference; Determine, according to each panoramic image frame in the panoramic video and the initial panoramic image frame, a preselected panoramic image frame whose shooting time is after the initial panoramic image frame; Determine a preselected frame panoramic image that satisfies the displacement difference according to the displacement difference and a first camera displacement corresponding to the preselected frame panoramic image; A target frame panoramic image is determined according to the preselected frame panoramic image that satisfies the displacement difference.

3. The method according to claim 1, characterized in that The method further comprises: In response to the direction indicated by the direction control information being different from the initial viewing angle of the panoramic video, determining a target viewing angle corresponding to a next panoramic image frame of the initial panoramic image frame based on the direction control information, the movement speed information, and the initial viewing angle of the panoramic video; Determine a target frame panoramic image based on the first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information; Based on the initial frame panoramic image, the initial viewing angle, the target frame panoramic image and the target viewing angle, the panoramic video displayed during the first viewing angle control is adjusted.

4. The method according to claim 1, characterized in that: The position information includes: IMU information; the calculating the first camera displacement corresponding to each frame of panoramic image at least according to the position information matched by each frame of panoramic image includes: Obtaining acceleration information from the IMU information that matches each frame of the panoramic image; Adjusting the acceleration information to remove gravity from the acceleration information; The acceleration data after removing gravity is integrated to calculate the speed when each frame of the panoramic image is taken; Based on the speed when shooting each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is calculated.

5. The method according to claim 1, characterized in that The position information includes: GPS information; the calculating the first camera displacement corresponding to each panoramic image frame at least according to the position information matched by each panoramic image frame includes: According to the GPS information matched with each frame of the panoramic image and the time of one frame, a first camera displacement corresponding to each frame of the panoramic image is determined and calculated.

6. The method according to any one of claims 1 to 5, characterized in that The position information includes: IMU information, and the IMU information includes a roll angle, a pitch angle, and a yaw angle; before the position information matched at least according to each frame of the panoramic image, the method further includes: Construct a rotation matrix based on the roll angle, pitch angle and yaw angle of each frame of the panoramic image; Using the rotation matrix to adjust the posture of each frame of the panoramic image; In response to the presence of blank areas in the panoramic image after the posture adjustment, the blank areas are filled by using an interpolation method.

7. A panoramic video display control device, characterized in that: The device comprises: An information acquisition module, used to acquire a panoramic video and position information matching each panoramic image frame in the panoramic video; An instruction parsing module, used for receiving a control instruction for controlling a first viewing angle of the panoramic video, parsing the control instruction, and determining direction control information and movement speed information indicated by the control instruction; A displacement calculation module, used to calculate a first camera displacement corresponding to each frame of the panoramic image at least according to the position information matched with each frame of the panoramic image; an image determination module, configured to determine an initial frame panoramic image in the panoramic video, and in response to the direction indicated by the direction control information being the same as the initial viewing angle of the panoramic video, determine a target frame panoramic image based on a first camera displacement corresponding to each frame panoramic image, the initial frame panoramic image and the moving speed information; the target frame panoramic image is a next frame image after the initial frame panoramic image is displayed in the first viewing angle; The image determination module comprises: A displacement determination module, configured to determine, based on the movement speed information, a second camera displacement corresponding to a next panoramic image frame of the initial panoramic image frame; A determination submodule, configured to determine a target frame panoramic image according to a camera position corresponding to the initial frame panoramic image, a first camera displacement corresponding to each frame panoramic image, and the second camera displacement; The video adjustment module is used to adjust the panoramic video displayed during the first viewing angle control based on the initial frame panoramic image and the target frame panoramic image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images