Panoramic video storage optimization method, system, terminal and storage medium

Abstract

The present invention discloses a storage optimization method, system, terminal, and storage medium for panoramic videos. The method comprises: obtaining a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the motion range of the object, and segmenting the panoramic video into panoramic video blocks; obtaining a second panoramic image based on the panoramic video blocks, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane; performing reinforcement learning processing on the projection plane, and projecting the panoramic video blocks into a plane format before compressing and storing them. The present invention divides the panoramic video into multiple panoramic video blocks on the time axis, performs multi-degree-of-freedom spatial rotation on the projection coordinate system based on the content of each panoramic video block, and then projects the panoramic video into a plane format and compresses the video, thereby reducing the storage space required for the compressed panoramic video.

Description

Technical Field

[0001] The present invention relates to the field of panoramic video on demand or live broadcast, and in particular to a storage optimization method, system, terminal and storage medium for panoramic video. Background Art

[0002] Panoramic video is a 360-degree video shot with a 3D camera. Users can freely adjust the video to view it from any angle. Panoramic video provides a full range of vision, allowing users to freely choose any direction to view the scene, creating an immersive viewing experience.

[0003] In the prior art, when storing panoramic videos, since high-quality panoramic videos have a very high bit rate and the current bandwidth is insufficient to support real-time transmission of high-quality panoramic videos, the panoramic videos are usually divided into multiple panoramic video blocks of the same time length on the time axis. Then, using multiple projection coordinate systems, each panoramic image frame in the panoramic video block is projected into a polyhedron format. The various projection surfaces of the polyhedron are then spliced ​​into a single planar format image, and the continuous frames are compressed using traditional planar video compression technology, such as the H256 video coding standard. However, during the actual projection process, foreground objects such as cars in a single-frame panoramic image will be located in different projection surfaces, and the two middle projection surfaces in the second row will separate the car. The spatial discontinuity of the foreground objects is not conducive to the compression process of the planar video compression technology, resulting in a lower compression rate and more storage space occupation.

[0004] In addition, there may be multiple foreground objects moving simultaneously in each panoramic video block. The longer the time length of the panoramic video block, the larger the movement range of the foreground objects, and it is more likely that a large number of foreground objects will span more projection surfaces. On the contrary, if the time length is shorter, the temporal continuity will also be disturbed, which is also not conducive to the compression process of planar video compression technology in these cases.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The main purpose of the present invention is to provide a storage optimization method, system, terminal and storage medium for panoramic videos, aiming to solve the problems in the prior art that the spatial discontinuity of foreground objects and the short time length of panoramic video blocks are not conducive to the compression process of planar video compression technology, resulting in a lower compression rate and occupying more storage space.

[0007] To achieve the above object, the present invention provides a method for optimizing storage of panoramic videos, the method comprising the following steps:

[0008] Acquire a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks;

[0009] Acquiring a second panoramic image based on the panoramic video block, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane;

[0010] Reinforcement learning processing is performed on the projection plane, and the panoramic video block is projected into a plane format and then compressed and stored.

[0011] Optionally, the method for optimizing storage of panoramic videos, wherein the steps of obtaining a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating a motion range of the object, and dividing the panoramic video into panoramic video blocks, specifically include:

[0012] Uniformly sampling the panoramic video to obtain several frames of panoramic images, using an object recognition algorithm to identify objects in each frame of the panoramic image, and marking the same objects in consecutive frames;

[0013] When the center point of the object is used as the center of gravity of the object, a time length is set, and the distance of the center of gravity movement between the first frame of each object and the frame after the time length is calculated;

[0014] Calculating an average value of the center-of-gravity movement distances, using the average value as the average movement range of all objects within the time length, and setting a threshold value of the average range;

[0015] The size of the average motion range and the threshold is determined. If the average motion range is equal to the threshold, the panoramic video is divided into multiple panoramic video blocks on the time axis using the time length.

[0016] Optionally, in the storage optimization method for panoramic videos, the size of the threshold is set according to the projection format.

[0017] Optionally, in the panoramic video storage optimization method, the step of determining the difference between the average motion range and the threshold value further comprises:

[0018] If the average motion range is greater than the threshold, shortening the time length, and recalculating the average motion range of all objects within the shortened time length;

[0019] If the average motion range is smaller than the threshold, the time length is increased, and the average motion ranges of all objects within the increased time length are recalculated.

[0020] Optionally, the method for optimizing storage of panoramic videos, wherein the step of obtaining a second panoramic image based on the panoramic video block, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane, specifically includes:

[0021] Extracting a plurality of frames of second panoramic images from the panoramic video block, and inputting the second panoramic images into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result;

[0022] Based on the projection format of the semantic segmentation result, a projection coordinate system corresponding to the projection format is selected, and the semantic segmentation result is projected using the projection coordinate system to obtain multiple projection planes.

[0023] Optionally, in the storage optimization method for panoramic videos, the projection format includes an equirectangular projection format and a cubic projection format.

[0024] Optionally, the method for optimizing storage of panoramic videos, wherein the step of performing reinforcement learning on the projection plane and compressing and storing the panoramic video blocks after projecting them into a plane format, specifically includes:

[0025] Inputting the projection plane into the reinforcement learning method in time sequence for processing, and outputting the multi-degree-of-freedom rotation angle after the processing is completed;

[0026] completing a rotation of the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block using the rotated projection coordinate system to obtain a first panoramic video block in a planar format;

[0027] The first panoramic video block is compressed and stored, and the multiple free rotation angles of the first panoramic video block are stored in a record file.

[0028] Optionally, the method for optimizing storage of panoramic videos, wherein the step of rotating the projection coordinate system based on the multi-degree-of-freedom rotation angles further comprises:

[0029] If there are multiple suitable multi-freedom rotation angles, performing multiple iterative rotations on the projection coordinate system, and projecting the projection coordinate system after each rotation onto the panoramic video block to obtain multiple second panoramic video blocks in a planar format;

[0030] compressing all second panoramic video blocks and recording the compression rate of each second panoramic video block;

[0031] A second panoramic video block with the highest compression rate is selected for storage, and the multi-degree-of-freedom rotation angles of the second panoramic video block are recorded in a file.

[0032] Optionally, the panoramic video storage optimization method, wherein the rotating the projection coordinate system based on the multi-degree-of-freedom rotation angle is completed, further comprises:

[0033] According to the periodic change of the rotation angle, the range of the rotation angle within one cycle is obtained;

[0034] Based on the variation range, limiting the rotation angle output by the reinforcement learning method to the variation range of the cycle;

[0035] If the rotation angle after restriction is 0, the processing of the current panoramic video block is terminated.

[0036] Optionally, the method for optimizing storage of panoramic videos, wherein the first panoramic video block is compressed and stored, and the multiple free rotation angles of the first panoramic video block are stored in a record file, further comprises:

[0037] receiving an access request from a user, and transmitting a panoramic video block to a user client based on the access request;

[0038] When transmitting the panoramic video block, extracting multiple free rotation angles corresponding to the panoramic video block from the record file, and transmitting the multiple free rotation angles to the user client;

[0039] After the user client completes receiving the information, the projection coordinate system is rotated based on the multiple free rotation angles, and the rotated projection coordinate system is used to project the panoramic video block to complete the rendering process.

[0040] In addition, to achieve the above-mentioned purpose, the present invention further provides a panoramic video storage optimization system, wherein the panoramic video storage optimization system includes:

[0041] a video segmentation module, configured to obtain a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks;

[0042] a video projection module, configured to acquire a second panoramic image based on the panoramic video block, segment the second panoramic image to obtain a semantic segmentation result, and project the semantic segmentation result to obtain a projection plane;

[0043] The compression storage module is used to perform reinforcement learning processing on the projection plane, and compress and store the panoramic video block after projecting it into a plane format.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a storage optimization program for panoramic videos stored on the memory and runnable on the processor, and when the storage optimization program for panoramic videos is executed by the processor, the steps of the storage optimization method for panoramic videos as described above are implemented.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a storage optimization program for panoramic videos, and when the storage optimization program for panoramic videos is executed by a processor, the steps of the storage optimization method for panoramic videos as described above are implemented.

[0046] In the present invention, a first panoramic image is acquired from a panoramic video, objects in the first panoramic image are identified, the range of motion of the objects is calculated, and the panoramic video is segmented into panoramic video blocks. A second panoramic image is acquired based on the panoramic video blocks, the second panoramic image is segmented to obtain a semantic segmentation result, and the semantic segmentation result is projected to obtain a projection plane. The projection plane is subjected to reinforcement learning processing, and the panoramic video blocks are projected into a planar format and then compressed and stored. The present invention divides the panoramic video into multiple panoramic video blocks on the time axis. Based on the content of each panoramic video block, the projection coordinate system is spatially rotated with multiple degrees of freedom. The rotated projection coordinate system is then used to project the panoramic video into a planar format and compress the video, thereby reducing the storage space required for the compressed panoramic video. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a preferred embodiment of the storage optimization method of panoramic video in the present invention;

[0048] Figure 2 4 is a flowchart of step S10 in a preferred embodiment of the method for optimizing storage of panoramic videos of the present invention;

[0049] Figure 3 Schematic diagram of a panoramic image in equirectangular projection format according to a preferred embodiment of the present invention;

[0050] Figure 4 Schematic diagram of the recognition results of the object recognition algorithm of the panoramic video storage optimization method in the present invention;

[0051] Figure 5 is a flow chart of determining the time length of a panoramic video block in the method for optimizing the storage of panoramic videos in the present invention;

[0052] Figure 6 Flowchart of step S20 in a preferred embodiment of the method for optimizing storage of panoramic videos of the present invention;

[0053] Figure 7 2. It is a schematic diagram of semantic segmentation results in an isometric projection format of the storage optimization method for panoramic videos in the present invention;

[0054] Figure 8 This is a schematic diagram of a panoramic image in a cubic projection format according to a preferred embodiment of the present invention;

[0055] Figure 9 Schematic diagram of semantic segmentation results in cube projection format in a preferred embodiment of the present invention;

[0056] Figure 10 Schematic diagram of the panoramic video block projection process of the panoramic video storage optimization method of the present invention;

[0057] Figure 11 4 is a flowchart of step S30 in a preferred embodiment of the panoramic video storage optimization method of the present invention;

[0058] Figure 12 2. This is a schematic diagram of a panoramic image after the cube projection coordinate system is horizontally rotated 45° in a preferred embodiment of the present invention;

[0059] Figure 13 1 is an overall schematic diagram of a preferred embodiment of a storage optimization system for panoramic videos of the present invention;

[0060] Figure 14 1 is a schematic diagram of the principle of a preferred embodiment of a storage optimization system for panoramic videos of the present invention;

[0061] Figure 15 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0064] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0065] The storage optimization method of the panoramic video according to the preferred embodiment of the present invention is as follows: Figure 1 As shown, the storage optimization method of the panoramic video includes the following steps:

[0066] Step S10: Acquire a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks.

[0067] For the specific process, please refer to Figure 2 , which is a flowchart of step S10 in the panoramic video storage optimization method provided by the present invention.

[0068] like Figure 2 As shown, step S10 includes:

[0069] Step S11: uniformly sampling the panoramic video to obtain several frames of panoramic images, using an object recognition algorithm to identify objects in each frame of the panoramic image, and marking the same objects in consecutive frames;

[0070] Step S12: When the center point of the object is used as the center of gravity of the object, a time length is set, and a distance of movement of the center of gravity between the first frame of each object and the frame after the time length is calculated;

[0071] Step S13: Calculate the average value of the center-of-gravity movement distance, use the average value as the average movement range of all objects within the time length, and set a threshold value of the average range;

[0072] Step S14: Determine the size of the average motion range and the threshold. If the average motion range is equal to the threshold, use the time length to divide the panoramic video into multiple panoramic video blocks on the time axis.

[0073] Specifically, a plurality of panoramic images are uniformly sampled from the panoramic video, and an object recognition algorithm is used to identify objects in each panoramic image frame, and the same objects in consecutive frames are marked. For example, in the embodiment of the present invention, Figure 3 ( Figure 3 The object recognition algorithm is used to output the object and identify it. The identification result is as follows: Figure 4 As shown, the object recognition algorithm can recognize and identify the object position and object category in the image according to the image content. The object recognition algorithm in the embodiment of the present invention can use but is not limited to the Yolo-v4 algorithm; assuming that the center point of each object is its center of gravity, and then presetting a time length (for example, 1s), the center of gravity movement distance (great circle distance) between the first frame of each object and the frame after 1s is calculated, and then the average value of the center of gravity movement distance is calculated, and the average value is used as the average movement range of all objects in the time length, and a threshold of the average range is set, and the threshold value depends on the projection format (for example, cube projection format, A projection surface occupies 90°. In the experiment, the threshold can be selected as a value less than 90°, such as 45° or 60°. There is no fixed formula here, and sometimes the effect difference is not obvious). The size of the average motion range and the threshold is judged. If the average motion range is greater than the threshold, the time length is shortened, and the average motion range of all objects in the shortened time length is recalculated; if the average motion range is less than the threshold, the time length is increased, and the average motion range of all objects in the increased time length is recalculated; if it is close to or equal to the threshold, the time length is used to divide the panoramic video into multiple panoramic video blocks on the time axis.

[0074] Furthermore, if Figure 5 As shown in the flowchart of determining the time length of a panoramic video block, step S101: uniformly sampling several frames of panoramic images from a panoramic video; step S102: using an object recognition algorithm to identify objects in each frame of the panoramic image, and marking the same objects in consecutive frames; step S103: calculating the average motion range of all objects within a period of time; step S104: determining the size of the average motion range, if it exceeds a threshold, reducing the time length and repeating step S103; if it is less than the threshold, increasing the time length and repeating step S103; if it is close to or equal to the threshold, continuing to step S105; step S105: using the time length to divide the panoramic video into multiple panoramic video blocks on the time axis.

[0075] Step S20: Acquire a second panoramic image based on the panoramic video block, segment the second panoramic image to obtain a semantic segmentation result, and project the semantic segmentation result to obtain a projection plane.

[0076] For the specific process, please refer to Figure 6 , which is a flowchart of step S20 in the panoramic video storage optimization method provided by the present invention.

[0077] like Figure 6 As shown, step S20 includes:

[0078] Step S21: extracting a plurality of frames of second panoramic images from the panoramic video block, and inputting the second panoramic images into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result;

[0079] Step S22: Based on the projection format of the semantic segmentation result, select a projection coordinate system corresponding to the projection format, use the projection coordinate system to project the semantic segmentation result and obtain multiple projection planes.

[0080] Specifically, several frames of panoramic images are sampled from the panoramic video block, and the extracted panoramic images are used as input, and a semantic segmentation algorithm is used to output a semantic segmentation result, wherein the semantic segmentation algorithm can identify the pixels of the foreground object from the image. In the embodiment of the present invention, the foreground pixels are defined as common objects of interest such as people and cars (for example, using Figure 3 As input, the semantic segmentation algorithm is used to output the semantic segmentation result in the equirectangular projection format. The semantic segmentation result in the equirectangular projection format is as follows Figure 7 As shown in the figure, the white portion is the foreground pixel. In the embodiment of the present invention, the foreground pixel is defined as a common object of interest, such as a person or a car. It is worth noting that the semantic segmentation algorithm can be used with, but is not limited to, ugscnn; according to the projection format, the corresponding projection coordinate system is selected, wherein the projection format includes an equirectangular projection format and a cubic projection format. Figure 3 is an equirectangular projection format panoramic image, Figure 8 is a panoramic image in cube projection format; input the extracted panoramic image semantic segmentation result, and use the projection coordinate system to project the semantic segmentation result to obtain multiple projection planes. Taking the cube projection format as an example, the projection obtains the semantic segmentation result in cube projection format. The semantic segmentation result in cube projection format is as follows: Figure 9 As shown, the white part in the figure is the foreground pixel, and in the embodiment of the present invention, the foreground pixel is defined as a common object of attention such as a person or a car.

[0081] Furthermore, Figure 10To quickly complete the entire projection process of a panoramic video, several frames of panoramic images are sampled from a panoramic video block; the extracted panoramic image is used as input; a semantic segmentation algorithm is used to segment the image and output the semantic segmentation result; a corresponding projection coordinate system is selected according to the projection format; the semantic segmentation result of the extracted panoramic image is input, and the semantic segmentation result is projected using the projection coordinate system to obtain multiple projection planes; the projection planes are input in timeline order, and a reinforcement learning method is used to output multi-degree-of-freedom rotation angles; the projection coordinate system is rotated using the multi-degree-of-freedom rotation angles; the panoramic video block is projected into a planar format using the rotated projection coordinate system and compressed, and the compressed panoramic video block is input into the reinforcement learning method in the form of reward feedback.

[0082] Step S30: Perform reinforcement learning processing on the projection plane, and compress and store the panoramic video block after projecting it into a plane format.

[0083] For the specific process, please refer to Figure 11 , which is a flowchart of step S30 in the panoramic video storage optimization method provided by the present invention.

[0084] like Figure 11 As shown, step S30 includes:

[0085] Step S31: input the projection plane into the reinforcement learning method in time sequence, and output the multi-degree-of-freedom rotation angle after the processing is completed;

[0086] Step S32: rotating the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block using the rotated projection coordinate system to obtain a first panoramic video block in a planar format;

[0087] Step S33: compress and store the first panoramic video block, and store the multiple free rotation angles of the first panoramic video block in a record file.

[0088] Specifically, the projection plane is input into the reinforcement learning method in time axis order to output a multi-degree-of-freedom rotation angle; wherein the reinforcement learning method in the embodiment of the present invention can be used with but not limited to a continuous action space reinforcement learning method example DDPG or a discrete action space reinforcement learning method example DQN; the output multi-degree-of-freedom rotation angle is used to rotate the projection coordinate system (for example, a cube projection format panoramic image can be rotated in the horizontal and vertical directions of the projection coordinate system;); the rotated projection coordinate system is used to project the panoramic video block into a plane format and compress it. Taking the cube projection format as an example, the projection is projected by rotating 45° to the right in the horizontal direction. Figure 3 get Figure 12 , Figure 12Foreground objects such as people and cars are well preserved in the projection plane, and only a small number of foreground objects span different projection planes, thereby improving the compression rate of the panoramic video block; the multi-degree-of-freedom rotation angles of the panoramic video block are stored in the record file to prepare for subsequent rendering.

[0089] Considering that multiple suitable multi-degree-of-freedom rotation angles may exist during the rotation of the projected coordinate system, the projected coordinate system is rotated multiple times iteratively. After each iterative rotation of the projected coordinate system, the panoramic video needs to be compressed. The compression ratios of the panoramic video blocks under different projected coordinate systems are compared, and the panoramic video block with the highest compression ratio is selected for storage. The multi-degree-of-freedom rotation angle that results in the highest compression ratio is recorded in the file. In addition, to prevent infinite iterative rotation, in addition to determining that the output rotation angle is 0, an upper limit on the iteration can also be set.

[0090] Furthermore, due to the heterogeneity of polyhedrons, it is necessary to restrict the spatial rotation angle output by reinforcement learning. Taking the cube projection format as an example, the integer multiples of 90° horizontal rotation only changes the arrangement order and direction of the projection plane; the restriction method is specifically based on the periodic change of the rotation angle to obtain the range of change of the rotation angle within a cycle. Taking the cube projection format as an example, the horizontal rotation period is [0°, 90°); the rotation angle output by the reinforcement learning method is restricted to the range of change of one cycle; if the spatial rotation angle after restriction is 0, the reasoning process of the current panoramic video block is terminated; since different degrees of freedom are independent of each other, the above restriction method is applicable to any degree of freedom.

[0091] Furthermore, the rotated panoramic video block is rendered.

[0092] Specifically, a user's access request is received, and a panoramic video block is transmitted to the user client based on the access request; when transmitting the panoramic video block, multiple free rotation angles corresponding to the panoramic video block are extracted from the record file, and the multiple free rotation angles are transmitted to the user client; when the user client completes the reception, the projection coordinate system is rotated based on the multiple free rotation angles, and the rotated projection coordinate system is used to project the panoramic video block to complete the rendering process.

[0093] Furthermore, the overall process of the preferred embodiment of the panoramic video storage optimization system of the present invention is as follows: Figure 13 The specific steps are as follows:

[0094] Step S501: inputting several panoramic image frames in a panoramic video, and outputting object recognition results for each frame using an object recognition algorithm;

[0095] Step S502: Determine the motion range of the object and determine the time length of each panoramic video block;

[0096] Step S503: dividing the panoramic video into multiple panoramic video blocks on the time axis;

[0097] Step S504: extracting several frames of panoramic images from the panoramic video block as input, and using a semantic segmentation algorithm to output a semantic segmentation result;

[0098] Step S505: Projecting the semantic segmentation results corresponding to all extracted panoramic images to obtain multiple projection planes;

[0099] Step S506: input the projection plane in time sequence, and use the reinforcement learning method to output the multi-degree-of-freedom rotation angle;

[0100] Step S507: rotating the projection coordinate system using multi-degree-of-freedom rotation angles;

[0101] Step S508: Project the panoramic video block into a planar format using the rotated projection coordinate system and compress it.

[0102] Furthermore, if Figure 14 As shown, based on the above-mentioned panoramic video storage optimization method, the present invention also provides a panoramic video storage optimization system, wherein the panoramic video storage optimization system includes:

[0103] A video segmentation module 51 is configured to obtain a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks;

[0104] a video projection module 52 configured to acquire a second panoramic image based on the panoramic video block, segment the second panoramic image to obtain a semantic segmentation result, and project the semantic segmentation result to obtain a projection plane;

[0105] The compression storage module 53 is used to perform reinforcement learning processing on the projection plane, and compress and store the panoramic video block after projecting it into a plane format.

[0106] Furthermore, if Figure 15 As shown, based on the above-mentioned panoramic video storage optimization method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 15 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0107] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a storage optimization program 40 for panoramic video is stored on the memory 20, and the storage optimization program 40 for panoramic video can be executed by the processor 10, thereby realizing the storage optimization method for panoramic video in the present application.

[0108] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the storage optimization method for the panoramic video.

[0109] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0110] In one embodiment, when the processor 10 executes the storage optimization program 40 for the panoramic video in the memory 20, the following steps are implemented:

[0111] Acquire a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks;

[0112] Acquiring a second panoramic image based on the panoramic video block, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane;

[0113] Reinforcement learning processing is performed on the projection plane, and the panoramic video block is projected into a plane format and then compressed and stored.

[0114] The acquiring of a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating a motion range of the object, and segmenting the panoramic video into panoramic video blocks specifically includes:

[0115] Uniformly sampling the panoramic video to obtain several frames of panoramic images, using an object recognition algorithm to identify objects in each frame of the panoramic image, and marking the same objects in consecutive frames;

[0116] When the center point of the object is used as the center of gravity of the object, a time length is set, and the distance of the center of gravity movement between the first frame of each object and the frame after the time length is calculated;

[0117] Calculating an average value of the center-of-gravity movement distances, using the average value as the average movement range of all objects within the time length, and setting a threshold value of the average range;

[0118] The size of the average motion range and the threshold is determined. If the average motion range is equal to the threshold, the panoramic video is divided into multiple panoramic video blocks on the time axis using the time length.

[0119] The threshold value is set according to the projection format.

[0120] The determining of the difference between the average motion range and the threshold value further includes:

[0121] If the average motion range is greater than the threshold, shortening the time length, and recalculating the average motion range of all objects within the shortened time length;

[0122] If the average motion range is smaller than the threshold, the time length is increased, and the average motion ranges of all objects within the increased time length are recalculated.

[0123] The acquiring of a second panoramic image based on the panoramic video block, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane specifically includes:

[0124] Extracting a plurality of frames of second panoramic images from the panoramic video block, and inputting the second panoramic images into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result;

[0125] Based on the projection format of the semantic segmentation result, a projection coordinate system corresponding to the projection format is selected, and the semantic segmentation result is projected using the projection coordinate system to obtain multiple projection planes.

[0126] The projection formats include an equidistant projection format and a cubic projection format.

[0127] The performing reinforcement learning processing on the projection plane and compressing and storing the panoramic video block after projecting it into a plane format specifically includes:

[0128] Inputting the projection plane into the reinforcement learning method in time sequence for processing, and outputting the multi-degree-of-freedom rotation angle after the processing is completed;

[0129] completing a rotation of the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block using the rotated projection coordinate system to obtain a first panoramic video block in a planar format;

[0130] The first panoramic video block is compressed and stored, and the multiple free rotation angles of the first panoramic video block are stored in a record file.

[0131] The rotating of the projected coordinate system based on the multi-degree-of-freedom rotation angle further includes:

[0132] If there are multiple suitable multi-freedom rotation angles, performing multiple iterative rotations on the projection coordinate system, and projecting the projection coordinate system after each rotation onto the panoramic video block to obtain multiple second panoramic video blocks in a planar format;

[0133] compressing all second panoramic video blocks and recording the compression rate of each second panoramic video block;

[0134] A second panoramic video block with the highest compression rate is selected for storage, and the multi-degree-of-freedom rotation angles of the second panoramic video block are recorded in a file.

[0135] The method further includes rotating the projection coordinate system based on the multi-degree-of-freedom rotation angle:

[0136] According to the periodic change of the rotation angle, the range of the rotation angle within one cycle is obtained;

[0137] Based on the variation range, limiting the rotation angle output by the reinforcement learning method to the variation range of the cycle;

[0138] If the rotation angle after restriction is 0, the processing of the current panoramic video block is terminated.

[0139] The method further comprises compressing and storing the first panoramic video block, and storing the multiple free rotation angles of the first panoramic video block in a record file, and then:

[0140] receiving an access request from a user, and transmitting a panoramic video block to a user client based on the access request;

[0141] When transmitting the panoramic video block, extracting multiple free rotation angles corresponding to the panoramic video block from the record file, and transmitting the multiple free rotation angles to the user client;

[0142] After the user client completes receiving the information, the projection coordinate system is rotated based on the multiple free rotation angles, and the rotated projection coordinate system is used to project the panoramic video block to complete the rendering process.

[0143] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a storage optimization program for panoramic videos, and when the storage optimization program for panoramic videos is executed by a processor, the steps of the panoramic video storage optimization method as described above are implemented.

[0144] In summary, the present invention provides a method for optimizing the storage of panoramic videos, the method comprising: obtaining a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating the motion range of the object, and segmenting the panoramic video into panoramic video blocks; obtaining a second panoramic image based on the panoramic video blocks, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane; performing reinforcement learning processing on the projection plane, and compressing and storing the panoramic video blocks after projecting them into a plane format. The present invention divides the panoramic video into multiple panoramic video blocks on the time axis, performs multi-degree-of-freedom spatial rotation on the projection coordinate system based on the content of each panoramic video block, and then uses the rotated projection coordinate system to project the panoramic video into a plane format and compress the video, thereby reducing the storage space required for the compressed panoramic video.

[0145] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0146] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0147] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for optimizing storage of panoramic videos, characterized in that: The storage optimization method of the panoramic video includes: Acquire a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks; Acquiring a second panoramic image based on the panoramic video block, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane; The acquiring a second panoramic image based on the panoramic video block, segmenting the second panoramic image to obtain a semantic segmentation result, and projecting the semantic segmentation result to obtain a projection plane specifically includes: Extracting a plurality of frames of second panoramic images from the panoramic video block, and inputting the second panoramic images into a semantic segmentation algorithm for segmentation to obtain a semantic segmentation result; Based on the projection format of the semantic segmentation result, selecting a projection coordinate system corresponding to the projection format, and using the projection coordinate system to project the semantic segmentation result to obtain multiple projection planes; Performing reinforcement learning processing on the projection plane, and compressing and storing the panoramic video block after projecting it into a plane format; The performing reinforcement learning processing on the projection plane and compressing and storing the panoramic video block after projecting it into a plane format specifically includes: Inputting the projection plane into the reinforcement learning method in time sequence for processing, and outputting the multi-degree-of-freedom rotation angle after the processing is completed; completing a rotation of the projection coordinate system based on the multi-degree-of-freedom rotation angle, and projecting the panoramic video block using the rotated projection coordinate system to obtain a first panoramic video block in a planar format; The first panoramic video block is compressed and stored, and the multiple free rotation angles of the first panoramic video block are stored in a record file.

2. The method for optimizing the storage of panoramic videos according to claim 1, wherein: The acquiring a first panoramic image in a panoramic video, identifying an object in the first panoramic image, calculating a motion range of the object, and dividing the panoramic video into panoramic video blocks specifically includes: Uniformly sampling the panoramic video to obtain several frames of panoramic images, using an object recognition algorithm to identify objects in each frame of the panoramic image, and marking the same objects in consecutive frames; When the center point of the object is used as the center of gravity of the object, a time length is set, and the distance of the center of gravity movement between the first frame of each object and the frame after the time length is calculated; Calculating an average value of the center-of-gravity movement distances, using the average value as the average movement range of all objects within the time length, and setting a threshold value for the average movement range; The size of the average motion range and the threshold is determined. If the average motion range is equal to the threshold, the panoramic video is divided into multiple panoramic video blocks on the time axis using the time length.

3. The method for optimizing the storage of panoramic videos according to claim 2, wherein: The size of the threshold is set according to the projection format.

4. The method for optimizing the storage of panoramic videos according to claim 2, wherein: The determining of the difference between the average motion range and the threshold value further includes: If the average motion range is greater than the threshold, shortening the time length, and recalculating the average motion range of all objects within the shortened time length; If the average motion range is smaller than the threshold, the time length is increased, and the average motion ranges of all objects within the increased time length are recalculated.

5. The method for optimizing storage of panoramic videos according to claim 1, wherein: The projection formats include an equidistant projection format and a cubic projection format.

6. The method for optimizing storage of panoramic videos according to claim 1, wherein: The method further comprises: rotating the projection coordinate system based on the multi-degree-of-freedom rotation angle; If there are multiple suitable multi-freedom rotation angles, performing multiple iterative rotations on the projection coordinate system, and projecting the projection coordinate system after each rotation onto the panoramic video block to obtain multiple second panoramic video blocks in a planar format; compressing all second panoramic video blocks and recording the compression rate of each second panoramic video block; A second panoramic video block with the highest compression rate is selected for storage, and the multi-degree-of-freedom rotation angles of the second panoramic video block are recorded in a file.

7. The method for optimizing storage of panoramic videos according to claim 1, wherein: The method further comprises: rotating the projection coordinate system based on the multi-degree-of-freedom rotation angle, and then: According to the periodic change of the rotation angle, the range of the rotation angle within one cycle is obtained; Based on the variation range, limiting the rotation angle output by the reinforcement learning method to the variation range of the cycle; If the rotation angle after restriction is 0, the processing of the current panoramic video block is terminated.

8. The method for optimizing storage of panoramic videos according to claim 1, wherein: The step of compressing and storing the first panoramic video block and storing the multiple free rotation angles of the first panoramic video block in a record file further includes: receiving an access request from a user, and transmitting a panoramic video block to a user client based on the access request; When transmitting the panoramic video block, extracting multiple free rotation angles corresponding to the panoramic video block from the record file, and transmitting the multiple free rotation angles to the user client; After the user client completes receiving the information, the projection coordinate system is rotated based on the multiple free rotation angles, and the rotated projection coordinate system is used to project the panoramic video block to complete the rendering process.

9. A panoramic video storage optimization system, characterized in that: The panoramic video storage optimization system is applied to the panoramic video storage optimization method according to any one of claims 1 to 8, and the panoramic video storage optimization system includes: a video segmentation module, configured to obtain a first panoramic image in a panoramic video, identify an object in the first panoramic image, calculate a motion range of the object, and segment the panoramic video into panoramic video blocks; a video projection module, configured to acquire a second panoramic image based on the panoramic video block, segment the second panoramic image to obtain a semantic segmentation result, and project the semantic segmentation result to obtain a projection plane; The compression storage module is used to perform reinforcement learning processing on the projection plane, and compress and store the panoramic video block after projecting it into a plane format.

10. A terminal, characterized in that: The terminal includes a memory, a processor, and a storage optimization program for panoramic videos stored in the memory and runnable on the processor. When the storage optimization program for panoramic videos is executed by the processor, the steps of the storage optimization method for panoramic videos as described in any one of claims 1-8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the panoramic video storage optimization method according to any one of claims 1 to 8.

Images