Method and apparatus for tone mapping of panoramic images

Abstract

The application provides a tone mapping method and device for a panoramic image. The tone mapping method for the panoramic image comprises the following steps: determining one or more target metadata information units of a first pixel point from a plurality of metadata information units, wherein the plurality of metadata information units are obtained by analyzing a code stream, the first pixel point is any pixel point in a two-dimensional plane projection image of a panoramic video to be processed, the plurality of metadata information units correspond to a plurality of divided areas included in a three-dimensional spherical surface expression panoramic image, and the two-dimensional plane projection image of the panoramic video and the three-dimensional spherical surface expression panoramic image have a mapping relationship; and performing tone mapping on a pixel value of the first pixel point according to the one or more target metadata information units to obtain a target tone mapping value of the first pixel point. The application can improve the processing efficiency of the panoramic image.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method and apparatus for tone mapping of panoramic images. Background Technology

[0002] Dynamic range is used in many fields to represent the ratio of the maximum to the minimum value of a variable. In digital imaging, dynamic range represents the ratio between the maximum and minimum grayscale values ​​within the displayable range of an image. In the real world, the dynamic range of the same scene is typically around 10⁻³ candela per square meter (cd / m²). 2 Up to 106 cd / m 2 The range between 0 and 255 is called high dynamic range (HDR). In most current color digital images, the dynamic range is 0 to 255, which is called low dynamic range (LDR).

[0003] Because there is a difference between the illuminance range of a display device and the dynamic range of the real world, it is necessary to map the dynamic range of the real world to the illuminance range of the display device; this is called dynamic range mapping. Dynamic range mapping can be applied to the adaptation of the front-end HDR signal and the display device with HDR capabilities. For example, if the front-end receives a signal with 4000 cd / m² illuminance... 2 The light signal, while the HDR display capability of the display device is 500 cd / m². 2 4000cd / m 2 The light signal is mapped to 500 cd / m 2 On display devices, this is a high-to-low tone mapping (TM) process. Dynamic range mapping can also be applied to adapt front-end SDR signals to display devices with HDR capabilities, for example, when the front-end receives a signal of 100 cd / m². 2 The light signal, while the HDR display capability of the display device is 2000 cd / m². 2 100cd / m 2 The light signal is mapped to 2000 cd / m 2 On display devices, it is a TM process from low to high.

[0004] Currently, dynamic range mapping (TM) methods can be divided into two types: static and dynamic. Static mapping methods perform the entire TM process based on a single data point from the same video content or hard drive content. This means the mapping curve is typically the same for all scenes. The advantage of this method is that it requires less data to be carried by the image and the processing flow is relatively simple. The disadvantage is that using the same mapping curve for all scenes can lead to information loss in some scenarios. For example, if the mapping curve focuses on protecting bright areas, details may be lost or completely invisible in extremely dark scenes, affecting the image display quality. Dynamic mapping methods, on the other hand, dynamically adjust the mapping curve based on specific regions, for each scene or each frame. The advantage of this method is that it allows for differentiated processing for different scenes or frames. The disadvantage is that it requires carrying relevant scene information in each frame or scene, resulting in a larger amount of data to be carried by the image.

[0005] However, neither of the above methods can be compatible with the dynamic range mapping of panoramic videos. Summary of the Invention

[0006] This application provides a tone mapping method and apparatus for panoramic images to improve the processing efficiency of panoramic images.

[0007] In a first aspect, this application provides a tone mapping method for a panoramic image, comprising: determining one or more target metadata information units for a first pixel from a plurality of metadata information units, wherein the plurality of metadata information units are obtained by parsing a bitstream, the first pixel is any pixel in a two-dimensional planar projection image of a panoramic video to be processed, the plurality of metadata information units correspond to a plurality of segmented regions included in a three-dimensional spherical panoramic image of the panoramic video, and the two-dimensional planar projection image of the panoramic video and the three-dimensional spherical panoramic image of the panoramic video have a mapping relationship; and performing tone mapping on the pixel value of the first pixel according to the one or more target metadata information units to obtain a target tone mapping value for the first pixel.

[0008] This application divides the panoramic video three-dimensional spherical representation of the panoramic image into multiple segmented regions. Based on the pixels corresponding to the mapping points contained in each segmented region, a metadata information unit is generated for that segmented region. The metadata information unit guides the display end to generate a tone mapping curve corresponding to that segmented region, thereby realizing tone mapping from image to display. This not only allows for the determination of matching tone mapping parameters for different dynamic range regions on the panoramic image, but also conforms to the display dynamic range of the display end, improving the processing efficiency of panoramic images.

[0009] In one possible implementation, before determining one or more target metadata information units of the first pixel from a plurality of metadata information units, the method further includes: acquiring the plurality of segmented regions.

[0010] In one possible implementation, obtaining the plurality of segmented regions includes: dividing the panoramic video three-dimensional spherical panoramic image according to a preset division method to obtain the plurality of segmented regions.

[0011] In one possible implementation, obtaining the plurality of segmented regions includes: dividing the panoramic video three-dimensional spherical panoramic image according to the segmentation method obtained by parsing the bitstream to obtain the plurality of segmented regions.

[0012] In one possible implementation, the multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or...

[0013] The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

[0014] In one possible implementation, dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the plurality of segmented regions includes: clustering the plurality of pixels included in the panoramic video two-dimensional planar projection image to obtain a plurality of pixel set; mapping the plurality of pixels to the panoramic video three-dimensional spherical representation panoramic image respectively; and taking the region formed by the mapping points of the pixels included in the first pixel set on the panoramic video three-dimensional spherical representation panoramic image as the first segmented region, wherein the first pixel set is one of the plurality of pixel set, and the first segmented region is one of the plurality of segmented regions.

[0015] In one possible implementation, dividing the panoramic video's three-dimensional spherical representation panoramic image to obtain the multiple segmented regions includes: mapping multiple pixels included in the panoramic video's two-dimensional planar projection image to the panoramic video's three-dimensional spherical representation panoramic image to obtain multiple mapping points; clustering the multiple mapping points to obtain multiple mapping point sets; and using the region formed by the mapping points included in the first mapping point set as a second segmented region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmented region is one of the multiple segmented regions.

[0016] In one possible implementation, obtaining the plurality of segmented regions includes: obtaining the plurality of segmented regions based on indication information of the plurality of segmented regions obtained by parsing the bitstream.

[0017] This application can use one of the above-mentioned segmentation methods to divide the panoramic image of the three-dimensional spherical representation of the panoramic video into multiple segmented regions, which can be adapted to the pixel distribution of the two-dimensional planar projection image of the panoramic video. This allows tone mapping to be achieved on a segmented region basis within a small dynamic range, thereby improving tone mapping efficiency.

[0018] In one possible implementation, determining one or more target metadata information units for the first pixel from a plurality of metadata information units includes: determining the correspondence between the plurality of metadata information units and a plurality of segmentation regions, wherein one metadata information unit corresponds to one or more segmentation regions; determining one or more target segmentation regions according to a set mapping point; when there is only one target segmentation region, determining the metadata information unit corresponding to the one target segmentation region as the one target metadata information unit; or, when there are multiple target segmentation regions, determining the metadata information units corresponding to the multiple target segmentation regions respectively as the multiple target metadata information units.

[0019] In one possible implementation, determining the correspondence between the plurality of metadata information units and the plurality of segmented regions includes: extracting the current metadata information unit from the plurality of metadata information units in a first preset order; extracting the current segmented region from the plurality of segmented regions in a second preset order; and establishing a correspondence between the current segmented region and the current metadata information unit.

[0020] In one possible implementation, determining the correspondence between the plurality of metadata information units and the plurality of segmented regions includes: extracting the current metadata information unit from the plurality of metadata information units in a first preset order; extracting the current segmented region from the plurality of segmented regions in a traversal order obtained by parsing the bitstream; and establishing a correspondence between the current segmented region and the current metadata information unit.

[0021] In one possible implementation, determining the correspondence between the plurality of metadata information units and the plurality of segmented regions includes: extracting the current metadata information unit from the plurality of metadata information units in a first preset order; obtaining one or more coordinates included in the current metadata information unit; determining one or more mapping points on the panoramic video 3D spherical panoramic image based on the one or more coordinates; when there is only one mapping point, establishing a correspondence between the segmented region to which the mapping point belongs and the current metadata information unit; when there are multiple mapping points, establishing a correspondence between at least one segmented region to which the multiple mapping points belong and the current metadata information unit.

[0022] This application uses one of the above-mentioned methods for obtaining correspondence to determine the correspondence between multiple segmented regions and multiple metadata information units. It does not require transmitting the correspondence to the display end through the bitstream, which can reduce the information carried in the bitstream.

[0023] In one possible implementation, the set mapping point is a first mapping point of the first pixel on the panoramic image of the three-dimensional spherical representation of the panoramic video; the step of determining one or more target segmentation regions based on the set mapping point includes: determining a first coverage area on the panoramic image of the three-dimensional spherical representation of the panoramic video with the first mapping point as the center; when the first coverage area includes a segmentation region, determining the segmentation region as the target segmentation region; or, when the first coverage area includes multiple segmentation regions, determining the multiple segmentation regions as the multiple target segmentation regions.

[0024] In one possible implementation, the set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical representation of the panoramic video; the step of determining one or more target segmentation regions based on the set mapping point includes: determining a second segmentation region to which the viewpoint center belongs, the plurality of segmentation regions including the second segmentation region; and determining the second segmentation region as the target segmentation region.

[0025] In one possible implementation, the set mapping point is the viewpoint center on the panoramic image of the three-dimensional spherical representation of the panoramic video; determining one or more target segmentation regions based on the set mapping point includes: determining a second coverage area on the panoramic image of the three-dimensional spherical representation of the panoramic video with the viewpoint center as the center; when the second coverage area includes a segmentation region, determining the segmentation region as the target segmentation region; or, when the second coverage area includes multiple segmentation regions, determining the multiple segmentation regions as the multiple target segmentation regions.

[0026] In one possible implementation, the step of performing tone mapping on the pixel value of the first pixel according to the one or more target metadata information units to obtain a target tone mapping value for the first pixel includes: obtaining one or more tone mapping curves according to the one or more target metadata information units; when there is only one tone mapping curve, performing tone mapping on the pixel value of the first pixel according to the one tone mapping curve to obtain the target tone mapping value; or, when there are multiple tone mapping curves, performing tone mapping on the pixel value of the first pixel according to the multiple tone mapping curves respectively to obtain multiple intermediate tone values ​​for the first pixel; and obtaining the target tone mapping value based on the multiple intermediate tone values.

[0027] In one possible implementation, obtaining the target tone mapping value based on the plurality of intermediate tone values ​​includes: determining the median value among the plurality of intermediate tone values ​​as the target tone mapping value; or, performing a weighted average of the plurality of intermediate tone values ​​to obtain the target tone mapping value, wherein the weighting coefficients of the plurality of intermediate tone values ​​are preset, or the weighting coefficients of the plurality of intermediate tone values ​​are related to the distance between the first pixel and the center point of each segmented region, or the weighting coefficients of the plurality of intermediate tone values ​​are related to the area proportion of the segmented regions corresponding to the plurality of target metadata information units respectively.

[0028] This application uses one of the above methods to obtain the target tone mapping value of the first pixel, and can generate the tone mapping curve of the first pixel by referring to the metadata information unit of one or more segmented regions, thereby improving the accuracy of tone mapping.

[0029] Secondly, this application provides a tone mapping method for panoramic images, comprising: obtaining at least one mapping point included in a first segmented region, wherein the first segmented region is one of a plurality of segmented regions included in a panoramic video three-dimensional spherical representation panoramic image, the panoramic video three-dimensional spherical representation panoramic image and a panoramic video two-dimensional planar projection image to be processed have a mapping relationship, and the at least one mapping point corresponds to at least one pixel on the panoramic video two-dimensional planar projection image; generating metadata information units of the first segmented region based on the at least one pixel; and writing the metadata information units of the first segmented region into the bitstream.

[0030] This application divides the panoramic video three-dimensional spherical representation of the panoramic image into multiple segmented regions. Based on the pixels corresponding to the mapping points contained in each segmented region, a metadata information unit is generated for that segmented region. The metadata information unit guides the display end to generate a tone mapping curve corresponding to that segmented region, thereby realizing tone mapping from image to display. This not only allows for the determination of matching tone mapping parameters for different dynamic range regions on the panoramic image, but also conforms to the display dynamic range of the display end, improving the processing efficiency of panoramic images.

[0031] In one possible implementation, after generating the metadata information unit of the first segmented region based on the at least one pixel, the method further includes: when the histogram and / or brightness of the first segmented region and the second segmented region meet the set conditions, fusing the metadata information unit of the first segmented region and the second metadata information unit of the second segmented region to obtain the metadata information units of the first segmented region and the second segmented region, wherein the second segmented region is one of the plurality of segmented regions.

[0032] This application fuses metadata information units from two or more segmented regions with small differences in histogram and / or brightness, thereby reducing the amount of metadata information units carried in the bitstream.

[0033] In one possible implementation, before obtaining at least one mapping point included in the first segmented region, the method further includes: mapping the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic image of the panoramic video; and dividing the three-dimensional spherical panoramic image of the panoramic video to obtain the plurality of segmented regions.

[0034] In one possible implementation, dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the multiple segmented regions includes: dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the multiple segmented regions according to a preset division method.

[0035] In one possible implementation, dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the multiple segmented regions includes: dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the multiple segmented regions according to a division method determined based on preset rules.

[0036] In one possible implementation, the multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or...

[0037] The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

[0038] In one possible implementation, dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the plurality of segmented regions includes: clustering the plurality of pixels included in the panoramic video two-dimensional planar projection image to obtain a plurality of pixel set; mapping the plurality of pixels to the panoramic video three-dimensional spherical representation panoramic image respectively; and taking the region formed by the mapping points corresponding to the pixels included in the first pixel set on the panoramic video three-dimensional spherical representation panoramic image as the first segmented region, wherein the first pixel set is one of the plurality of pixel set, and the first segmented region is one of the plurality of segmented regions.

[0039] In one possible implementation, dividing the panoramic video three-dimensional spherical representation panoramic image to obtain the plurality of segmented regions includes: mapping a plurality of pixels included in the panoramic video two-dimensional planar projection image to the panoramic video three-dimensional spherical representation panoramic image to obtain a plurality of mapping points; clustering the plurality of mapping points to obtain a plurality of mapping point sets; and using the region formed by the mapping points included in the first mapping point set as the second segmented region, wherein the first mapping point set is one of the plurality of mapping point sets, and the second segmented region is one of the plurality of segmented regions.

[0040] This application can use one of the above-mentioned segmentation methods to divide the panoramic image of the three-dimensional spherical representation of the panoramic video into multiple segmented regions, which can be adapted to the pixel distribution of the two-dimensional planar projection image of the panoramic video. This allows tone mapping to be achieved on a segmented region basis within a small dynamic range, thereby improving tone mapping efficiency.

[0041] Thirdly, this application provides a display device comprising: a partitioning module, a determining module, and a mapping module. Wherein,

[0042] A determining module is used to determine one or more target metadata information units for a first pixel from multiple metadata information units, wherein the multiple metadata information units are obtained by parsing the bitstream, and the first pixel is any pixel in the two-dimensional planar projection map of the panoramic video to be processed. The multiple metadata information units correspond to multiple segmented regions included in the three-dimensional spherical panoramic image of the panoramic video, and the two-dimensional planar projection map of the panoramic video and the three-dimensional spherical panoramic image of the panoramic video have a mapping relationship. A mapping module is used to perform tone mapping on the pixel value of the first pixel according to the one or more target metadata information units to obtain the target tone mapping value of the first pixel.

[0043] In one possible implementation, a segmentation module is used to obtain the multiple segmented regions.

[0044] In one possible implementation, the segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a preset segmentation method to obtain the multiple segmented regions.

[0045] In one possible implementation, the segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to the segmentation method obtained by parsing the bitstream, so as to obtain the multiple segmented regions.

[0046] In one possible implementation, the multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or...

[0047] The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

[0048] In one possible implementation, the segmentation module is specifically used to cluster multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; map the multiple pixels to the three-dimensional spherical panoramic image of the panoramic video respectively; and take the region formed by the mapping points of the pixels included in the first pixel set on the three-dimensional spherical panoramic image of the panoramic video as the first segmentation region, wherein the first pixel set is one of the multiple pixel sets, and the first segmentation region is one of the multiple segmentation regions.

[0049] In one possible implementation, the segmentation module is specifically used to map multiple pixels included in the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic representation map of the panoramic video to obtain multiple mapping points; cluster the multiple mapping points to obtain multiple mapping point sets; and take the region formed by the mapping points included in the first mapping point set as the second segmentation region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

[0050] In one possible implementation, the segmentation module is specifically used to obtain the multiple segmented regions based on the indication information of the multiple segmented regions obtained by parsing the bitstream.

[0051] In one possible implementation, the determining module is specifically used to determine the correspondence between the plurality of metadata information units and the plurality of segmented regions, wherein one metadata information unit corresponds to one or more segmented regions; determine one or more target segmented regions according to a set mapping point; when there is only one target segmented region, determine the metadata information unit corresponding to the one target segmented region as the one target metadata information unit; or, when there are multiple target segmented regions, determine the metadata information units corresponding to the multiple target segmented regions as the multiple target metadata information units.

[0052] In one possible implementation, the determining module is specifically used to extract the current metadata information unit from the plurality of metadata information units in a first preset order; extract the current segmentation region from the plurality of segmentation regions in a second preset order; and establish a correspondence between the current segmentation region and the current metadata information unit.

[0053] In one possible implementation, the determining module is specifically used to extract the current metadata information unit from the plurality of metadata information units according to a first preset order; extract the current segmented region from the plurality of segmented regions according to the traversal order obtained by parsing the bitstream; and establish a correspondence between the current segmented region and the current metadata information unit.

[0054] In one possible implementation, the determining module is specifically configured to extract the current metadata information unit from the plurality of metadata information units according to a first preset order; obtain one or more coordinates included in the current metadata information unit; determine one or more mapping points on the panoramic video three-dimensional spherical panoramic image based on the one or more coordinates; when there is only one mapping point, establish a correspondence between the segmented region to which the mapping point belongs and the current metadata information unit; when there are multiple mapping points, establish a correspondence between at least one segmented region to which the multiple mapping points belong and the current metadata information unit.

[0055] In one possible implementation, the set mapping point is the first mapping point of the first pixel on the panoramic image of the three-dimensional spherical representation of the panoramic video; the determining module is specifically used to determine a first coverage area on the panoramic image of the three-dimensional spherical representation of the panoramic video with the first mapping point as the center; when the first coverage area includes a segmented region, the segmented region is determined as the target segmented region; or, when the first coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

[0056] In one possible implementation, the set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical representation of the panoramic video; the determining module is specifically used to determine the second segmentation region to which the viewpoint center belongs, the plurality of segmentation regions including the second segmentation region; and to determine the second segmentation region as the target segmentation region.

[0057] In one possible implementation, the set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video; the determining module is specifically used to determine a second coverage area on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video with the viewpoint center as the center; when the second coverage area includes a segmented region, the segmented region is determined as the target segmented region; or, when the second coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

[0058] In one possible implementation, the mapping module is specifically configured to obtain one or more tone mapping curves based on the one or more target metadata information units; when there is only one tone mapping curve, the pixel value of the first pixel is tone mapped according to the one tone mapping curve to obtain the target tone mapping value; or, when there are multiple tone mapping curves, the pixel value of the first pixel is tone mapped according to the multiple tone mapping curves respectively to obtain multiple intermediate tone values ​​of the first pixel; and the target tone mapping value is obtained based on the multiple intermediate tone values.

[0059] In one possible implementation, the mapping module is specifically used to determine the median value among the plurality of intermediate hue values ​​as the target hue mapping value; or, to obtain the target hue mapping value by performing a weighted average of the plurality of intermediate hue values, wherein the weighting coefficients of the plurality of intermediate hue values ​​are preset, or the weighting coefficients of the plurality of intermediate hue values ​​are related to the distance between the first pixel and the center point of each segmented region, or the weighting coefficients of the plurality of intermediate hue values ​​are related to the area proportion of the segmented regions corresponding to the plurality of target metadata information units respectively.

[0060] Fourthly, this application provides a front-end device comprising: a partitioning module, an acquisition module, and a generation module. Wherein,

[0061] An acquisition module is used to acquire at least one mapping point included in a first segmented region, wherein the first segmented region is one of multiple segmented regions included in a panoramic image of a three-dimensional spherical representation of a panoramic video, the panoramic image of the three-dimensional spherical representation of a panoramic video and the two-dimensional planar projection image of the panoramic video to be processed have a mapping relationship, and the at least one mapping point corresponds to at least one pixel on the two-dimensional planar projection image of the panoramic video; a generation module is used to generate metadata information units of the first segmented region based on the at least one pixel; and write the metadata information units of the first segmented region into the bitstream.

[0062] In one possible implementation, the generation module is further configured to fuse the metadata information unit of the first segmented region and the second metadata information unit of the second segmented region to obtain the metadata information unit of the first segmented region and the second segmented region when the histogram and / or brightness of the first segmented region and the second segmented region meet the set conditions, wherein the second segmented region is one of the plurality of segmented regions.

[0063] In one possible implementation, a segmentation module is used to map the two-dimensional planar projection image of the panoramic video to the three-dimensional spherical panoramic image of the panoramic video; and to segment the three-dimensional spherical panoramic image of the panoramic video to obtain the multiple segmented regions.

[0064] In one possible implementation, the segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a preset segmentation method to obtain the multiple segmented regions.

[0065] In one possible implementation, the segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a segmentation method determined based on preset rules to obtain the multiple segmented regions.

[0066] In one possible implementation, the multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or...

[0067] The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

[0068] In one possible implementation, the segmentation module is specifically used to cluster multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; map the multiple pixels to the three-dimensional spherical panoramic image of the panoramic video respectively; and take the region formed by the mapping points of the pixels included in the first pixel set on the three-dimensional spherical panoramic image of the panoramic video as the first segmentation region, wherein the first pixel set is one of the multiple pixel sets, and the first segmentation region is one of the multiple segmentation regions.

[0069] In one possible implementation, the segmentation module is specifically used to map multiple pixels included in the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic representation map of the panoramic video to obtain multiple mapping points; cluster the multiple mapping points to obtain multiple mapping point sets; and take the region formed by the mapping points included in the first mapping point set as the second segmentation region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

[0070] Fifthly, this application provides a terminal device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described in any one of the first or second aspects above.

[0071] Sixthly, this application provides a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects above.

[0072] In a seventh aspect, this application provides a computer program product comprising computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first to second aspects above. Attached Figure Description

[0073] Figure 1 This is an example diagram of dynamic range mapping in a real-world imaging process.

[0074] Figure 2 A schematic diagram of a curve representing the PQ photoelectric transfer function;

[0075] Figure 3 A schematic diagram of the HLG photoelectric transfer function;

[0076] Figure 4 This is a schematic diagram of the photoelectric transfer function of SLF;

[0077] Figure 5 A schematic diagram of a sigmoidal curve;

[0078] Figure 6 A schematic diagram of a Bezier curve;

[0079] Figure 7 This is a schematic diagram of an S-shaped curve;

[0080] Figure 8 This is an exemplary block diagram of the panoramic video playback system of this application;

[0081] Figure 9 This is an exemplary structural diagram of a terminal device;

[0082] Figure 10a Example diagram of latitude and longitude mapping;

[0083] Figure 10b A schematic diagram showing the mapping of a sphere to a regular polyhedron and its corresponding unfolded image;

[0084] Figure 10c and Figure 10d A schematic diagram illustrating the mapping relationship between a sphere and a hexahedron;

[0085] Figure 10e A schematic diagram showing the mapping relationship between spheres and hexahedrons and octahedrons;

[0086] Figures 11a to 11d This is an exemplary schematic diagram of several segmented regions for representing a panoramic image in a three-dimensional spherical format using panoramic video, as described in this application.

[0087] Figure 12 The flowchart of the tone mapping method for panoramic images in this application is shown in step 1200.

[0088] Figure 13 The flowchart of the tone mapping method for panoramic images in this application is shown in step 1300.

[0089] Figure 14 This is an exemplary structural diagram of the display device 1400 of this application;

[0090] Figure 15 This is an exemplary structural diagram of the front-end device 1500 of this application. Detailed Implementation

[0091] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0092] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0093] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0094] The following are some related technologies involved in the embodiments of this application:

[0095] I. Dynamic range

[0096] Dynamic range is used in many fields to represent the ratio of the maximum to the minimum value of a variable. In digital imaging, dynamic range is used to represent the ratio between the maximum and minimum grayscale values ​​within the displayable range of an image.

[0097] In nature, the brightness of a night scene under the stars is approximately 0.001 cd / m². 2 The sun itself has a brightness of up to 10. 9 cd / m 2 The dynamic range can reach 10. 9 / 0.001=1012 cd / m 2 The magnitude is not specified. However, in the real world, the brightness of the sun and the brightness of starlight are not simultaneously obtained. Therefore, for the same scene in the real world, its dynamic range is usually within 10^6. -3 cd / m 2 Up to 10 6 cd / m 2 The range between these values ​​is called high dynamic range (HDR). In most current color digital images, the grayscale values ​​of the red (R), green (G), and blue (B) channels are each stored using one byte. That is, the grayscale ranges of the R, G, and B channels are 0 to 255, respectively. This range of 0 to 255 is the dynamic range of the image, also known as low dynamic range (LDR).

[0098] II. Optical-electro transfer function (OETF)

[0099] The imaging process of a digital camera is actually a mapping from the high dynamic range of the real world to the low dynamic range of the digital image. Figure 1 An example diagram of dynamic range mapping in real-world imaging processes, such as... Figure 1 As shown, in the real world, in addition to the brightness of starlight and the sun, the brightness of moonlight is also included, which is 1 cd / m². 2 The indoor lighting brightness is 100 cd / m². 2 The outdoor brightness during cloudy weather is 500 cd / m². 2 The outdoor brightness on a sunny day is 2000 cd / m². 2 In the real world, the brightness range is 100 cd / m². 2 Up to 2000 cd / m 2 The brightness range corresponding to the storage method of the display device is 1 cd / m². 2 Up to 200 cd / m 2 A mapping relationship is formed.

[0100] Since the storage methods of display devices cannot achieve the high brightness levels of the real world, an electro-optical transfer function (EOTF) is needed to represent the brightness in the real world as corresponding to the brightness levels stored in the display device. For example, the brightness in the real world is 10000 cd / m². 2 If the display device uses 10 bits to store brightness information, then the maximum value that can be stored using this storage method is 1023, therefore 10000 cd / m² can be stored. 2 It is represented as 1023.

[0101] Early display devices were cathode ray tube (CRT) displays, whose photoelectric transfer function was the gamma function. This gamma function is defined in ITU-R Recommendation BT.1886.

[0102]

[0103] With the upgrading of display devices, the illuminance range of display devices continues to increase, with existing HDR displays reaching an illuminance of 600 cd / m². 2 High-end HDR displays can achieve an illuminance of 2000 cd / m². 2 Therefore, improved photoelectric transfer functions are needed to adapt to the upgrade of display devices. At present, the common photoelectric transfer functions are perception quantization (PQ) photoelectric transfer function, hybrid log-gamma (HLG) photoelectric transfer function, and scene luminance fidelity (SLF) photoelectric transfer function.

[0104] (1) PQ photoelectric transfer function: Unlike the traditional Gamma function, the PQ photoelectric transfer function is proposed based on the contrast perception model of the human eye under different brightness levels. The PQ photoelectric transfer function represents the transformation relationship between the linear signal values ​​of image frame pixels and the nonlinear signal values ​​in the PQ domain. Figure 2 This is a schematic diagram of the PQ photoelectric transfer function. The PQ photoelectric transfer function can be expressed as:

[0105]

[0106] The parameters corresponding to R, G, and B in the above formula can be obtained through the formula. The calculation is performed, where L represents the linear signal value of the image frame pixel, which is normalized to [0,1]; L' represents the nonlinear signal value in the PQ domain, which ranges from [0,1]; m1, m2, c1, c2, and c3 are all PQ photoelectric transfer coefficients.

[0107]

[0108]

[0109]

[0110]

[0111]

[0112] (2) HLG photoelectric transfer function: This is an improvement on the traditional Gamma curve. The HLG photoelectric transfer function uses the traditional Gamma curve in the lower segment and supplements it with a log curve in the higher segment. The HLG photoelectric transfer function represents the conversion relationship between the linear signal values ​​of image frame pixels and the nonlinear signal values ​​in the HLG domain. Figure 3 This is a schematic diagram of the HLG photoelectric transfer function. The HLG photoelectric transfer function can be expressed as:

[0113]

[0114] Where L represents the linear signal value of the image frame pixel, and its value range is [0,12]; L' represents the nonlinear signal value in the HLG domain, and its value range is [0,1]; a, b and c are all HLG photoelectric transfer coefficients, a = 0.17883277, b = 0.28466892, c = 0.55991073.

[0115] (3) SLF photoelectric transfer function: Obtained based on the brightness distribution of HDR scenes, while satisfying the optical characteristics of the human eye. The SLF photoelectric transfer function represents the conversion relationship between the linear signal values ​​of image frame pixels and the nonlinear signal values ​​in the SLF domain. Figure 4 This is a schematic diagram of the SLF photoelectric transfer function. The SLF photoelectric transfer function can be expressed as:

[0116]

[0117] The parameters corresponding to R, G, and B in the above formula can be obtained through the formula.

[0118] The calculation is performed, where L represents the linear signal value of the pixel in the image frame, and its value is normalized to [0,1]; L' represents the nonlinear signal value in the PQ domain, and its value range is [0,1]; p, m, a and b are all SLF photoelectric transfer coefficients, p = 2.3, m = 0.14, a = 1.12762, b = -0.12762.

[0119] III. Mapping of Dynamic Range

[0120] Dynamic range mapping can be applied to adapt the front-end HDR signal to the display device with HDR capabilities. For example, if the front-end receives a signal with 4000 cd / m², the dynamic range mapping can be used to adapt the signal to the display device with HDR capabilities. 2 The light signal, while the HDR display capability of the display device is 500 cd / m². 2 4000cd / m 2 The light signal is mapped to 500 cd / m 2On display devices, this is a high-to-low tone mapping (TM) process. Dynamic range mapping can also be applied to adapt the front-end SDR signal to the display device with HDR, for example, when the front-end receives a signal of 100 cd / m². 2 The light signal, while the HDR display capability of the display device is 2000 cd / m². 2 100cd / m 2 The light signal is mapped to 2000 cd / m 2 On display devices, it is a TM process from low to high.

[0121] Currently, dynamic range (TM) mapping methods can be divided into two types: static and dynamic. Static mapping methods perform a single TM process based on the same video content or hard drive content, meaning the mapping curve is usually the same for all scenarios. The advantage of this method is that it requires less data to be carried in the video and the processing flow is relatively simple. The disadvantage is that using the same mapping curve for all scenarios can lead to information loss in some scenes. For example, if the mapping curve focuses on protecting bright areas, details may be lost or completely invisible in extremely dark scenes, affecting the video display quality. Dynamic mapping methods dynamically adjust the mapping curve based on specific regions, for each scene or each frame. The advantage of this method is that it allows for differentiated processing for different scenes or frames. The disadvantage is that it requires carrying relevant scene information in each frame or scene, resulting in a larger amount of data to be carried in the video.

[0122] IV. TM Technology

[0123] (1) TM process based on sigmoidal curve

[0124] Figure 5 This is a schematic diagram of a sigmoidal curve.

[0125] (2) TM process based on Bézier curve

[0126] Figure 6 This is a schematic diagram of a Bezel curve.

[0127] (3) TM process based on S-curve perceived by human eye

[0128] Figure 7 This is a schematic diagram of an S-curve. The photoelectric transfer function corresponding to the S-curve can be expressed as:

[0129]

[0130] Where L and L' are normalized electrical or optical signals, respectively, and their values ​​are both in the range of [0,1]; the value range of a is [0,1]; the value range of b is [0,1]; the value ranges of p, n and m are all in the range of [0,N], where N is a rational number greater than 0.1; k1, k2 and k3 are all rational numbers.

[0131] V. Panoramic Video

[0132] Panoramic videos offer an immersive visual experience. This is mainly due to: (1) providing spherical scene content depicting 360 degrees horizontally and 180 degrees vertically; and (2) allowing users to freely choose which direction to view through a VR headset. Compared to the narrow field of view of ordinary 2D videos (33 degrees horizontally and 19 degrees vertically), panoramic videos cover a much wider area.

[0133] Correspondingly, panoramic video introduces a more severe problem of insufficient dynamic range for device display. Because 2D video has a smaller field of view and similar lighting conditions, its dynamic range is still within a controllable range; however, panoramic video has a larger field of view. For example, if the video content is a daytime scene, it may include both very bright areas such as the sun in the sky and darker areas such as the interior of buildings or shadows. Its dynamic range will be very large, which will bring great challenges to description and display.

[0134] VI. Metadata Information

[0135] The front end (video capture and / or production) will include parameters related to the mapping curve in the metadata information sent to the display end (video display).

[0136] (1) sigmoidal curve

[0137] The metadata definition related to the sigmoidal curve proposed in St2094-10 not only includes statistical values ​​such as the maximum value (Maximum PQ-encoded maxRGB), minimum value (Minimum PQ-encoded maxRGB), and average value (Average PQ-encoded maxRGB) of video brightness, but also includes sigmoid curve-related parameters such as tone mapping offset, tone mapping gain, and tone mapping gamma, which are used to directly generate the sigmoidal curve.

[0138] However, the generation method of the above parameters is relatively fixed, and the parameters contained in the metadata information cannot provide more flexibility for curve generation.

[0139] (2) Bezel curve

[0140] The metadata information definition based on Bézier curves proposed in St2094-40 includes histogram information (Distribution MaxRGB) and Bézier curve-related parameters (bezier curve anchors) for directly generating Bézier curves.

[0141] In addition, the ST2094 series standards all include the target system display maximum (peak) luminance (TSDAPL).

[0142] However, the above parameters correspond to TSDAPL when generated on the front end, but different display devices use the same curve on the display end, which cannot achieve the best display effect.

[0143] (3) S-curve

[0144] The metadata information can include statistical values ​​such as the maximum, minimum, and average values ​​of video brightness, as well as parameters related to S-curves such as p, m, a, b, n, K1, K2, and K3.

[0145] Figure 8 This is an exemplary block diagram of the panoramic video playback system of this application. Figure 8 As shown, the panoramic video playback system is divided into a front-end and a display end. The front-end includes modules for acquiring and / or producing panoramic video, while the display end includes HDR and / or SDR display modules. The front-end transmits the acquired panoramic video data to the display end after preprocessing, also carrying metadata information related to the panoramic video data. The display end, based on the panoramic video data and the corresponding metadata information, performs frame-by-frame enhancement processing on the multiple image frames contained in the panoramic video to obtain images with excellent color, brightness, and contrast, which are then displayed.

[0146] In this application, the front-end and the display end can be independent and different physical devices. For example, the front-end can be a device with video capture capabilities, such as a camera, image processing machine, or other image capture device, while the display end can be a device with panoramic video playback capabilities, such as virtual reality (VR) glasses, a mobile phone, or a tablet. In this case, a wireless or wired connection can be established between the front-end and the display end. The wireless connection can employ technologies such as Long Term Evolution (LTE), 5th Generation (5G) mobile communication, and future mobile communication technologies. Wireless connections can also include technologies such as Wireless Fidelity (WiFi), Bluetooth, and Near Field Communication (NFC). Wired connections can include Ethernet connections, local area network (LAN) connections, etc. No specific limitations are imposed on this.

[0147] This application can also integrate the functions of the front end and the display end onto the same physical device, such as mobile phones, tablets, and other terminal devices with video recording capabilities. This application can also integrate some functions of the front end and some functions of the display end onto the same physical device. No specific limitations are made in this regard.

[0148] Figure 9 This is an exemplary structural diagram of a terminal device. Figure 9 As shown, the terminal device 900 includes: an application processor 901, a microcontroller unit (MCU) 902, a memory 903, a modem 904, a radio frequency (RF) module 905, a Wireless-Fidelity (Wi-Fi) module 906, a Bluetooth module 907, a sensor 908, an input / output (I / O) device 909, a positioning module 910, and other components. These components can communicate via one or more communication buses or signal lines. The aforementioned communication bus or signal line can be the CAN bus provided in this application. Those skilled in the art will understand that the terminal device 900 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0149] The following is combined with Figure 9 A detailed introduction to each component of the terminal device 900:

[0150] The application processor 901 is the control center of the terminal device 900, connecting various components of the terminal device 900 via various interfaces and buses. In some embodiments, the processor 901 may include one or more processing units.

[0151] The memory 903 stores computer programs, such as Figure 9 The diagram shows an operating system 911 and an application program 912. An application processor 901 is configured to execute computer programs stored in memory 903, thereby implementing the functions defined by those programs. For example, the application processor 901 executes the operating system 911 to implement various functions of the operating system on the terminal device 900. Memory 903 also stores other data besides the computer programs, such as data generated during the operation of the operating system 911 and the application program 912. Memory 903 is a non-volatile storage medium, generally including main memory and secondary storage. Main memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), or cache. Secondary storage includes, but is not limited to, flash memory, hard disks, optical disks, and universal serial bus (USB) disks. Computer programs are typically stored on secondary storage, and the processor loads the program from secondary storage into main memory before executing it.

[0152] The memory 903 can be independent and connected to the application processor 901 via a bus; the memory 903 can also be integrated with the application processor 901 into a chip subsystem.

[0153] MCU 902 is a coprocessor used to acquire and process data from sensor 908. MCU 902 has lower processing power and power consumption than application processor 901, but features an "always-on" characteristic, allowing it to continuously collect and process sensor data while application processor 901 is in sleep mode, ensuring normal sensor operation with extremely low power consumption. In one embodiment, MCU 902 can be a sensor hub chip. Sensor 908 may include a light sensor and a motion sensor. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of display 9091 according to the ambient light level, and the proximity sensor can turn off the display power when the terminal device 900 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity. Sensor 908 may also include other sensors such as a gyroscope, barometer, hygrometer, thermometer, and infrared sensor, which will not be described in detail here. The MCU 902 and sensor 908 can be integrated onto the same chip or are separate components connected via a bus.

[0154] The modem 904 and RF module 905 constitute the communication subsystem of the terminal device 900, used to implement the main functions of the wireless communication standard protocol. The modem 904 is used for encoding / decoding, signal modulation / demodulation, and equalization. The RF module 905 is used for receiving and transmitting wireless signals, and includes, but is not limited to, an antenna, at least one amplifier, a coupler, and a duplexer. The RF module 905 works in conjunction with the modem 904 to achieve wireless communication functionality. The modem 904 can be a standalone chip or integrated with other chips or circuits to form a system-on-a-chip (SoC) or integrated circuit. These chips or integrated circuits can be used in all terminal devices that implement wireless communication functions, including: mobile phones, computers, laptops, tablets, routers, wearable devices, automobiles, and home appliances.

[0155] The terminal device 900 can also use a Wi-Fi module 906, a Bluetooth module 907, etc., for wireless communication. The Wi-Fi module 906 provides network access to the terminal device 900 in accordance with Wi-Fi related standard protocols. The terminal device 900 can access a Wi-Fi access point through the Wi-Fi module 906 and thus access the Internet. In some other embodiments, the Wi-Fi module 906 can also act as a Wi-Fi wireless access point, providing Wi-Fi network access to other terminal devices. The Bluetooth module 907 enables short-range communication between the terminal device 900 and other terminal devices (such as mobile phones, smartwatches, etc.). In the embodiments of this application, the Wi-Fi module 906 can be an integrated circuit or a Wi-Fi chip, and the Bluetooth module 907 can be an integrated circuit or a Bluetooth chip.

[0156] The positioning module 910 is used to determine the geographical location of the terminal device 900. It can be understood that the positioning module 910 can specifically be a receiver for a global positioning system (GPS), such as the BeiDou Navigation Satellite System or the Russian GLONASS.

[0157] The Wi-Fi module 906, Bluetooth module 907, and positioning module 910 can each be a separate chip or integrated circuit, or they can be integrated together. For example, in one embodiment, the Wi-Fi module 906, Bluetooth module 907, and positioning module 910 can be integrated onto the same chip. In another embodiment, the Wi-Fi module 906, Bluetooth module 907, positioning module 910, and MCU 902 can also be integrated into the same chip.

[0158] Input / output devices 909 include, but are not limited to: display 9091, touch screen 9092, and audio circuit 9093, etc.

[0159] The touchscreen 9092 can collect touch events from the user of the terminal device 900 on or near the touchscreen 9092 (such as user actions using a finger, stylus, or any suitable object on or near the touchscreen 9092), and send the collected touch events to other devices (such as the application processor 901). User actions near the touchscreen 9092 can be called hover touch; through hover touch, the user can select, move, or drag objects (such as icons) without directly touching the touchscreen 9092. Furthermore, the touchscreen 9092 can be implemented using various types of touchscreens, including resistive, capacitive, infrared, and surface acoustic wave.

[0160] A display (also called a screen) 9091 is used to display information input by the user or information shown to the user. The display can be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other similar methods. A touchscreen 9092 can be placed over the display 9091. When the touchscreen 9092 detects a touch event, it transmits the information to the application processor 901 to determine the type of touch event. The application processor 901 then provides corresponding visual output on the display 9091 based on the type of touch event. Although... Figure 9 In this embodiment, the touchscreen 9092 and the display 9091 are two separate components for implementing the input and output functions of the terminal device 900. However, in some embodiments, the touchscreen 9092 and the display 9091 can be integrated to achieve the input and output functions of the terminal device 900. Furthermore, the touchscreen 9092 and the display 9091 can be configured as a full-panel display on the front of the terminal device 900 to achieve a borderless structure.

[0161] Audio circuit 9093, speaker 9094, and microphone 9095 provide an audio interface between the user and terminal device 900. Audio circuit 9093 converts received audio data into electrical signals and transmits them to speaker 9094, where speaker 9094 converts them into sound signals for output. On the other hand, microphone 9095 converts collected sound signals into electrical signals, which are received by audio circuit 9093, converted into audio data, and then transmitted to, for example, another terminal device via modem 904 and radio frequency module 905, or output to memory 903 for further processing.

[0162] In addition, the terminal device 900 may also have fingerprint recognition functionality. For example, a fingerprint acquisition device can be configured on the back of the terminal device 900 (e.g., below the rear camera) or on the front of the terminal device 900 (e.g., below the touchscreen 9092). Alternatively, the fingerprint acquisition device can be integrated into the touchscreen 9092 to implement fingerprint recognition functionality; that is, the fingerprint acquisition device can be integrated with the touchscreen 9092 to achieve the fingerprint recognition function of the terminal device 900. In this case, the fingerprint acquisition device is configured in the touchscreen 9092, and may be part of the touchscreen 9092 or configured in the touchscreen 9092 in other ways. The main component of the fingerprint acquisition device in this embodiment is a fingerprint sensor, which can employ any type of sensing technology, including but not limited to optical, capacitive, piezoelectric, or ultrasonic sensing technologies.

[0163] Furthermore, the operating system 911 running on the terminal device 900 can provide... This application does not impose any restrictions on other operating systems, including those used in the embodiments of the present application.

[0164] With Taking the terminal device 900 with an operating system as an example, the terminal device 900 can be logically divided into a hardware layer, an operating system 911, and an application layer. The hardware layer includes hardware resources such as the application processor 901, MCU 902, memory 903, modem 904, Wi-Fi module 906, sensor 908, and positioning module 910, as described above. The application layer includes one or more applications, such as application 912, which can be any type of application, such as a social application, e-commerce application, or browser. The operating system 911, as software middleware between the hardware layer and the application layer, is a computer program that manages and controls hardware and software resources.

[0165] In one embodiment, the operating system 911 includes a kernel, a hardware abstraction layer (HAL), libraries and runtime, and a framework. The kernel provides low-level system components and services, such as power management, memory management, thread management, and hardware drivers. Hardware drivers include Wi-Fi drivers, sensor drivers, and positioning module drivers. The HAL encapsulates the kernel drivers, providing interfaces to the framework and shielding them from low-level implementation details. The HAL runs in user space, while the kernel drivers run in kernel space.

[0166] Libraries and runtimes, also known as runtime libraries, provide the necessary library files and execution environment for executable programs at runtime. In one embodiment, libraries and runtimes include the Android runtime (ART), libraries, and scene package runtimes. ART is a virtual machine or virtual machine instance capable of converting application bytecode into machine code. Libraries are program libraries that provide support for executable programs at runtime, including browser engines (such as WebKit), script execution engines (such as JavaScript engines), and graphics processing engines. The scene package runtime is the runtime environment for scene packages, mainly including the page execution environment and the script execution environment. The page execution environment parses page code in HTML, CSS, and other formats by calling corresponding libraries, while the script execution environment parses and executes code or executable files implemented in scripting languages ​​such as JavaScript by calling corresponding function libraries.

[0167] The framework provides various basic public components and services for applications in the application layer, such as window management, location management, and so on. In one embodiment, the framework includes a geofencing service, a policy service, a notification manager, and so on.

[0168] The functions of each component of the operating system 911 described above can be implemented by the application processor 901 executing the program stored in the memory 903.

[0169] Those skilled in the art can understand that terminal device 900 may include more than Figure 9 Fewer or more components are shown. Figure 9 The terminal device shown only includes components that are more relevant to the various implementations disclosed in this application.

[0170] As mentioned above, panoramic video can provide spherical scene content with a horizontal 360-degree and a vertical 180-degree angle. When a user watches the panoramic video, with the user's head as the center point, the user's head can rotate 360 ​​degrees horizontally and tilt 180 degrees vertically. Therefore, the user's field of view can reach 360 degrees horizontally and 180 degrees vertically. It can be seen that if the panoramic video is simulated as the surface of a sphere, as long as the user's head is at the center of the sphere, the user can view the complete panoramic video by rotating and / or tilting their head.

[0171] Therefore, the panoramic view of the virtual user can be represented by a three-dimensional spherical representation of the panoramic video based on the multi-frame three-dimensional (3D) panoramic images contained in the panoramic video. Any point on the panoramic view of the panoramic video can be represented by spherical coordinates, and any pixel on the 3D panoramic image can find a corresponding point on the panoramic view of the panoramic video. This point can be called the mapping point of the aforementioned pixel.

[0172] Because spherical three-dimensional (3D) panoramic images are not easily represented, stored, and indexed, related technologies typically unfold the 3D panoramic image into a plane to obtain a two-dimensional (2D) panoramic image (in this application, the 2D panoramic image can be referred to as a two-dimensional planar projection of the panoramic video), and then compress, process, store, and transmit this 2D panoramic image. The operation of unfolding a 3D panoramic image to obtain a 2D panoramic image is called mapping. Currently, there are various mapping methods, and correspondingly, there are various 2D panoramic image formats.

[0173] 1. The most common 2D panoramic image format is called a latitude and longitude map, such as... Figure 10a As shown, the content is a 3D panoramic image of the Earth unfolded into a 2D panoramic image. The images of the areas near the North and South Poles were obtained by stretching, resulting in severe distortion and data redundancy.

[0174] 2. To mitigate the strong distortion present in latitude and longitude maps, the 3D panoramic image can be projected onto the surface of a regular polyhedron, thus representing the sphere as a polyhedron composed of several equally sized triangular, square, or pentagonal faces. For example, a regular tetrahedron (such as...) Figure 10b As shown in (a) in the figure, a regular hexahedron (as shown in the figure) Figure 10b As shown in (b)), a regular octahedron (as shown in...) Figure 10b As shown in (c) in the figure), a regular dodecahedron (as shown in the figure) Figure 10b As shown in (d) in the figure, a regular icosahedron (as shown in the figure) Figure 10b (as shown in (e)). It should be noted that this application does not limit the polyhedron to a regular polyhedron; a sphere can also be represented as a polyhedron composed of several polygons of unequal size.

[0175] The polyhedron is then unfolded into a 2D panoramic image. Figure 10b Expand (a) in Figure 10b The 2D panoramic image shown in (f) is as follows. Figure 10b (b) in the middle expands into Figure 10b The 2D panoramic image shown in (g) is shown in the image. Figure 10b Expanding (c) into Figure 10b The 2D panoramic image shown in (h) is as follows. Figure 10b (d) in the middle expands into Figure 10b The 2D panoramic image shown in (i) is as follows. Figure 10b (e) in the middle expands into Figure 10b The 2D panoramic image shown in (j) is an example. These 2D panoramic images can be collectively referred to as polyhedral format panoramic images.

[0176] The specific projection process is as follows: A sphere is placed inside a polyhedron, making it an inscribed sphere. A line is drawn from the center of the sphere (or the body center of the polyhedron) to a point on the sphere's surface, and extended to intersect the polyhedron's surface. The pixel value at this point on the sphere is the pixel value at the intersection point on the polyhedron's surface. By combining this with interpolation methods, the pixel values ​​of all pixels on the polyhedron can be obtained. Alternatively, to obtain the pixel value of a specific pixel on the polyhedron, a line can be drawn directly from this pixel to the center of the sphere, intersecting at a point on the sphere's surface. The pixel value at this intersection point on the sphere is then used as the pixel value of that specific pixel.

[0177] Taking the hexahedral projection process as an example, the sphere is inscribed in the plane as shown in the figure. Figure 10c In the hexahedron ABCDEFGH shown, to obtain the pixel value at point M' on the polyhedron, a line is drawn connecting the center O of the sphere to M', intersecting at point M on the sphere. The pixel value at point M is then the pixel value at point M'. Following this method, the pixel values ​​of all pixels within the ABCD plane on plane A'B'C'D' can be obtained. The pixels within the ABCD plane constitute the surface image of the ABCD face, and plane A'B'C'D' is called the projection plane of the ABCD face. Similarly, the surface images of other faces of the hexahedron and their corresponding projection planes can be obtained. It should be noted that the surface image mentioned in this application is an N×N array composed of spatially adjacent pixels. These pixels are located on the same projection plane during the mapping process from the sphere to the polyhedron surface, and the pixel values ​​at these pixel positions are known.

[0178] After unfolding the polyhedron surface into a 2D image, each facet of the polyhedron surface becomes an image of a region or a sub-image in the 2D panoramic image. The facet of the polyhedron surface refers to the image of that region or sub-image, and the facet of the polyhedron surface refers to the covered area of ​​that region or sub-image in the 2D panoramic image. For example, Figure 10d The hexahedral surface of (a) unfolds into Figure 10d The 2D panoramic image in (b) will have its top face image on the hexahedron surface as the image of the top face. Figure 10d The sub-image in the upper left corner of (b) is called the top surface image of the 2D panoramic image. The top surface in the 2D panoramic image refers to the area covered by the top surface image. For any pixel in the top surface, the top surface is called the surface containing that pixel.

[0179] In image processing such as compression, it can be processed directly. Figure 10bThe irregular 2D images shown in (f) to (j) can also be processed by selecting the smallest rectangular region enclosing the irregular 2D image. The remaining portion of this rectangular region outside the irregular 2D image can be filled with default content, such as pure gray, pure black, or pure white. For example, Figure 10e (b) in the middle is Figure 10e Expanding (a) in the middle, we get Figure 10e (d) in the text is Figure 10e The result is a two-dimensional rectangular image obtained by expanding (c) and setting the pixel values ​​of the remaining parts to the corresponding gray values. Alternatively, these faces can be directly stitched together into a regular rectangle, thus avoiding the need for filling. In actual encoding processing, the images processed are generally rectangular images.

[0180] Using the methods described above or other methods not listed in this application, a 3D panoramic image can be projected into a 2D planar projection image of the panoramic video. Then, this 2D planar projection image can be mapped to a 3D spherical panoramic image, meaning that multiple pixels in the 2D planar projection image are mapped to multiple mapping points in the 3D spherical panoramic image. This transforms the processing of the 2D planar projection image into the processing of the 3D spherical panoramic image. Therefore, regardless of the format used to represent the 2D planar projection image of the panoramic video, it can be mapped to the 3D spherical panoramic image, thus freeing the tone mapping of panoramic images from the limitations of the 2D panoramic video format.

[0181] In this application, the panoramic image representing a three-dimensional spherical surface in a panoramic video can be divided into multiple segmented regions. These multiple segmented regions can be obtained using any of the following methods:

[0182] 1. Multiple segmented regions are obtained by dividing the panoramic video into three-dimensional spherical representations of the panoramic image according to preset viewpoint interval rules.

[0183] In this application, the panoramic video 3D spherical representation of the panoramic image can be divided into multiple segmented regions with preset viewing angle intervals, for example, such as Figure 11a As shown, in the multiple segmented regions, the horizontal angles of two adjacent latitude segments are equal, and the pitch angles of two adjacent longitude segments are equal.

[0184] 2. Multiple segmented regions are obtained by dividing the panoramic image of the panoramic video into three-dimensional spherical representations along the latitude direction; and / or, multiple segmented regions are obtained by dividing the panoramic image of the panoramic video into three-dimensional spherical representations along the longitude direction.

[0185] In this application, the panoramic video can be divided into multiple segmented regions along the latitudinal direction using a three-dimensional spherical representation of the panoramic image. For example, ... Figure 11bAs shown, the panoramic video's 3D spherical representation is divided into two equal-sized segments; or, the panoramic video's 3D spherical representation is evenly divided into multiple segments along the longitude direction, for example, as... Figure 11c As shown, the panoramic video's 3D spherical representation can be divided into two equal-sized regions; alternatively, it can be divided into multiple regions along the latitude direction and also into multiple regions along the longitude direction, for example, as... Figure 11d As shown, the panoramic video is divided into four equal-sized vertical and horizontal spherical regions to represent the panoramic image.

[0186] It should be understood that the above examples of division methods do not limit the division method. Alternatively, the panoramic video 3D spherical representation of the panoramic image can be divided into two equal parts, and then each part can be divided into four equal sections. Or, the panoramic video 3D spherical representation of the panoramic image can be divided into two equal parts, and then each part can be divided into four equal sections, and finally, each quarter section can be divided into regions with equal viewing angle intervals. Furthermore, the division of the panoramic video 3D spherical representation of the panoramic image does not have to be equal, or all division steps do not have to be equal; this application does not impose specific limitations on this.

[0187] 3. Cluster the multiple pixels in the panoramic image to obtain multiple pixel sets, and map each pixel to a 3D spherical panoramic image of the panoramic video. The region formed by the mapping points of the pixels in the first pixel set on the 3D spherical panoramic image of the panoramic video is taken as the first segmentation region. This first pixel set is one of multiple pixel sets, and this first segmentation region is one of multiple segmentation regions.

[0188] 4. Map the multiple pixels of the panoramic image to a 3D spherical representation of the panoramic video to obtain multiple mapping points. Cluster these mapping points to obtain multiple mapping point sets. Use the region formed by the mapping points in the first mapping point set as the second segmentation region. The first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

[0189] The clustering methods mentioned above include:

[0190] a. Divide the panoramic video 3D spherical representation panoramic image into multiple segmented regions with equal viewing angle intervals according to the preset initial division method. It should be understood that the initial division method may also include any one of the division methods in 1 or 2 above, without specific limitation;

[0191] b. Determine the mapping points included in each of the multiple segmented regions. The above mapping process yields multiple mapping points on the panoramic image representing the three-dimensional spherical surface of the panoramic video. These mapping points can be identified by their spherical coordinates, determining which segmented region they belong to. It is possible that every segmented region contains mapping points, or some segmented regions may not contain mapping points.

[0192] c. Determine the color component value of the mapping point located at the center position of the i-th segmented region, or the average color component value of the i-th segmented region. The color component value can refer to at least one of the components Y, U, and V in the YUV color format; at least one of the components R, G, B, and max(R,G,B) in the RBG color format; and at least one of the components L, a, and b in the Lab color format. The average color component value of the i-th segmented region can refer to the average of the color component values ​​of the multiple mapping points included in the i-th segmented region. The i-th segmented region is one of multiple segmented regions; the processing procedure for each segmented region is illustrated here using the processing procedure for the i-th segmented region as an example. It should be understood that this application can also obtain other feature information of the i-th segmented region, such as the color component value of a specified mapping point in the i-th segmented region, without specific limitations. It should be noted that if the i-th segmented region does not have a mapping point located at the center, the average x-coordinate avgx and the average y-coordinate avgy of all mapping points in the i-th segmented region can be calculated, and the coordinates (avgx, avgy) can be used to replace the mapping point located at the center.

[0193] d1. Calculate the clustering distortion values ​​of other mapping points in the i-th segmentation region (using the j-th mapping point as an example below) and the mapping point located at the center position in the i-th segmentation region respectively:

[0194]

[0195] Among them, D j Q represents the clustering distortion value of the j-th mapping point. c Q represents the color component value of the mapping point located at the center of the i-th segmented region. j Represents the color component value of the j-th mapping point, (x c ,y c ,z c (x) represents the coordinates of the mapping point located at the center of the i-th segmented region. j ,y j ,z j ) represents the coordinates of the j-th mapping point. Normalization (Q) c -Q j Q can be used c and Qj The maximum range of values ​​is normalized to between 0 and 1.0. Normalization The average area of ​​the segmented regions (i.e., the area of ​​the sphere divided by the number of segmented regions) can be used. Assuming that the segmented regions are circles or squares, the radius (corresponding to the circular segmented region) or the side length (corresponding to the square segmented region) of the obtained segmented regions can be estimated. The calculated Euclidean distance is divided by the radius or the side length of the square. If it is greater than 1, it is set to 0, which can normalize it to between 0 and 1.0.

[0196] d2. Calculate the clustering distortion values ​​of other mapping points in the i-th segmentation region (using the j-th mapping point as an example below) and the mapping point located at the center position in the i-th segmentation region respectively:

[0197]

[0198] Among them, D j Qa represents the clustering distortion value of the j-th mapping point. c Q represents the average color component value of the i-th segmented region. j Represents the color component value of the j-th mapping point, (x c ,y c ,z c (x) represents the coordinates of the mapping point located at the center of the i-th segmented region. j ,y j ,z j ) represents the coordinates of the j-th mapping point.

[0199] The steps d1 and d2 above are two options to choose one of.

[0200] e. Determine the segmentation region to which the multiple mapping points belong based on the clustering distortion values ​​of the multiple mapping points contained in the i-th segmentation region. For example, if the clustering distortion value D of the j-th mapping point... j If the clustering distortion value D of the j-th mapping point is less than or equal to a preset threshold, then the j-th mapping point is determined to belong to the i-th segmentation region; if the clustering distortion value D of the j-th mapping point is less than or equal to a preset threshold, then the j-th mapping point is determined to belong to the i-th segmentation region. j If the value exceeds a preset threshold, then the j-th mapping point is determined not to belong to the i-th segmentation region. The clustering distortion value of the j-th mapping point is then recalculated using methods d1 and d2 described above. The difference is that the mapping point located at the center of the i-th segmentation region is replaced with the mapping point located at the center of a neighboring segmentation region of the i-th segmentation region. The recalculated clustering distortion value is then used to determine whether the j-th mapping point belongs to a neighboring segmentation region. This process continues until the segmentation region to which the j-th mapping point belongs is determined.

[0201] f. After determining the segmentation regions to which all mapping points belong using the above steps, the region formed by multiple mapping points belonging to the same segmentation region (e.g., the i-th segmentation region) after clustering (these multiple mapping points constitute a set of mapping points mentioned above) is the final segmentation region.

[0202] The difference between the above-mentioned partitioning method 3 and partitioning method 4 is that partitioning method 3 uses the above-mentioned clustering method to cluster multiple pixels in the panoramic image to obtain multiple pixel sets, and then maps the pixels to the three-dimensional sphere of the panoramic video to express the panoramic image. The region formed by the mapping points corresponding to the pixels in a pixel set is the final segmented region.

[0203] It should be understood that, in addition to the four classification methods mentioned above, other classification methods may also be used, and this application does not make any specific restrictions on them.

[0204] Figure 12 This is a flowchart of process 1200 for the tone mapping method of panoramic images in this application. Process 1200 can be performed by... Figure 8 The process 1200 is executed in the front end of the panoramic video playback system shown. Process 1200 is described as a series of steps or operations; it should be understood that process 1200 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 12 The execution order is shown. Process 1200 may include:

[0205] Step 1201: Obtain at least one mapping point included in the first segmentation region.

[0206] In this application, the front-end can use the above-described method to divide the panoramic image of the panoramic video into multiple segmented regions. Regarding the specific segmentation method, the front-end can use a preset method. For example, if the first segmentation method is preset, the front-end can divide the panoramic image of the panoramic video into multiple segmented regions with equal viewing angle intervals. This preset segmentation method is applicable to most panoramic images, improving image processing efficiency. Alternatively, the front-end can use a segmentation method determined based on preset rules. For example, after acquiring the panoramic image to be processed, the front-end can perform feature analysis on the panoramic image to determine the segmentation method, which better matches the characteristics of the panoramic image, thus improving image processing efficiency. Alternatively, the front-end can also provide a user selection interface, allowing the user to select or input the desired segmentation method, thus making the segmentation method more consistent with the dynamic range of the panoramic image.

[0207] It should be understood that, in addition to the three methods mentioned above for determining the division method, other methods may also be used to determine the division method, and this application does not specifically limit them.

[0208] The first segmented region is one of multiple segmented regions obtained after the front-end divides the panoramic video into a 3D spherical panoramic image. Based on the mapping relationship between the panoramic image and the 3D spherical panoramic image, multiple pixels in the panoramic image can be mapped to the 3D spherical panoramic image, resulting in multiple mapped points, with a one-to-one correspondence between the pixels and the mapped points. Therefore, after determining the range of the first segmented region, at least one mapped point within it can be obtained.

[0209] Step 1202: Generate metadata information units for the first segmented region based on at least one pixel.

[0210] Typically, panoramic video data includes video data and metadata. Video data can be related data from multiple image frames within the panoramic video, such as pixel data for each frame. Metadata can include data related to the video data and data related to the mapping curve. Video data-related data may include, for example, the format and characteristics of the panoramic video data, the target system display maximum luminance (targeted_system_display_maximum_luminance), the maximum luminance (MaxSource), minimum luminance (MinSource), and average luminance (AvgSource) that can be stored in memory, and the storable range of variation, etc. The principle of the storable range of variation is similar to variance or distributed confidence intervals, used to describe the luminance concentration range of the video signal. Mapping curve-related data may include parameters related to the mapping curve, such as shrinkage factor, offset factor, and curvature factor. For example, data related to an S-curve includes eight parameters: a, b, p, n, m, k1, k2, and k3. It should be noted that panoramic video data includes, but is not limited to, the above content; this application does not specifically limit it.

[0211] For example, let's take the i-th segmented region as an example.

[0212] 1. Generate the "maximum brightness value" in the metadata unit:

[0213] a. Calculate the brightness value Y[i] of the mapping point contained in the i-th segmented region (optionally, the maximum value of the RGB components of the mapping point can also be calculated here), i∈[0,n-1], where n represents the number of mapping points contained in the i-th segmented region. It should be noted that the brightness value of the mapping point is actually the brightness value of the pixel in the panoramic image corresponding to that mapping point, and the same applies below.

[0214] b. Obtain the maximum brightness value MaxY = max(Y[i]) among the brightness values ​​of the mapping points contained in the i-th segmented region.

[0215] c. Adjust MaxY according to the preset style or parameters to obtain MaxY1.

[0216] d. Write MaxY1 as the "curve information value" into the metadata information unit of the i-th segmented region.

[0217] 2. Generate the "curve information value" in the metadata information unit.

[0218] a. Calculate the histogram hiss[i] of all mapping points contained in the i-th segmented region, where i∈[0,n-1], and n represents the number of mapping points contained in the i-th segmented region. It should be noted that the histogram of all mapping points is actually the histogram of multiple pixels in the panoramic image corresponding to all mapping points, and the same applies below.

[0219] b. Process the histogram his[i] (including no operation or truncation after exceeding a certain size) to generate his1[i].

[0220] c. Generate a mapping curve based on the processed histogram his1[i], Ratio[i] = his1[i] / Sum of(his1[i]).

[0221] d. Adjust Ratio[i] according to the preset style or algorithm parameters to obtain Ratio1[i] = w[i] × Ratio[i].

[0222] e. Write Ratio1[n] as the "curve information value" into the metadata information unit of the i-th segmented region.

[0223] In one possible implementation, when the histograms and / or brightness of the first segmented region and the second segmented region meet the set conditions, the front end can fuse the metadata information units of the first segmented region and the second metadata information units of the second segmented region to obtain the metadata information units of the first segmented region and the second segmented region, wherein the second segmented region is one of multiple segmented regions.

[0224] It should be understood that the above examples describe two methods for generating the content in metadata information units, but do not constitute a limitation on the generation method or the content contained in the metadata information units, and this application does not make any specific limitations in this regard.

[0225] The front end acquires histograms and / or brightness values ​​of multiple segmented regions. If the histograms and / or brightness values ​​of two or more segmented regions are similar—for example, if the distortion values ​​of the histograms and / or brightness values ​​of these two or more segmented regions are less than a set threshold—the dynamic range of the image regions corresponding to these two or more segmented regions can be considered to be highly similar. Therefore, the same metadata information unit can be determined for these two or more segmented regions. This metadata information unit can be obtained by weighted averaging of the metadata information units of the two or more segmented regions obtained in the above steps, or by recalculating the metadata information unit for the two or more segmented regions obtained in the above steps as a single segmented region.

[0226] For example, the following method can be used to determine whether the distortion value of the histogram of two or more segmented regions is less than a set threshold:

[0227] 1. Calculate the histogram hisA[n] of all mapping points contained in the first segmentation region.

[0228] 2. Calculate the histogram hisB[n] of all mapping points contained in the second segmentation region.

[0229] 3. Calculate hisDiff[i]=(hisA[i]-hisB[i])×(hisA[i]-hisB[i]).

[0230] 4. Calculate the total distortion value hisDiffSum based on the preset weights: hisDiffSum = ∑hisDiff[i] × w[i]

[0231] 5. If hisDiffSum is less than the preset threshold T, then it is considered that "the histograms of the first segmentation region and the second segmentation region meet the set conditions".

[0232] At this point, the front end can add indication information to the metadata information unit to indicate which segmentation regions the metadata information unit corresponds to. For example, multiple spherical coordinates can be written into the metadata information unit, and these multiple spherical coordinates are located in the aforementioned two or more segmentation regions. In this way, when the display terminal reads the metadata information unit, it can determine the segmentation region to which the spherical coordinates belong based on the spherical coordinates included therein, and thus determine it as the segmentation region corresponding to the metadata information unit.

[0233] It can be seen that when the front end generates metadata information units for each segmented region, it fully considers the characteristics of the pixels corresponding to the mapping points contained in the segmented region. The characteristics of the pixels can reflect the dynamic range of the segmented region. Therefore, when the display end obtains the tone mapping curve of the corresponding segmented region based on the metadata information unit, it obtains a tone mapping curve that matches the dynamic range characteristics of the segmented region, thereby achieving more accurate tone mapping processing for panoramic images.

[0234] Step 1203: Write the metadata information unit of the first segmented region into the bitstream.

[0235] After the front-end determines the first segmented region and its corresponding metadata information unit, it can write the metadata information unit into the bitstream so that the display end can perform corresponding tone mapping when displaying the panoramic image. It should be understood that the metadata units for other segmented regions besides the first segmented region can be determined using the above method, which will not be elaborated here.

[0236] Optionally, the front end can traverse multiple segmented regions in a set order, and then write the corresponding metadata information unit into the bitstream for each segmented region traversed.

[0237] Optionally, the front-end can determine the order of traversing multiple segmented regions according to preset rules, and then write the corresponding metadata information unit into the bitstream for each segmented region traversed according to the order. At the same time, the front-end also needs to write the determined order into the bitstream.

[0238] The above order can be a top-down or front-back order, representing the panoramic image from the three-dimensional spherical surface of the panoramic video.

[0239] In one possible implementation, the front end can perform the processes of generating metadata information units and writing the bitstream alternately. That is, the front end traverses multiple segmented regions in the above order. For each segmented region, the corresponding metadata information unit is generated using the method in step 1202, and then the metadata information unit is written into the bitstream.

[0240] This application divides the panoramic video three-dimensional spherical representation of the panoramic image into multiple segmented regions. Based on the pixels corresponding to the mapping points contained in each segmented region, a metadata information unit is generated for that segmented region. The metadata information unit guides the display end to generate a tone mapping curve corresponding to that segmented region, thereby realizing tone mapping from image to display. This not only allows for the determination of matching tone mapping parameters for different dynamic range regions on the panoramic image, but also conforms to the display dynamic range of the display end, improving the processing efficiency of panoramic images.

[0241] Figure 13 This is a flowchart of process 1300 for the tone mapping method of panoramic images in this application. Process 1300 can be performed by... Figure 8 The process 1300 is executed at the display end of the panoramic video playback system shown. Process 1300 is described as a series of steps or operations; it should be understood that process 1300 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 13 The execution order is shown. Process 1300 may include:

[0242] Step 1301: Determine one or more target metadata information units of the first pixel from multiple metadata information units.

[0243] In this application, the display end can use the above method to divide the panoramic image of the three-dimensional spherical representation of the panoramic video into multiple segmented regions. Regarding the specific segmentation method used, the display end can use a preset segmentation method or a segmentation method obtained by parsing the bitstream.

[0244] It should be understood that, in addition to the two methods mentioned above for determining the division method, other methods may also be used to determine the division method, and this application does not make specific limitations on this.

[0245] The display device can also parse the bitstream to obtain multiple metadata information units. Additionally, the display device can parse the numbering information of these multiple metadata information units from the bitstream. This numbering information can identify the multiple metadata information units and also indicate their sequential order; alternatively, the display device can determine the sequential order of the parsed metadata information units as the overall order of the metadata information units.

[0246] The display can determine the correspondence between multiple segmented regions and multiple metadata information units using the following methods:

[0247] 1. The display terminal can extract the current metadata information unit from multiple metadata information units in a first preset order, extract the current segmented region from multiple segmented regions in a second preset order, and establish a correspondence between the current segmented region and the current metadata information unit.

[0248] 2. The display end can extract the current metadata information unit from multiple metadata information units in a first preset order, extract the current segmented region from multiple segmented regions in a traversal order obtained from parsing the bitstream, and establish a correspondence between the current segmented region and the current metadata information unit.

[0249] 3. The display terminal can extract the current metadata information unit from multiple metadata information units according to a first preset order, obtain one or more coordinates included in the current metadata information unit, and determine one or more mapping points on the panoramic video 3D spherical panoramic image based on the one or more coordinates. When there is only one mapping point, a correspondence is established between the segmented region to which the mapping point belongs and the current metadata information unit; when there are multiple mapping points, a correspondence is established between at least one segmented region to which the multiple mapping points belong and the current metadata information unit.

[0250] The current metadata information unit can be considered as the first one among multiple metadata information units. As mentioned above, there is a sequential order among multiple metadata information units. The first preset order can refer to extracting the first metadata information unit in the sequential order of multiple metadata information units, which is the current metadata information unit. Once the correspondence between the current metadata information unit and one or more of the segmentation units is determined, it is removed from the first order, and the next metadata information unit is determined as the new current metadata information unit. This process is repeated multiple times until the correspondence between multiple segmentation regions and multiple metadata information units is determined.

[0251] In this application, the display terminal can determine one or more target segmentation regions based on the set mapping point. When there is only one target segmentation region, the metadata information unit corresponding to the target segmentation region is determined as one target metadata information unit; or, when there are multiple target segmentation regions, the metadata information units corresponding to the multiple target segmentation regions are determined as multiple target metadata information units.

[0252] In one possible implementation, a mapping point is set as the first mapping point, and a first coverage area is determined on the panoramic image of the three-dimensional spherical representation of the panoramic video with the first mapping point as the center. When the first coverage area includes a segmented region, the segmented region is determined as a target segmented region; or, when the first coverage area includes multiple segmented regions, the multiple segmented regions are determined as multiple target segmented regions.

[0253] The mapping point is defined as the first mapping point of the first pixel on the panoramic image expressed by the three-dimensional sphere of the panoramic video. The display end can determine the first coverage area with the first mapping point as the center and the first length as the radius. In this case, the first coverage area is a circular area. Alternatively, the display end can also determine the first coverage area with the first mapping point as the center and the first length as the side length. In this case, the first coverage area is a square area. The first length can be a preset length, or it can be specified in the first metadata information unit corresponding to the first segmented region to which the first mapping point belongs. Multiple segmented regions include the first segmented region, and multiple metadata information units include the first metadata information unit. It should be understood that the first coverage area can also be a region of other shapes. The relevant information about its shape and size can be preset or specified in the aforementioned first metadata information unit. This application does not specifically limit this.

[0254] For example, a first coverage area centered on a first mapping point and defined by a first length as the radius / side length might only cover the segmented region to which the first mapping point belongs; in this case, the segmented region to which the first mapping point belongs is a single target segmented region. Alternatively, the first coverage area might cover the segmented region to which the first mapping point belongs and one or more adjacent segmented regions; in this case, the segmented region to which the first mapping point belongs and one or more adjacent segmented regions constitute multiple target segmented regions. Furthermore, the first coverage area might only cover a portion of the aforementioned adjacent segmented regions; in this case, those adjacent segmented regions can also be considered target segmented regions.

[0255] In one possible implementation, the mapping point is set as the viewpoint center of the panoramic image expressed by the three-dimensional spherical surface of the panoramic video. The second segmentation region to which the viewpoint center belongs is determined. Multiple segmentation regions include the second segmentation region, and the second segmentation region is determined as a target segmentation region.

[0256] The mapping point is set as the viewpoint center on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video. Regardless of the position of the first pixel in the panoramic image, the second segmentation region to which it belongs can be determined as a target segmentation region by referring to the viewpoint center on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video.

[0257] In one possible implementation, the mapping point is set as the viewpoint center of the panoramic image expressed by the three-dimensional spherical surface of the panoramic video. A second coverage area is determined on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video with the viewpoint center as the center. When the second coverage area includes a segmented region, the segmented region is determined as a target segmented region; or, when the second coverage area includes multiple segmented regions, the multiple segmented regions are determined as multiple target segmented regions.

[0258] The mapping point is set as the center of the viewpoint on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video. The display end can determine the second coverage area with the visual center as the center and the second length as the radius / side length. In this case, the second coverage area is a circular area. Alternatively, the display end can also determine the second coverage area with the visual center as the center and the second length as the side length. In this case, the second coverage area is a square area. The second length can be a preset length, or it can be specified in the second metadata information unit corresponding to the second segmented region to which the viewpoint center belongs. Multiple segmented regions include the second segmented region, and multiple metadata information units include the second metadata information unit. It should be understood that the second coverage area can also be a region of other shapes. The relevant information about its shape and size can be preset or specified in the aforementioned second metadata information unit. This application does not specifically limit this.

[0259] For example, unlike determining the first coverage area with the first mapping point as the center and the first length as the radius, this time the second coverage area is determined with the visual center as the center and the second length as the radius. The segmented region covered by the second coverage area is then determined as the target segmented region. Similarly, the second coverage area may only cover the segmented region belonging to the visual center, in which case the segmented region belonging to the visual center is one target segmented region. Alternatively, the second coverage area may cover the segmented region belonging to the visual center and one or more adjacent segmented regions, in which case the segmented region belonging to the visual center and one or more adjacent segmented regions constitute multiple target segmented regions. The second coverage area may only cover a portion of the aforementioned adjacent segmented regions, in which case those adjacent segmented regions can also be considered target segmented regions.

[0260] Based on the correspondence between multiple segmented regions and multiple metadata information units, the metadata information units corresponding to one or more target segmented regions determined in the above steps can be identified as one or more target metadata information units of the first pixel.

[0261] Step 1302: Perform tone mapping on the pixel value of the first pixel based on one or more target metadata information units to obtain the target tone mapping value of the first pixel.

[0262] In this application, the display end can obtain one or more tone mapping curves based on one or more target metadata information units. When there is only one tone mapping curve, the pixel value of the first pixel is tone mapped according to the tone mapping curve to obtain the target tone mapping value; or, when there are multiple tone mapping curves, the pixel value of the first pixel is tone mapped according to the multiple tone mapping curves respectively to obtain multiple intermediate tone values ​​of the first pixel, and then the target tone mapping value is obtained based on the multiple intermediate tone values.

[0263] As mentioned above, the metadata information unit includes relevant parameters of the tone mapping curve. Therefore, the corresponding tone mapping curve can be obtained based on these parameters. It should be noted that this application does not impose specific limitations on the parameters of the tone mapping curve or the tone mapping curve itself.

[0264] Based on this, if the first pixel has only one target metadata information unit, then a tone mapping curve can be obtained for the first pixel; if the first pixel has multiple target metadata information units, then multiple tone mapping curves can be obtained for the first pixel.

[0265] Correspondingly,

[0266] By using the aforementioned tone mapping curve to tone map the pixel value of the first pixel, the target tone mapping value of the first pixel can be directly obtained.

[0267] By applying the aforementioned multiple tone mapping curves to the pixel values ​​of the first pixel, multiple intermediate tone values ​​for the first pixel can be obtained. These intermediate tone values ​​can then be further processed. For example, the median value among the multiple intermediate tone values ​​can be taken (e.g., if the multiple intermediate tone values ​​include 1, 2, and 3, then the median value is 2), or a weighted average of the multiple intermediate tone values ​​can be applied to obtain the target intermediate tone value for the first pixel. The weighting coefficients corresponding to each intermediate tone value in the aforementioned weighted average can be related to the distance between the first mapping point and the center point of each segmented region. That is, the distance (e.g., Euclidean distance) between the first mapping point and the center point of each segmented region in the multiple target segmented regions is calculated, the sum of the distances corresponding to each of the multiple target segmented regions is calculated, and the ratio of each distance to the sum of the distances is used as the metadata information unit of the corresponding target segmented region to obtain the weighting coefficient of the intermediate tone value; or, the weighting coefficients corresponding to each intermediate tone value in the aforementioned weighted average can be related to the area proportion of the segmented region corresponding to each of the multiple target metadata information units. That is, calculate the area of ​​multiple segmented regions contained in the first or second coverage area. If a segmented region is completely covered by the first or second coverage area, calculate the full area of ​​the segmented region. If a segmented region is partially covered by the first or second coverage area, calculate the area of ​​the partial area of ​​the segmented region. Calculate the sum of the areas of multiple target segmented regions. Use the ratio of the area of ​​each segmented region to the sum of the areas as the metadata information unit of the corresponding target segmented region to obtain the weighting coefficient of the hue median value.

[0268] For example, the tone mapping curve can be obtained by retrieving the tone mapping curve parameters a, b, p, m, n from the target metadata information unit of the first pixel:

[0269]

[0270] The mapping relationship between the normalized pixel value of the first pixel and the normalized HDR display data can be obtained from the tone mapping curve described above. It should be noted that L and L' can be normalized optical or electrical signals, and this application does not impose specific limitations. The normalization calculation can be a non-linear space of PQ or a linear space of 0 to 1. The normalization can be 0-10000 nits or 0.001-100000 nits. This application does not impose specific limitations on the normalization range and process of the data.

[0271] Based on the maximum and minimum display capabilities (typically 0) of the HDR display device, the normalized HDR display data L' is inversely normalized to a range between the maximum and minimum display capabilities of the display device. The target hue value of the first pixel is the inversely normalized HDR display data. It should be noted that the inverse normalization calculation can be in a non-linear space of PQ or in a linear space normalized to 0-1. The inverse normalization range can be 0-10000 nits or 0.001-100000 nits. This application does not specifically limit the range and process of the inverse normalization of the data.

[0272] This application divides the panoramic video three-dimensional spherical representation of the panoramic image into multiple segmented regions. Based on the pixels corresponding to the mapping points contained in each segmented region, metadata information units are generated for that segmented region. These metadata information units then guide the display end to generate a tone mapping curve corresponding to that segmented region, thereby achieving tone mapping from image to display. This approach can not only determine matching tone mapping parameters for different dynamic range regions on the panoramic image but also conform to the display dynamic range of the display end, thus improving the processing efficiency of panoramic images.

[0273] In one possible implementation, the display terminal can parse the bitstream to obtain multiple metadata information units. In addition, the display terminal can parse the numbering information of the multiple metadata information units from the bitstream. This numbering information can identify the multiple metadata information units on the one hand, and indicate the order of the multiple metadata information units on the other hand. Alternatively, the display terminal can parse the bitstream to obtain multiple metadata information units, and determine the order in which the metadata information units are parsed as the order of the multiple metadata information units.

[0274] In this application, the display end can divide the panoramic video into three-dimensional spherical panoramic images to obtain multiple segmented regions.

[0275] In one possible implementation, the display end can obtain a preset division method, and then divide the panoramic video 3D spherical representation panoramic image according to the preset division method to obtain multiple segmented regions.

[0276] Users can pre-set the segmentation method based on historical or laboratory data, which is applicable to most panoramic images and improves image processing efficiency. Alternatively, users can select or input their desired segmentation method through the user interface provided by the panoramic image processing system, which better matches the dynamic range of the panoramic image.

[0277] In one possible implementation, the display end can parse the bitstream to obtain the segmentation method set by the front end.

[0278] After acquiring the panoramic image to be processed, the front-end can perform feature analysis to determine a segmentation method that better matches the characteristics of the panoramic image, thereby improving image processing efficiency. To maintain consistency with the display end, the front-end can write the determined segmentation method into the bitstream. The display end can then parse the bitstream to obtain the aforementioned segmentation method.

[0279] Using the above method, the display device can determine that the division method is based on viewing angle rules. The display device can divide the panoramic video's 3D spherical representation into multiple segmented regions with equal viewing angle intervals, for example... Figure 11a As shown, in the multiple segmented regions, the horizontal angles between two adjacent latitude segmented regions are equal, and the pitch angles between two adjacent longitude segmented regions are equal; alternatively, the display end can divide the panoramic video into multiple segmented regions along the latitude direction using a 3D spherical representation, for example, as... Figure 11b As shown, the display end divides the panoramic video 3D spherical representation into two equal-sized segments; or, it divides the panoramic video 3D spherical representation into multiple segments along the longitude direction, for example, as... Figure 11c As shown, the display end divides the panoramic video 3D spherical representation into two equally sized segments; alternatively, the display end can divide the panoramic video 3D spherical representation into multiple segments along the latitude direction and also evenly divide it into multiple segments along the longitude direction, for example, as... Figure 11d As shown, the display end divides the panoramic video into four equal-sized vertical and horizontal spherical regions, representing the panoramic image.

[0280] It should be understood that the above examples of division methods do not constitute a limitation on the division method. The display end can also divide the panoramic video 3D spherical representation of the panoramic image into two equal parts, and then divide each part into four equal parts; or the display end can divide the panoramic video 3D spherical representation of the panoramic image into two equal parts, and then divide each part into four equal parts, and finally divide each quarter region into regions with equal viewing angle intervals. In addition, the division of the panoramic video 3D spherical representation of the panoramic image by the display end does not have to be equal, or not all division steps are equal divisions, and there are no specific limitations on this.

[0281] In this application, the display end can also directly obtain the multiple segmented regions included in the panoramic image of the three-dimensional spherical representation of the panoramic video by parsing the bitstream. After the front end divides the panoramic image of the three-dimensional spherical representation of the panoramic video into multiple segmented regions, it can write the division information of these multiple segmented regions into the bitstream, such as the shape, size, and coordinates of the upper left corner of the segmented region, or the description information of the dividing lines. In this way, the display end can parse the bitstream to obtain the division information of the above-mentioned multiple segmented regions, and thus determine the multiple segmented regions.

[0282] Figure 14 This is an exemplary structural diagram of the display device 1400 of this application, as shown below. Figure 14 As shown, the display device 1400 of this embodiment can be applied to the display terminal in the above embodiments, and also to the terminal device in the above embodiments. The display device 1400 may include: a partitioning module 1401, a determining module 1402, and a mapping module 1403.

[0283] The determining module 1402 is used to determine one or more target metadata information units of a first pixel from a plurality of metadata information units, wherein the plurality of metadata information units are obtained by parsing the bitstream, and the first pixel is any pixel in the two-dimensional planar projection map of the panoramic video to be processed. The plurality of metadata information units correspond to a plurality of segmented regions included in the three-dimensional spherical panoramic image of the panoramic video, and the two-dimensional planar projection map of the panoramic video and the three-dimensional spherical panoramic image of the panoramic video have a mapping relationship. The mapping module 1403 is used to perform tone mapping on the pixel value of the first pixel according to the one or more target metadata information units to obtain the target tone mapping value of the first pixel.

[0284] In one possible implementation, the segmentation module 1401 is used to obtain the plurality of segmented regions.

[0285] In one possible implementation, the segmentation module 1401 is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a preset segmentation method to obtain the multiple segmented regions.

[0286] In one possible implementation, the segmentation module 1401 is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to the segmentation method obtained by parsing the bitstream, so as to obtain the multiple segmented regions.

[0287] In one possible implementation, the multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or...

[0288] The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

[0289] In one possible implementation, the segmentation module 1401 is specifically used to cluster multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; map the multiple pixels to the three-dimensional spherical panoramic image of the panoramic video respectively; and take the region formed by the mapping points of the pixels included in the first pixel set on the three-dimensional spherical panoramic image of the panoramic video as the first segmentation region, wherein the first pixel set is one of the multiple pixel sets, and the first segmentation region is one of the multiple segmentation regions.

[0290] In one possible implementation, the segmentation module 1401 is specifically used to map multiple pixels included in the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic representation map of the panoramic video to obtain multiple mapping points; cluster the multiple mapping points to obtain multiple mapping point sets; and take the region formed by the mapping points included in the first mapping point set as the second segmentation region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

[0291] In one possible implementation, the segmentation module 1401 is specifically used to obtain the multiple segmented regions based on the indication information of the multiple segmented regions obtained by parsing the bitstream.

[0292] In one possible implementation, the determining module 1402 is specifically used to determine the correspondence between the plurality of metadata information units and the plurality of segmented regions, wherein one metadata information unit corresponds to one or more segmented regions; determine one or more target segmented regions according to a set mapping point; when there is only one target segmented region, determine the metadata information unit corresponding to the one target segmented region as the one target metadata information unit; or, when there are multiple target segmented regions, determine the metadata information units corresponding to the multiple target segmented regions as the multiple target metadata information units.

[0293] In one possible implementation, the determining module 1402 is specifically used to extract the current metadata information unit from the plurality of metadata information units in a first preset order; extract the current segmentation region from the plurality of segmentation regions in a second preset order; and establish a correspondence between the current segmentation region and the current metadata information unit.

[0294] In one possible implementation, the determining module 1402 is specifically used to extract the current metadata information unit from the plurality of metadata information units according to a first preset order; extract the current segmented region from the plurality of segmented regions according to the traversal order obtained by parsing the bitstream; and establish a correspondence between the current segmented region and the current metadata information unit.

[0295] In one possible implementation, the determining module 1402 is specifically configured to extract the current metadata information unit from the plurality of metadata information units according to a first preset order; obtain one or more coordinates included in the current metadata information unit; determine one or more mapping points on the panoramic video three-dimensional spherical panoramic image based on the one or more coordinates; when there is only one mapping point, establish a correspondence between the segmented region to which the mapping point belongs and the current metadata information unit; when there are multiple mapping points, establish a correspondence between at least one segmented region to which the multiple mapping points belong and the current metadata information unit.

[0296] In one possible implementation, the set mapping point is the first mapping point of the first pixel on the panoramic image of the three-dimensional spherical representation of the panoramic video; the determining module 1402 is specifically used to determine a first coverage area on the panoramic image of the three-dimensional spherical representation of the panoramic video with the first mapping point as the center; when the first coverage area includes a segmented region, the segmented region is determined as the target segmented region; or, when the first coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

[0297] In one possible implementation, the set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical representation of the panoramic video; the determining module 1402 is specifically used to determine the second segmentation region to which the viewpoint center belongs, the plurality of segmentation regions including the second segmentation region; and to determine the second segmentation region as the target segmentation region.

[0298] In one possible implementation, the set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video; the determining module 1402 is specifically used to determine a second coverage area on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video with the viewpoint center as the center; when the second coverage area includes a segmented region, the segmented region is determined as the target segmented region; or, when the second coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

[0299] In one possible implementation, the mapping module 1403 is specifically configured to obtain one or more tone mapping curves based on the one or more target metadata information units; when there is only one tone mapping curve, the pixel value of the first pixel is tone mapped according to the one tone mapping curve to obtain the target tone mapping value; or, when there are multiple tone mapping curves, the pixel value of the first pixel is tone mapped according to the multiple tone mapping curves respectively to obtain multiple intermediate tone values ​​of the first pixel; and the target tone mapping value is obtained based on the multiple intermediate tone values.

[0300] In one possible implementation, the mapping module 1403 is specifically used to determine the median value among the plurality of hue median values ​​as the target hue mapping value; or, to obtain the target hue mapping value by performing a weighted average of the plurality of hue median values, wherein the weighting coefficients of the plurality of hue median values ​​are preset, or the weighting coefficients of the plurality of hue median values ​​are related to the distance between the first pixel and the center point of each segmented region, or the weighting coefficients of the plurality of hue median values ​​are related to the area proportion of the segmented regions corresponding to the plurality of target metadata information units respectively.

[0301] The apparatus of this embodiment can be used to perform Figure 13 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0302] Figure 15 This is an exemplary structural diagram of the front-end device 1500 of this application, such as... Figure 15 As shown, the front-end device 1500 of this embodiment can be applied to the front-end in the above embodiments, and also to the terminal device in the above embodiments. The front-end device 1500 may include: a segmentation module 1501, an acquisition module 1502, and a generation module 1503. Wherein,

[0303] The acquisition module 1502 is used to acquire at least one mapping point included in the first segmented region, wherein the first segmented region is one of multiple segmented regions included in the panoramic video three-dimensional spherical expression panoramic image, the panoramic video three-dimensional spherical expression panoramic image and the panoramic video two-dimensional planar projection image to be processed have a mapping relationship, and the at least one mapping point corresponds to at least one pixel on the panoramic video two-dimensional planar projection image; the generation module 1503 is used to generate metadata information units of the first segmented region based on the at least one pixel; and write the metadata information units of the first segmented region into the bitstream.

[0304] In one possible implementation, the generation module 1503 is further configured to fuse the metadata information unit of the first segmented region and the second metadata information unit of the second segmented region to obtain the metadata information unit of the first segmented region and the second segmented region when the histogram and / or brightness of the first segmented region and the second segmented region meet the set conditions, wherein the second segmented region is one of the plurality of segmented regions.

[0305] In one possible implementation, the segmentation module 1501 is used to map the two-dimensional planar projection image of the panoramic video to the three-dimensional spherical panoramic image of the panoramic video; and to segment the three-dimensional spherical panoramic image of the panoramic video to obtain the plurality of segmented regions.

[0306] In one possible implementation, the segmentation module 1501 is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a preset segmentation method to obtain the multiple segmented regions.

[0307] In one possible implementation, the segmentation module 1501 is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a segmentation method determined based on preset rules to obtain the multiple segmented regions.

[0308] In one possible implementation, the multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or...

[0309] The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

[0310] In one possible implementation, the segmentation module 1501 is specifically used to cluster multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; map the multiple pixels to the three-dimensional spherical panoramic image of the panoramic video respectively; and take the region formed by the mapping points of the pixels included in the first pixel set on the three-dimensional spherical panoramic image of the panoramic video as the first segmentation region, wherein the first pixel set is one of the multiple pixel sets, and the first segmentation region is one of the multiple segmentation regions.

[0311] In one possible implementation, the segmentation module 1501 is specifically used to map multiple pixels included in the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic representation map of the panoramic video to obtain multiple mapping points; cluster the multiple mapping points to obtain multiple mapping point sets; and take the region formed by the mapping points included in the first mapping point set as the second segmentation region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

[0312] The apparatus of this embodiment can be used to perform Figure 12 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0313] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or implemented by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0314] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0315] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0316] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0317] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0318] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0319] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0320] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0321] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A tone mapping method for panoramic images, characterized in that, include: One or more target metadata information units for a first pixel are determined from multiple metadata information units, wherein the multiple metadata information units are obtained by parsing the bitstream, and the first pixel is any pixel in the two-dimensional planar projection map of the panoramic video to be processed. The multiple metadata information units correspond to multiple segmented regions included in the three-dimensional spherical panoramic image of the panoramic video, and the two-dimensional planar projection map of the panoramic video and the three-dimensional spherical panoramic image of the panoramic video have a mapping relationship. The pixel value of the first pixel is tone-mapped according to the one or more target metadata information units to obtain the target tone-mapped value of the first pixel. Wherein, the step of performing tone mapping on the pixel value of the first pixel based on the one or more target metadata information units to obtain the target tone mapping value of the first pixel includes: One or more tone mapping curves are obtained based on the one or more target metadata information units; When there is only one color bar mapping curve, the pixel value of the first pixel is tinted according to the tint mapping curve to obtain the target tint mapping value; Alternatively, when there are multiple color bar mapping curves, the pixel value of the first pixel is tinted according to the multiple tint mapping curves to obtain multiple tint intermediate values ​​of the first pixel. The target hue mapping value is obtained based on the multiple hue intermediate values.

2. The method according to claim 1, characterized in that, Before determining one or more target metadata information units of the first pixel from multiple metadata information units, the method further includes: Obtain the multiple segmented regions.

3. The method according to claim 2, characterized in that, The process of obtaining the plurality of segmented regions includes: The panoramic video 3D spherical representation panoramic image is divided according to a preset division method to obtain the multiple segmented regions.

4. The method according to claim 2, characterized in that, The process of obtaining the plurality of segmented regions includes: The panoramic video 3D spherical representation panoramic image is divided according to the division method obtained by parsing the bitstream to obtain the multiple segmented regions.

5. The method according to claim 3 or 4, characterized in that, The multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or... The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

6. The method according to claim 3 or 4, characterized in that, The process of dividing the panoramic video into three-dimensional spherical panoramic images to obtain the multiple segmented regions includes: Cluster the multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; The plurality of pixels are respectively mapped to the three-dimensional spherical surface of the panoramic video to represent the panoramic image; The region formed by the mapping points of the pixels included in the first pixel set on the panoramic image of the three-dimensional spherical representation of the panoramic video is taken as the first segmentation region. The first pixel set is one of the plurality of pixel sets, and the first segmentation region is one of the plurality of segmentation regions.

7. The method according to claim 3 or 4, characterized in that, The process of dividing the panoramic video into three-dimensional spherical panoramic images to obtain the multiple segmented regions includes: The multiple pixels included in the two-dimensional planar projection map of the panoramic video are respectively mapped to the three-dimensional spherical panoramic image of the panoramic video to obtain multiple mapping points; Cluster the multiple mapping points to obtain multiple sets of mapping points; The region formed by the mapping points included in the first mapping point set is used as the second segmentation region, where the first mapping point set is one of the plurality of mapping point sets, and the second segmentation region is one of the plurality of segmentation regions.

8. The method according to claim 2, characterized in that, The process of obtaining the plurality of segmented regions includes: The multiple segmented regions are obtained based on the indication information of the multiple segmented regions obtained by parsing the bitstream.

9. The method according to any one of claims 1-8, characterized in that, The step of determining one or more target metadata information units of the first pixel from multiple metadata information units includes: Determine the correspondence between the plurality of metadata information units and the plurality of segmented regions, wherein one metadata information unit corresponds to one or more of the segmented regions; One or more target segmentation regions are determined based on the set mapping points; When there is only one target segmentation region, the metadata information unit corresponding to the target segmentation region is determined as the target metadata information unit; Alternatively, when there are multiple target segmentation regions, the metadata information units corresponding to the multiple target segmentation regions are determined as the multiple target metadata information units.

10. The method according to claim 9, characterized in that, Determining the correspondence between the plurality of metadata information units and the plurality of segmented regions includes: The current metadata information unit is extracted from the plurality of metadata information units in a first preset order; The current segmented region is extracted from the plurality of segmented regions according to a second preset order; Establish a correspondence between the current segmented region and the current metadata information unit.

11. The method according to claim 9, characterized in that, Determining the correspondence between the plurality of metadata information units and the plurality of segmented regions includes: The current metadata information unit is extracted from the plurality of metadata information units in a first preset order; The current segmented region is extracted from the multiple segmented regions according to the traversal order obtained by parsing the bitstream; Establish a correspondence between the current segmented region and the current metadata information unit.

12. The method according to claim 9, characterized in that, Determining the correspondence between the plurality of metadata information units and the plurality of segmented regions includes: The current metadata information unit is extracted from the plurality of metadata information units in a first preset order; Obtain one or more coordinates included in the current metadata information unit; Based on the one or more coordinates, determine one or more mapping points on the panoramic view of the three-dimensional spherical representation of the panoramic video; When there is only one mapping point, a correspondence is established between the segmented region to which the mapping point belongs and the current metadata information unit; When there are multiple mapping points, a correspondence is established between at least one segmented region to which the multiple mapping points belong and the current metadata information unit.

13. The method according to any one of claims 9-12, characterized in that, The set mapping point is the first mapping point of the first pixel on the panoramic image of the three-dimensional spherical representation of the panoramic video. The step of determining one or more target segmentation regions based on set mapping points includes: A first coverage area is determined on the panoramic image of the three-dimensional spherical representation of the panoramic video, with the first mapping point as the center. When the first coverage area includes a segmented region, the segmented region is determined as the target segmented region; Alternatively, when the first coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

14. The method according to any one of claims 9-12, characterized in that, The set mapping point is the center of viewpoint on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video; the step of determining one or more target segmentation regions based on the set mapping point includes: Determine the second segmented region to which the viewpoint center belongs, wherein the plurality of segmented regions include the second segmented region; The second segmented region is determined as the target segmented region.

15. The method according to any one of claims 9-12, characterized in that, The set mapping point is the center of viewpoint on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video; the step of determining one or more target segmentation regions based on the set mapping point includes: A second coverage area is determined on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video, with the visual center as the center. When the second coverage area includes a segmented region, the segmented region is determined as the target segmented region; Alternatively, when the second coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

16. The method according to claim 1, characterized in that, The step of obtaining the target hue mapping value based on the plurality of hue intermediate values ​​includes: The median value among the plurality of hue median values ​​is determined as the target hue mapping value; Alternatively, the target tone mapping value can be obtained by weighted averaging of the multiple intermediate tone values, where the weighting coefficients of the multiple intermediate tone values ​​are preset, or the weighting coefficients of the multiple intermediate tone values ​​are related to the distance between the first pixel and the center point of each segmented region, or the weighting coefficients of the multiple intermediate tone values ​​are related to the area proportion of the segmented regions corresponding to the multiple target metadata information units.

17. A tone mapping method for panoramic images, characterized in that, include: Obtain at least one mapping point included in the first segmentation region, wherein the first segmentation region is one of multiple segmentation regions included in the panoramic image of the three-dimensional spherical representation of the panoramic video, the panoramic image of the three-dimensional spherical representation of the panoramic video and the panoramic image of the two-dimensional planar projection of the panoramic video to be processed have a mapping relationship, and the at least one mapping point corresponds to at least one pixel point on the panoramic image of the two-dimensional planar projection of the panoramic video. Based on the at least one pixel, a metadata information unit for the first segmented region is generated. The metadata information unit is used to generate a tone mapping curve corresponding to the first segmented region. The tone mapping curve is used to tone map the pixel value of the at least one pixel to obtain the target tone mapping value of the at least one pixel. Write the metadata information units of the first segmented region into the bitstream.

18. The method according to claim 17, characterized in that, After generating the metadata information unit of the first segmented region based on the at least one pixel, the method further includes: When the histograms and / or brightness of the first segmented region and the second segmented region meet the set conditions, the metadata information units of the first segmented region and the second metadata information units of the second segmented region are fused to obtain the metadata information units of the first segmented region and the second segmented region, wherein the second segmented region is one of the plurality of segmented regions.

19. The method according to claim 17 or 18, characterized in that, Before obtaining at least one mapping point included in the first segmented region, the method further includes: Map the two-dimensional planar projection of the panoramic video onto the three-dimensional spherical representation of the panoramic video; The panoramic video is divided into multiple segmented regions by dividing the three-dimensional spherical panoramic image.

20. The method according to claim 19, characterized in that, The process of dividing the panoramic video into three-dimensional spherical panoramic images to obtain the multiple segmented regions includes: The panoramic video 3D spherical panoramic image is divided according to a preset division method to obtain the multiple segmented regions.

21. The method according to claim 19, characterized in that, The process of dividing the panoramic video into three-dimensional spherical panoramic images to obtain the multiple segmented regions includes: The panoramic video 3D spherical representation panoramic image is divided according to a division method determined based on preset rules to obtain the multiple segmented regions.

22. The method according to claim 20 or 21, characterized in that, The multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or... The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

23. The method according to claim 20 or 21, characterized in that, The process of dividing the panoramic video into three-dimensional spherical panoramic images to obtain the multiple segmented regions includes: Cluster the multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; The plurality of pixels are respectively mapped to the three-dimensional spherical surface of the panoramic video to represent the panoramic image; The region formed by the mapping points of the pixels included in the first pixel set on the panoramic image of the three-dimensional spherical representation of the panoramic video is taken as the first segmentation region. The first pixel set is one of the plurality of pixel sets, and the first segmentation region is one of the plurality of segmentation regions.

24. The method according to claim 20 or 21, characterized in that, The process of dividing the panoramic video into three-dimensional spherical panoramic images to obtain the multiple segmented regions includes: The multiple pixels included in the two-dimensional planar projection map of the panoramic video are respectively mapped to the three-dimensional spherical panoramic image of the panoramic video to obtain multiple mapping points; Cluster the multiple mapping points to obtain multiple sets of mapping points; The region formed by the mapping points included in the first mapping point set is used as the second segmentation region, where the first mapping point set is one of the plurality of mapping point sets, and the second segmentation region is one of the plurality of segmentation regions.

25. A display device, characterized in that, include: The determining module is used to determine one or more target metadata information units of the first pixel from multiple metadata information units. The multiple metadata information units are obtained by parsing the bitstream. The first pixel is any pixel in the two-dimensional planar projection map of the panoramic video to be processed. The multiple metadata information units correspond to multiple segmented regions included in the three-dimensional spherical panoramic image of the panoramic video. The two-dimensional planar projection map of the panoramic video and the three-dimensional spherical panoramic image of the panoramic video have a mapping relationship. The mapping module is used to perform tone mapping on the pixel value of the first pixel according to the one or more target metadata information units to obtain the target tone mapping value of the first pixel. The mapping module is specifically used to obtain one or more tone mapping curves based on the one or more target metadata information units; When there is only one color bar mapping curve, the pixel value of the first pixel is tinted according to the single tint mapping curve to obtain the target tint mapping value; or, when there are multiple color bar mapping curves, the pixel value of the first pixel is tinted according to the multiple tint mapping curves to obtain multiple intermediate tint values ​​of the first pixel; the target tint mapping value is obtained based on the multiple intermediate tint values.

26. The apparatus according to claim 25, characterized in that, Also includes: The segmentation module is used to obtain the multiple segmented regions.

27. The apparatus according to claim 26, characterized in that, The segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a preset segmentation method to obtain the multiple segmented regions.

28. The apparatus according to claim 26, characterized in that, The segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to the segmentation method obtained by parsing the bitstream, so as to obtain the multiple segmented regions.

29. The apparatus according to claim 27 or 28, characterized in that, The multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or... The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

30. The apparatus according to claim 27 or 28, characterized in that, The segmentation module is specifically used to cluster multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; map the multiple pixels to the three-dimensional spherical panoramic image of the panoramic video; and take the region formed by the mapping points of the pixels included in the first pixel set on the three-dimensional spherical panoramic image of the panoramic video as the first segmentation region, wherein the first pixel set is one of the multiple pixel sets, and the first segmentation region is one of the multiple segmentation regions.

31. The apparatus according to claim 27 or 28, characterized in that, The segmentation module is specifically used to map multiple pixels included in the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic representation map of the panoramic video to obtain multiple mapping points; to cluster the multiple mapping points to obtain multiple mapping point sets; and to use the region formed by the mapping points included in the first mapping point set as the second segmentation region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

32. The apparatus according to claim 26, characterized in that, The segmentation module is specifically used to obtain the multiple segmented regions based on the indication information of the multiple segmented regions obtained by parsing the bitstream.

33. The apparatus according to any one of claims 25-32, characterized in that, The determining module is specifically used to determine the correspondence between the plurality of metadata information units and the plurality of segmented regions, wherein one metadata information unit corresponds to one or more of the segmented regions; One or more target segmentation regions are determined based on the set mapping points; When there is only one target segmentation region, the metadata information unit corresponding to the target segmentation region is determined as the target metadata information unit; Alternatively, when there are multiple target segmentation regions, the metadata information units corresponding to the multiple target segmentation regions are determined as the multiple target metadata information units.

34. The apparatus according to claim 33, characterized in that, The determining module is specifically used to extract the current metadata information unit from the plurality of metadata information units according to a first preset order; and to extract the current segmentation region from the plurality of segmentation regions according to a second preset order. Establish a correspondence between the current segmented region and the current metadata information unit.

35. The apparatus according to claim 33, characterized in that, The determining module is specifically used to extract the current metadata information unit from the plurality of metadata information units in a first preset order; and to extract the current segmented region from the plurality of segmented regions in a traversal order obtained by parsing the bitstream. Establish a correspondence between the current segmented region and the current metadata information unit.

36. The apparatus according to claim 33, characterized in that, The determining module is specifically used to extract the current metadata information unit from the plurality of metadata information units according to a first preset order; obtain one or more coordinates included in the current metadata information unit; and determine one or more mapping points on the panoramic video three-dimensional spherical panoramic image based on the one or more coordinates. When there is only one mapping point, a correspondence is established between the segmented region to which the mapping point belongs and the current metadata information unit; when there are multiple mapping points, a correspondence is established between at least one segmented region to which the multiple mapping points belong and the current metadata information unit.

37. The apparatus according to any one of claims 33-36, characterized in that, The set mapping point is the first mapping point of the first pixel on the panoramic image of the three-dimensional spherical representation of the panoramic video; the determining module is specifically used to determine a first coverage area on the panoramic image of the three-dimensional spherical representation of the panoramic video with the first mapping point as the center; when the first coverage area includes a segmented region, the segmented region is determined as the target segmented region; or, when the first coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

38. The apparatus according to any one of claims 33-36, characterized in that, The set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video; the determining module is specifically used to determine the second segmentation region to which the viewpoint center belongs, and the multiple segmentation regions include the second segmentation region; the second segmentation region is determined as the target segmentation region.

39. The apparatus according to any one of claims 33-36, characterized in that, The set mapping point is the viewpoint center on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video; the determining module is specifically used to determine a second coverage area on the panoramic image expressed by the three-dimensional spherical surface of the panoramic video with the visual center as the center; when the second coverage area includes a segmented region, the segmented region is determined as the target segmented region; or, when the second coverage area includes multiple segmented regions, the multiple segmented regions are determined as the multiple target segmented regions.

40. The apparatus according to claim 25, characterized in that, The mapping module is specifically used to determine the median value among the plurality of intermediate hue values ​​as the target hue mapping value; or, to perform a weighted average of the plurality of intermediate hue values ​​to obtain the target hue mapping value, wherein the weighting coefficients of the plurality of intermediate hue values ​​are preset, or the weighting coefficients of the plurality of intermediate hue values ​​are related to the distance between the first pixel and the center point of each segmented region, or the weighting coefficients of the plurality of intermediate hue values ​​are related to the area proportion of the segmented regions corresponding to the plurality of target metadata information units respectively.

41. A front-end device, characterized in that, include: The acquisition module is used to acquire at least one mapping point included in the first segmentation region. The first segmentation region is one of multiple segmentation regions included in the panoramic video three-dimensional spherical expression panoramic image. The panoramic video three-dimensional spherical expression panoramic image and the panoramic video two-dimensional planar projection image to be processed have a mapping relationship. The at least one mapping point corresponds to at least one pixel point on the panoramic video two-dimensional planar projection image. The generation module is used to generate metadata information units of the first segmented region based on the at least one pixel. The metadata information units are used to generate tone mapping curves corresponding to the first segmented region. The tone mapping curves are used to tone map the pixel values ​​of the at least one pixel to obtain the target tone mapping value of the at least one pixel. The metadata information units of the first segmented region are written into the bitstream.

42. The apparatus according to claim 41, characterized in that, The generation module is further configured to, when the histograms and / or brightness of the first segmented region and the second segmented region meet the set conditions, fuse the metadata information unit of the first segmented region and the second metadata information unit of the second segmented region to obtain the metadata information unit of the first segmented region and the second segmented region, wherein the second segmented region is one of the plurality of segmented regions.

43. The apparatus according to claim 41 or 42, characterized in that, Also includes: A segmentation module is used to map the two-dimensional planar projection image of the panoramic video to the three-dimensional spherical panoramic image of the panoramic video. The panoramic video is divided into multiple segmented regions by dividing the three-dimensional spherical panoramic image.

44. The apparatus according to claim 43, characterized in that, The segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a preset segmentation method to obtain the multiple segmented regions.

45. The apparatus according to claim 43, characterized in that, The segmentation module is specifically used to segment the panoramic video three-dimensional spherical panoramic image according to a segmentation method determined based on preset rules to obtain the multiple segmented regions.

46. ​​The apparatus according to claim 44 or 45, characterized in that, The multiple segmented regions are obtained by dividing the panoramic video's three-dimensional spherical representation of the panoramic image according to a preset viewpoint interval rule; or... The multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the latitude direction; and / or, the multiple segmented regions are obtained by dividing the panoramic image of the three-dimensional spherical representation of the panoramic video along the longitude direction.

47. The apparatus according to claim 44 or 45, characterized in that, The segmentation module is specifically used to cluster multiple pixels included in the two-dimensional planar projection map of the panoramic video to obtain multiple pixel sets; map the multiple pixels to the three-dimensional spherical panoramic image of the panoramic video; and take the region formed by the mapping points of the pixels included in the first pixel set on the three-dimensional spherical panoramic image of the panoramic video as the first segmentation region, wherein the first pixel set is one of the multiple pixel sets, and the first segmentation region is one of the multiple segmentation regions.

48. The apparatus according to claim 44 or 45, characterized in that, The segmentation module is specifically used to map multiple pixels included in the two-dimensional planar projection map of the panoramic video to the three-dimensional spherical panoramic representation map of the panoramic video to obtain multiple mapping points; to cluster the multiple mapping points to obtain multiple mapping point sets; and to use the region formed by the mapping points included in the first mapping point set as the second segmentation region, wherein the first mapping point set is one of the multiple mapping point sets, and the second segmentation region is one of the multiple segmentation regions.

49. A terminal device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-24.

50. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-24.

Images