Dynamic range mapping method and apparatus for panoramic video

Abstract

A dynamic range mapping method and apparatus for panoramic video related to the field of video image processing is provided. The method includes the steps of: dividing a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in a panoramic video to obtain a plurality of regions of the two-dimensional panoramic projection image; obtaining metadata of the two-dimensional panoramic projection image, the metadata including metadata information elements corresponding to the plurality of regions separately, and the metadata information element corresponding to one region including dynamic mapping information of the region; determining dynamic mapping information of each pixel within a current angle of view based on the dynamic mapping information of each of the plurality of regions; and performing dynamic range mapping for each pixel based on the dynamic mapping information of each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, the two-dimensional panoramic projection image being used for display or subsequent processing. According to the method, the dynamic range mapping of the panoramic video can be effectively implemented to improve the display effect of the panoramic video.

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202111555880.5, entitled "DYNAMIC RANGE MAPPING METHOD AND APPARATUS FOR PANORAMIC VIDEO," filed with the State Intellectual Property Office of China on December 17, 2021, which is incorporated herein by reference in its entirety.

[0002] FIELD Embodiments of the present application relate to the field of video image processing, and in particular to a dynamic range mapping method and apparatus for panoramic videos. [Background technology]

[0003] In the field of video / image processing, dynamic range is the ratio of the maximum grayscale value (also called luminance value) of a pixel in an image to the minimum grayscale value of the pixel in the image.

[0004] After capturing a video / image, the electronic device may display the video / image or transmit the video / image to another display device for display. Typically, different display devices have different dynamic ranges of pixels that can be displayed. When a display device displays the video / image, the dynamic range of the captured video / image may be different from the dynamic range supported by the display device. Therefore, the dynamic range of the captured video / image needs to be adjusted to the dynamic range supported by the display device to smoothly display the video / image on the display device. The above process of adjusting the dynamic range of a video / image is dynamic range mapping.

[0005] Currently, there are several mature dynamic range mapping methods for 2D videos / images (i.e., captured videos / images). However, with the development of technology, 3D videos / images (e.g., panoramic videos) are becoming increasingly popular, and there is no dynamic range mapping technique that is effective for 3D videos / images. Summary of the Invention

[0006] The embodiments of the present application provide a dynamic range mapping method and apparatus for panoramic video, to effectively implement dynamic range mapping on panoramic video, thereby improving the display effect of panoramic video. [Means for solving the problem]

[0007] To achieve the aforementioned objectives, the following technical solutions are used in the embodiments of this application.

[0008] According to a first aspect, an embodiment of the present application provides a dynamic range mapping method for panoramic video, the method being applied to an electronic device. The method includes a step in which the electronic device divides a 2D panoramic projection image to obtain multiple regions of the 2D panoramic projection image. The 2D panoramic projection image is a 2D panoramic projection image corresponding to any frame of a 3D panoramic image in the panoramic video, and each of the multiple regions satisfies a preset condition, the preset condition including at least one of the following conditions: pixels that are adjacent in the 2D panoramic projection image but not adjacent in the 3D panoramic image are not located in the same region; and pixels that are not adjacent in the 3D panoramic image are not located in the same region. The adjacent pixel points in the 2D panoramic projection image include pixels on at least two adjacent projection planes of the 3D panoramic image, and the non-adjacent pixels in the 3D panoramic image include pixels that are located in the 3D panoramic image and correspond to at least two non-adjacent projection planes. The electronic device then acquires metadata for the 2D panoramic projection image. The metadata includes metadata information elements corresponding to the multiple regions of the 2D panoramic projection image, and each metadata information element corresponding to a region includes dynamic mapping information for that region. The electronic device determines dynamic mapping information for each pixel within the current angular field of view based on the dynamic mapping information for each of the plurality of regions, and performs dynamic range mapping for each pixel within the current angular field of view based on the dynamic mapping information for each pixel within the current angular field of view to obtain a two-dimensional panoramic projection image within the current angular field of view, which is used for display or subsequent processing.

[0009] In this embodiment of the present application, when the regions obtained by the segmentation satisfy a preset condition, it is possible to ensure that pixels with similar brightness (similar brightness indicates similar dynamic ranges) are segmented into the same region, while pixels with significantly different brightness (large brightness difference indicates large dynamic range difference) are not segmented into the same region. In a 3D panoramic projection image, pixel 1 may be a sky pixel, and pixel 2 may be a ground pixel. Because the brightness of sky pixels is significantly different from the brightness of ground pixels, pixel 1 and pixel 2 are not located in the same region during the segmentation of the 2D panoramic projection image. Because the dynamic ranges of pixels in the same region are similar, dynamic range mapping can be performed for pixels in the same region based on the same dynamic range mapping information. Because the dynamic ranges of pixels in different regions may be significantly different, dynamic range mapping can be performed for pixels in different regions based on different dynamic range mapping information. Therefore, after the 2D panoramic projection image is segmented into multiple regions using this segmentation method, subsequent processing (i.e., dynamic range mapping) is performed for each region. The characteristics of the pixels within each region can be adaptively adapted to the process to improve processing efficiency.

[0010] In this embodiment of the present application, in relation to the large dynamic range of the panoramic video, the electronic device divides the 2D panoramic projection image corresponding to the 3D panoramic image to obtain regions, and performs dynamic range mapping for different regions, thereby effectively implementing dynamic range mapping on the panoramic video, thereby improving the display effect of the panoramic video.

[0011] In one possible implementation, the dynamic range mapping method for panoramic video provided in this embodiment of the present application further includes a step of projecting the three-dimensional panoramic image onto the surface of a polyhedron to obtain a two-dimensional panoramic projection image, and expanding the polyhedron into a two-dimensional plane, wherein the polyhedron includes at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0012] In this embodiment of the present application, the process of projecting a 3D panoramic image onto a 2D panoramic projection image using a polyhedron includes the steps of: placing a sphere capable of representing the 3D panoramic image (the 3D panoramic image is represented by a 3D sphere) on the polyhedron so that the sphere is an inscribed sphere of the polyhedron (it should be understood that the center of the polyhedron coincides with the spherical center of the sphere); connecting the sphere center to any point (pixel point) on the sphere and extending a connecting line to intersect with the surface of the polyhedron, where the point obtained by projecting the sphere point onto the surface of the polyhedron is the intersection point of the extension of the connecting line connecting the sphere center to the sphere point and the surface of the polyhedron; or connecting a point on the surface of the polyhedron to the sphere center, where the connecting line intersects with the sphere at a point, and the intersection point on the sphere and the point on the polyhedron have a mutual projection relationship. It should be understood that the pixel value of the point on the sphere is the pixel value of the intersection point on the surface of the polyhedron. Thus, in the aforementioned projection process, all pixels on the 3D sphere are projected to obtain projected pixels on each surface of the polyhedron (interpolation is performed for projected pixels on the 2D plane based on different cases). Finally, the polyhedron is expanded onto a 2D plane according to certain rules to obtain a 2D panoramic projection image.

[0013] The two-dimensional panoramic projection image obtained by expanding a polyhedron onto a two-dimensional plane may be an irregular projection image. In a subsequent processing process, the irregular projection image may be optionally processed directly, or the irregular image may be converted into a regular image for processing. In one embodiment, a rectangular region surrounding the irregular image and having a minimum area may be processed, and other regions other than the projection region within the rectangular region may be filled, for example, with preset pixel values. In another embodiment, the surface image within the irregular projection image may be stitched into a projection image of a regular shape (e.g., a rectangle). In this way, pixel filling does not need to be performed.

[0014] In this embodiment of the present application, the polyhedron is not limited to a regular polyhedron, in other words, a two-dimensional panoramic projection of a three-dimensional panoramic image may contain several polygons of different sizes.

[0015] In one possible embodiment, the step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image particularly includes a step of dividing the two-dimensional panoramic projection image using a first division method to obtain a plurality of regions of the two-dimensional panoramic projection image, the first division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions.

[0016] For example, the polyhedron used for projection is a regular hexahedron. After the 3D panoramic projection image is projected using the regular hexahedron, the resulting 2D panoramic projection image includes six projection surfaces. The first division method is a method of dividing pixels belonging to the same surface of the polyhedron into regions in the 2D panoramic projection image, i.e., each projection surface of the 2D panoramic projection image is used as a region. For example, the 2D panoramic projection image may be divided into six regions, Region A, Region B, Region C, Region D, Region E, and Region F, as shown in FIG. 8, using the first division method. Each of the multiple regions obtained by dividing the 2D panoramic projection image using the first division method satisfies a preset condition (including Condition 1 or Condition 2), i.e., pixels that are adjacent in the 2D panoramic projection image but not adjacent in the 3D panoramic image are not located in the same region.

[0017] In one possible embodiment, the step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image particularly includes a step of dividing the two-dimensional panoramic projection image using a second division method to obtain a plurality of regions of the two-dimensional panoramic projection image, the second division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions to obtain a plurality of intermediate regions, and further dividing at least one of the plurality of intermediate regions.

[0018] For example, the polyhedron used for projection is still a regular hexahedron. After the 3D panoramic projection image is projected using the regular hexahedron, the resulting 2D panoramic projection image includes six projection surfaces. The second division method divides pixels in the 2D panoramic projection image belonging to the same surface of the polyhedron into regions to obtain multiple intermediate regions, and further divides at least one of the multiple intermediate regions. In other words, the 2D panoramic projection image is first divided using the first division method to obtain multiple regions as the multiple intermediate regions, and then one or more of the multiple intermediate regions are divided to obtain multiple regions as the final division result. Each of the multiple regions obtained by dividing the 2D panoramic projection image using the second division method satisfies a preset condition (including condition 1 or condition 2), i.e., pixels that are adjacent in the 2D panoramic projection image but not adjacent in the 3D panoramic image are not located in the same region.

[0019] In one possible embodiment, the step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image includes the steps of: dividing the two-dimensional panoramic projection image using a third division scheme to obtain a plurality of intermediate regions of the two-dimensional panoramic projection image, where the third division scheme is a scheme of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region; and dividing intermediate regions among the plurality of intermediate regions that do not satisfy a preset condition using a fourth division scheme to obtain a plurality of regions, where for any intermediate region that does not satisfy the preset condition, the fourth division scheme is a scheme of dividing the intermediate region along the boundaries of at least two surfaces of a polyhedron included in the intermediate region.

[0020] In this embodiment of the present application, the process of clustering all pixels in a two-dimensional panoramic projection image with the third division scheme includes:

[0021] A clustering center is first determined, and the clustering center is one pixel. Optionally, the 2D panoramic projection image may be first divided into multiple regions using a preset division method, and the pixel value at the center of each region is determined, and the pixel value at the center of each region is used as the clustering center. For example, assume that the 2D panoramic projection image is an image obtained by projection using a regular hexahedron. If the 2D panoramic projection image is divided into six regions, the 2D panoramic projection image may be divided into six regions using a first division method, and the pixel value at the center of each region is determined to obtain six clustering centers. If the 2D panoramic projection image is divided into 12 regions, the 2D panoramic projection image may be divided into 12 regions using a second division method, and the pixel value at the center of each region is determined to obtain 12 clustering centers.

[0022] Then, according to a preset clustering algorithm and the determined clustering centers, a calculation is performed on the distance between all pixel points and each clustering center (the distance is calculated based on the coordinates of the pixel and the coordinates of the pixel corresponding to the clustering center), where the distance can be denoted as Di. Then, a calculation is performed on the difference between the luminance values ​​of all pixel points and the luminance values ​​of pixel points corresponding to the clustering centers (the difference between the luminance values ​​of all pixel points and the average luminance value of the pixel points in the category corresponding to each clustering center (i.e., the category obtained by dividing using the preset division method) is performed, the difference between the color values ​​of all pixel points and the color values ​​of pixel points corresponding to the clustering centers is performed, or the difference between the color values ​​of all pixel points and the average color value of the pixel points in the category corresponding to each clustering center (i.e., the category obtained by dividing using the preset division method) is performed), where the difference is denoted as Ei. Each pixel is divided into the category corresponding to the corresponding clustering center based on the weighted values ​​of the distances Di and Ei to obtain a clustering result. The category with the smallest weighted value is the category corresponding to the pixel, and the pixel points of each category in the clustering result form an area.

[0023] Optionally, dividing the 2D panoramic projection image according to the aforementioned clustering method cannot guarantee whether the obtained intermediate regions satisfy the preset conditions (including condition 1 or condition 2). Therefore, the intermediate regions obtained by clustering need to be determined to determine whether the intermediate regions satisfy the preset conditions. If all of the intermediate regions satisfy the preset conditions, the intermediate regions are used as the final division results, i.e., the intermediate regions are used as the regions obtained by dividing the 2D panoramic projection image. If there are intermediate regions among the intermediate regions that do not satisfy the preset conditions, the electronic device continues to divide the intermediate regions among the intermediate regions that do not satisfy the preset conditions using a fourth division method to obtain multiple regions.

[0024] In one possible implementation, the first division scheme, the second division scheme, or the third division scheme is preset in the electronic device (eg, preset according to a protocol).

[0025] In one possible embodiment, the metadata of the 2D panoramic projection image further includes division scheme indication information. The division scheme indication information indicates whether the division scheme of the 2D panoramic projection image is the first division scheme, the second division scheme, or the third division scheme, i.e., the division scheme indication information can be transmitted in the bitstream. After determining the division scheme of the 2D panoramic projection image, the generating end adds indication information indicating the division scheme (i.e., the division scheme indication information) to the bitstream. Therefore, after receiving the bitstream, the displaying end can know the division scheme of the 2D panoramic projection image based on the division scheme indication information in the bitstream.

[0026] It will be understood that the first division scheme is related to the second division scheme, and the division result of the second division scheme is the result of further division based on the first division scheme.

[0027] In one embodiment, when the candidate division schemes for a two-dimensional panoramic projection image include a first division scheme and a second division scheme, whether further division needs to be performed after division using the first division scheme may be determined based on the content of the two-dimensional panoramic projection image, i.e., whether to use the first division scheme or the second division scheme.

[0028] Specifically, whether the segmentation result of the first segmentation method needs to be further segmented can be determined based on the following process: first segmenting the 2D panoramic projection image using the first segmentation method to obtain multiple intermediate regions; calculating a histogram for each of the multiple intermediate regions; dividing the intermediate region into multiple regions and calculating a histogram for each region; calculating a sum of difference values ​​between the histogram of the intermediate region and the histogram of the region obtained by segmenting the intermediate region; determining whether the sum of difference values ​​is greater than a preset threshold; and if the sum of difference values ​​is greater than the preset threshold, further segmenting the intermediate region (i.e., using a second segmentation method to segment the 2D panoramic projection image) is performed. It should be understood that if the sum of difference values ​​is less than or equal to the preset threshold, the intermediate region does not need to be further segmented, i.e., the 2D panoramic projection image is segmented using the first segmentation method. It should be understood that if the sum of difference values ​​is greater than the preset threshold, it indicates that the brightness difference between the portions within the intermediate region is large and further segmentation needs to be performed.

[0029] Optionally, in some cases, when the default division scheme is the first division scheme, an associated identifier may be used in the bitstream to indicate whether further division is to be performed. When the identifier indicates that further division is not to be performed, the division result obtained by the first division scheme is the final result. When the identifier indicates that further division needs to be performed, the division result of the first division scheme is further divided.

[0030] In another embodiment, when candidate division schemes for a 2D panoramic projection image include a first division scheme, a second division scheme, and a third division scheme, which division scheme to use can be determined by performing the following processes: first dividing the 2D panoramic projection image using the first division scheme, the second division scheme, and the third division scheme; separately calculating histograms of the regions obtained using the first division scheme, the second division scheme, and the third division scheme and a histogram of the 2D panoramic projection image; calculating the sum of difference values ​​between the histograms of the regions obtained using the division schemes and the histogram of the 2D panoramic projection image; and determining the division scheme that maximizes the sum of difference values ​​between the histograms as the final division scheme. It should be understood that a larger sum of difference values ​​between the histograms indicates a larger luminance difference between the regions obtained by the division, and indicates that the division is appropriate.

[0031] Optionally, the sizes of the metadata for the three division schemes may be further calculated, and the metadata includes dynamic mapping information corresponding to each region of the 2D panoramic projection image. Then, which division scheme is used is determined based on a weighted value of the sum of the histogram difference values ​​corresponding to the division schemes and the sizes of the metadata corresponding to the division schemes.

[0032] In one possible implementation, before the step of determining dynamic mapping information of each pixel in the current region based on the dynamic mapping information of each of the multiple regions, the dynamic range mapping method for panoramic videos provided in this embodiment of the present application further includes the steps of obtaining a correspondence between the region and a metadata information element, and determining metadata information elements corresponding to the multiple regions separately from the metadata according to the correspondence between the region and the metadata information element, so as to determine the dynamic mapping information of each of the multiple regions.

[0033] It should be understood that in this embodiment of the present application, the concept of performing dynamic range mapping on a panoramic video is to project all 3D panoramic images included in the panoramic video onto a 2D plane, perform dynamic range mapping on the obtained 2D panoramic projection images, and convert the 2D panoramic projection images obtained by dynamic range mapping into 3D panoramic images to obtain a panoramic video obtained by dynamic range mapping. In other words, the metadata of the obtained 2D panoramic projection images is used to perform dynamic range mapping of the 3D panoramic image (i.e., used to perform dynamic range mapping on the panoramic video), and the metadata of the 2D panoramic projection images may be considered as the metadata of the 3D panoramic image (or the metadata of the panoramic video).

[0034] Optionally, the correspondence between regions and metadata information elements may be a one-to-one relationship or a many-to-one relationship. In other words, one region of the 2D panoramic projection image corresponds to one metadata information element, or multiple regions of the 2D panoramic projection image correspond to one metadata information element. When the brightness difference between multiple regions of the 2D panoramic projection image is large and one type of dynamic mapping information is used for each region, the correspondence between regions and metadata information elements is a one-to-one relationship. When the brightness of some regions in the multiple regions is similar, some regions may use the same type of dynamic mapping information, and the correspondence between regions and metadata information elements is a many-to-one relationship.

[0035] In this embodiment of the present application, a preset algorithm may be used to determine whether the correspondence between multiple regions in the 2D panoramic image and metadata information elements is a one-to-one or many-to-one relationship. For example, whether to use a one-to-one or many-to-one correspondence may be determined by using the following method: multiple regions are clustered, and the number of clustered categories can be set based on requirements. For example, the number of categories may be set to half the number of regions or one-quarter the number of regions. For example, when the total number of regions is eight, the eight regions may be divided into two categories, or the eight regions may be divided into four categories, which corresponds to obtaining four large regions. Then, the sum of the differences between the histograms of the regions corresponding to the categories in the clustering result and the histogram of the 2D panoramic projection image is calculated, and the sum of the differences between the histograms of the multiple original regions and the histogram of the 2D panoramic image is calculated. Finally, based on the sum of the two differences between the histograms, it is determined whether the correspondence between the multiple regions and metadata information elements is a one-to-one or many-to-one relationship. For example, when eight regions are clustered to obtain two categories of regions, the sum of the differences between the histograms of the regions corresponding to each category of the clustering result and the histogram of the 2D panoramic projection image is denoted as DiffHIS_X2, and the sum of the differences between the histograms of multiple original regions and the histogram of the 2D panoramic image is denoted as DiffHIS_org. The determination method is as follows: if DiffHIS_org×(1-T2)>DiffHIS_X2, the correspondence between the regions and the metadata information elements is a many-to-one relationship, or if DiffHIS_org×(1-T2)≦DiffHIS_X2, the correspondence between the regions and the metadata information elements is a one-to-one relationship, where T2 is a preset threshold.

[0036] In one possible implementation, the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and an index of the region, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata element and the coordinates of one or more pixels within the region.

[0037] In this embodiment of the present application, the correspondence between the indexes of the metadata information elements and the indexes of the regions is essentially a traversal order relationship. For example, in the case of a 2D panoramic projection image including six regions, the six regions form a 2x3 rectangle (i.e., two rows, each row including three regions), and are, from left to right and top to bottom, Region 4, Region 0, Region 5, Region 3, Region 1, and Region 2, in that order. Therefore, in the table of correspondence between regions and metadata information elements, the six regions in the 2D panoramic projection image are traversed sequentially from left to right and top to bottom, and the metadata information elements corresponding to the six regions separately can be determined sequentially in the metadata. In one embodiment, the traversal order of multiple regions can be indicated in the bitstream. For example, instruction information indicating the traversal order can be carried in the metadata. The traversal order is not limited to the above order from left to right and top to bottom; the traversal order is specifically related to the position of each region in the 2D panoramic projection image.

[0038] Optionally, the traversal order of the multiple regions may be a preset order, or the generating end determines a traversal order from the candidate traversal orders according to an associated algorithm, adds an identifier to the metadata of the bitstream, and forwards the bitstream to the generating end to indicate the traversal order.

[0039] In this embodiment of the present application, which traversal order is used is determined by the following steps: computing a histogram for each of a plurality of regions; for each candidate traversal order, computing the sum of the differences between the histograms of adjacent regions; and selecting the traversal order that results in the smallest sum of the differences between the histograms in the candidate traversal order.order The final traversal order and determining the value as:

[0040] In this embodiment of the present application, the coordinates of pixels in the regions can also be carried in the metadata. Specifically, the coordinates of pixels in each region are carried in the metadata information element corresponding to each region. After dividing the 2D panoramic projection image to obtain multiple regions, for one of the regions, the display end determines the pixels whose coordinates belong to the region according to the correspondence relationship, and uses the metadata information element corresponding to the coordinates as the metadata information element of the region.

[0041] In one possible embodiment, the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of multiple regions within a first preset range, where the first preset range is a range centered on the pixel to be processed (method 1), the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a region to which a pixel in a two-dimensional panoramic projection image belongs, which corresponds to the viewpoint center of the current angle of view range (method 2), the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of multiple regions within a second preset range, where the second preset range is a range centered on a pixel in a two-dimensional panoramic projection image, which corresponds to the viewpoint center of the current angle of view range (method 3), or the dynamic mapping information of one pixel in the current region is dynamic mapping information of the current region (method 4).

[0042] It should be understood that when a user watches a panoramic video, the user (head) cannot see all the content of the panoramic video at the current pose, but can only see the content of the video within a certain range. The range can be understood as the user's angle of view range. It can be seen that the user's angle of view range is related to the user's current viewing pose. In this embodiment of the present application, performing dynamic range mapping on the panoramic video is performing dynamic range mapping on the content within the angle of view range of the user currently watching the panoramic video. After obtaining the dynamic mapping information of the multiple regions, the electronic device can determine the dynamic mapping information of each pixel within the current angle of view range based on the dynamic mapping information of the multiple regions in one of four ways.

[0043] Optionally, during actual application, the generating end may select an appropriate method from the four methods based on the actual requirements for determining the dynamic mapping information of the pixel to be processed, and transfer the selected method carried in the bitstream (metadata) to the display end. Alternatively, the generating end and the display end may predetermine the method to be used, which is not limited in this embodiment of the present application.

[0044] In one possible implementation, the dynamic mapping information in this embodiment of the present application is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range, or the dynamic mapping information is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a second dynamic range to the first dynamic range, where the first dynamic range is greater than the second dynamic range, i.e., the dynamic range mapping performed on the pixels may be a mapping from a high dynamic range to a low dynamic range or a mapping from a low dynamic range to a high dynamic range.

[0045] Correspondingly, after the dynamic mapping information for each pixel within the current angle of view is determined using this method, the pixel may correspond to multiple pieces of dynamic mapping information (e.g., multiple dynamic mapping curves). After dynamic range mapping is performed on the pixel, one dynamic mapping result may be obtained for each piece of dynamic mapping information. In this case, the multiple dynamic mapping results may be processed to obtain a final dynamic mapping result. To obtain the final dynamic mapping result, the processing method may include either using a median value of the multiple dynamic mapping results as the final dynamic mapping result or performing a weighted average on the multiple dynamic mapping results.

[0046] In one possible embodiment, the step of obtaining the metadata of the 2D panoramic projection image specifically includes a step of receiving the metadata of the 2D panoramic projection image from another electronic device. Specifically, after generating the metadata, the generating end encodes the metadata into a bitstream and sends the bitstream to the displaying end, so that the displaying end receives the bitstream and analyzes the bitstream to obtain the metadata.

[0047] In one possible implementation, the dynamic range mapping method for panoramic video in this embodiment of the present application further includes performing a subsequent process on the 2D panoramic projection image within the current angle of view range, where the subsequent process includes performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range, and the 3D panoramic image is used for display.

[0048] The obtained 2D panoramic projection image may be used for display. Furthermore, the 2D panoramic projection image may be further used for subsequent processing, for example, the 2D panoramic projection image may be converted into a 3D panoramic image for display. Furthermore, after all the 3D panoramic images of the panoramic video are obtained, the panoramic video may be played.

[0049] According to a second aspect, an embodiment of the present application provides a dynamic range mapping device for panoramic video, including a segmentation module, an acquisition module, a determination module, and a processing module. The segmentation module is configured to segment a 2D panoramic projection image to obtain multiple regions of the 2D panoramic projection image. The 2D panoramic projection image is a 2D panoramic projection image corresponding to any frame of a 3D panoramic image in the panoramic video, and each of the multiple regions satisfies a preset condition, the preset condition including at least one of the following conditions: pixels that are adjacent in the 2D panoramic projection image and are not adjacent in the 3D panoramic image are not located in the same region; and pixels that are not adjacent in the 3D panoramic image are not located in the same region. The adjacent pixel points in the 2D panoramic projection image include pixels on at least two adjacent projection planes of the 3D panoramic image, and the non-adjacent pixels in the 3D panoramic image include pixels that are in the 3D panoramic image and correspond to at least two non-adjacent projection planes. The acquisition module is configured to acquire metadata of the 2D panoramic projection image, the metadata including metadata information elements respectively corresponding to multiple regions of the 2D panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information for the region. The determination module is configured to determine dynamic mapping information for each pixel within the current angle of view range based on the dynamic mapping information for each of the multiple regions. The processing module is configured to perform dynamic range mapping for each pixel within the current angle of view range based on the dynamic mapping information for each pixel within the current angle of view range to obtain a 2D panoramic projection image within the current angle of view range, and the 2D panoramic projection image within the current angle of view range is used for display or subsequent processing.

[0050] In one possible implementation, the dynamic range mapping device for panoramic video provided in this embodiment of the present application further includes a transformation module configured to project the three-dimensional panoramic image onto a surface of a polyhedron and expand the polyhedron onto a two-dimensional plane to obtain a two-dimensional panoramic projection image. The polyhedron includes at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0051] In one possible embodiment, the division module is particularly configured to divide the two-dimensional panoramic projection image using a first division method to obtain multiple regions of the two-dimensional panoramic projection image, the first division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions.

[0052] In one possible embodiment, the division module is particularly configured to divide the two-dimensional panoramic projection image using a second division method to obtain multiple regions of the two-dimensional panoramic projection image, the second division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions to obtain multiple intermediate regions, and further dividing at least one of the multiple intermediate regions.

[0053] In one possible embodiment, the segmentation module is particularly configured to: segment the two-dimensional panoramic projection image using a third segmentation scheme to obtain multiple intermediate regions of the two-dimensional panoramic projection image, the third segmentation scheme being a scheme of clustering all pixels in the two-dimensional panoramic projection image and segmenting pixels belonging to the same category into the same intermediate region; and segment any intermediate region among the multiple intermediate regions that does not satisfy a preset condition using a fourth segmentation scheme to obtain multiple regions, and for any intermediate region that does not satisfy the preset condition, the fourth segmentation scheme being a scheme of dividing the intermediate region along the boundaries of at least two surfaces of a polyhedron included in the intermediate region.

[0054] In one possible embodiment, the first division scheme, the second division scheme or the third division scheme is preset in the electronic device.

[0055] In one possible embodiment, the metadata of the two-dimensional panoramic projection image further includes division method indication information, and the division method indication information indicates whether the division method of the two-dimensional panoramic projection image is a first division method, a second division method, or a third division method.

[0056] In one possible embodiment, the acquiring module is further configured to acquire a correspondence relationship between the regions and the metadata information elements, and the determining module is further configured to determine metadata information elements corresponding to the plurality of regions separately from the metadata according to the correspondence relationship between the regions and the metadata information elements, so as to determine dynamic mapping information for each of the plurality of regions.

[0057] In one possible implementation, the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and an index of the region, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata element and the coordinates of one or more pixels within the region.

[0058] In one possible embodiment, the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of multiple regions within a first preset range, the first preset range being a range centered on the pixel to be processed; the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a region to which a pixel in a 2D panoramic projection image corresponding to the viewpoint center of the current angle of view range belongs; the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of multiple regions within a second preset range, the second preset range being a range centered on a pixel in a 2D panoramic projection image corresponding to the viewpoint center of the current angle of view range; or the dynamic mapping information of one pixel in the current region is dynamic mapping information of the current region.

[0059] In one possible implementation, the dynamic mapping information in this embodiment of the present application is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range, or the dynamic mapping information is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a second dynamic range to a first dynamic range, where the first dynamic range is greater than the second dynamic range.

[0060] In one possible implementation, the acquisition module is specifically configured to receive metadata of the two-dimensional panoramic projection image from another electronic device.

[0061] In one possible embodiment, the transformation module is further configured to perform subsequent processing on the 2D panoramic projection image within the current angle of view range, the subsequent processing including performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range, which is used for display.

[0062] According to a third aspect, an embodiment of the present application provides an electronic device including a memory and at least one processor connected to the memory, the memory configured to store instructions, and the instructions being read by the at least one processor before the method of the first aspect and any one of the possible implementations of the first aspect is performed.

[0063] According to a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method of the first aspect and any one of possible implementations of the first aspect.

[0064] According to a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising instructions that, when run on a computer, perform the method of the first aspect or any one of the possible implementations of the first aspect.

[0065] According to a sixth aspect, an embodiment of the present application provides a chip including a memory and a processor, wherein the memory is configured to store computer instructions, and the processor is configured to retrieve and execute the computer instructions from the memory to perform a method according to the first aspect or any one of the possible implementations of the first aspect.

[0066] For the beneficial effects achieved by the technical solutions in the second to sixth aspects and the corresponding possible implementations in the embodiments of the present application, please refer to the aforementioned technical effects in the first aspect and the corresponding possible implementations, and the details will not be repeated here. [Brief explanation of the drawings]

[0067] [Figure 1] 1 is a schematic diagram of a framework of a video processing process according to an embodiment of the present application; [Figure 2] 1 is a schematic diagram of the hardware of a mobile phone according to an embodiment of the present application. [Figure 3] 1 is a schematic diagram 1 of a dynamic range mapping method for panoramic video according to an embodiment of the present application; [Figure 4] 1 is a schematic diagram 1 of a 3D panoramic image projection process according to an embodiment of the present application. [Figure 5] 2 is a schematic diagram 2 of a 3D panoramic image projection process according to an embodiment of the present application. [Figure 6] 3 is a schematic diagram 3 of a 3D panoramic image projection process according to an embodiment of the present application. [Figure 7] 2 is a schematic diagram 2 of a dynamic range mapping method for panoramic video according to an embodiment of the present application; [Figure 8] 1 is a schematic diagram 1 of a segmentation result of a 2D panoramic projection image according to an embodiment of the present application; [Figure 9] 3 is a schematic diagram 3 of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 10] 2 is a schematic diagram 2 of a segmentation result of a 2D panoramic projection image according to an embodiment of the present application; [Figure 11] 4 is a schematic diagram 4 of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 12] 3 is a schematic diagram 3 of a segmentation result of a 2D panoramic projection image according to an embodiment of the present application; [Figure 13] 5 is a schematic diagram 5 of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 14] 6 is a schematic diagram 6 of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 15] 1 is a schematic diagram 1 of the structure of a dynamic range mapping device for panoramic video according to an embodiment of the present application; [Figure 16] 2 is a schematic diagram 2 of the structure of a dynamic range mapping device for panoramic video according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0068] The term "and / or" in this specification only describes an association relationship for describing related objects and indicates that three relationships may exist. For example, A and / or B can represent three cases: when only A exists, when both A and B exist, or when only B exists.

[0069] In the specification and claims of the embodiments of the present application, terms such as "first" and "second" are intended to distinguish between different objects, but do not indicate a particular order of the objects. For example, terms such as "first division scheme" and "second division scheme" are used to distinguish between different division schemes, but are not used to describe a particular order of the division schemes.

[0070] In the embodiments of the present application, the words "example" or "for example" are used to provide an example, illustration, or explanation. In the embodiments of the present application, any embodiment or design scheme described as an "example" or "for example" should not be described as being preferred or having more advantages over another embodiment or design scheme. Rather, words such as "example" or "for example" are intended to present the relevant concept in a particular manner.

[0071] In describing the embodiments of the present application, unless otherwise specified, "plurality" means two or more. For example, "plurality of regions" means two or more regions.

[0072] In the following, we first explain some concepts related to the dynamic range mapping method and apparatus for panoramic video provided in the embodiments of the present application.

[0073] Dynamic range: In most fields, the dynamic range is the ratio between the maximum and minimum values ​​of a variable. In the digital image field, the dynamic range is the ratio between the maximum and minimum tonal values ​​of the pixels in the image. Tonal values ​​are also expressed as 1 nit = 1 cd / m 2 is measured in nits or candelas per square meter (cd / m 2 ) may be understood as a luminance value in units of .

[0074] The dynamic range in nature is generally large, with the luminance of a night scene under the stars being approximately 0.001 cd / m 2 and the brightness of the sun is 10 9 cd / m2 Thus, the dynamic range in nature can reach 10 9 / 0.001=10 12 In a real natural environment scene, the brightness of the sun and the brightness of the stars cannot be obtained simultaneously, so the dynamic range of a real natural environment scene is 10 12 In real-world scenes, the dynamic range is usually 10 -3 From 10 6 is.

[0075] Dynamic Range Mapping: After a video / image is captured in a real natural environment, in the process of displaying the captured video / image on a display device (e.g., a television or an iPad®), the dynamic range of the displayed video / image supported by the display device may be different from the original dynamic range of the video / image captured from the real natural environment. Therefore, if the video / image is directly displayed based on the original dynamic range of the video / image, the display device cannot display the video / image. Therefore, the dynamic range of the captured video / image needs to be adapted to the display device, i.e., the dynamic range of the captured video / image is adjusted to the dynamic range supported by the display device to display the captured video / image on the display device. The above process of adjusting the dynamic range of a video / image is dynamic range mapping (sometimes called tone mapping).

[0076] It will be appreciated that dynamic ranges are classified as high dynamic range (HDR) or low dynamic range, with low dynamic range sometimes referred to as standard dynamic range (SDR). -3 From 10 6An image with a luminance range of 0 to 255 is called a high dynamic range image, and an image with a luminance range of 0 to 255 is called a low dynamic range image. Currently, in most color digital images, one byte (i.e., 8 bits) represents a pixel for each of the R, G, and B channels, i.e., the luminance range of a pixel for each channel is 0 to 255, and 0 to 255 is the standard dynamic range of an image.

[0077] In the embodiment of the present application, the dynamic range mapping includes mapping from a high dynamic range to a low dynamic range and mapping from a low dynamic range to a high dynamic range. Note that "high" and "low" here are relative concepts and are different from the aforementioned concepts of a high dynamic range and a low dynamic range. For example, the dynamic range is mapped from dynamic range 1 to dynamic range 2. The relationship between dynamic range 1 and dynamic range 2 may be such that dynamic range 1 is larger than dynamic range 2, or dynamic range 1 may be smaller than dynamic range 2. However, both dynamic range 1 and dynamic range 2 may be high dynamic ranges or low dynamic ranges.

[0078] For example, the original dynamic range of a captured video / image is HDR 1000, and the dynamic range supported by the display device is SDR 100. In this case, the dynamic range mapping is from a high dynamic range to a low dynamic range, and the dynamic range before the mapping is HDR and the dynamic range after the mapping is SDR. In another example, the original dynamic range of a captured video / image is HDR 4000, and the dynamic range supported by the display device is HDR 500. In this case, the dynamic range mapping is also from a high dynamic range to a low dynamic range, but both the dynamic range before the mapping and the dynamic range after the mapping are HDR.

[0079] For example, the original dynamic range of a captured video / image is SDR of 100, and the dynamic range supported by the display device is HDR of 2000. In this case, the dynamic range mapping is from a low dynamic range to a high dynamic range, and the dynamic range before the mapping is SDR, and the dynamic range after the mapping is HDR. In another example, the original dynamic range of a captured video / image is SDR of 100, and the dynamic range supported by the display device is SDR of 200. In this case, the dynamic range mapping is from a low dynamic range to a high dynamic range, but both the dynamic range before the mapping and the dynamic range after the mapping are SDR.

[0080] Referring to the previous examples, Table 1 shows some examples of dynamic range mapping.

[0081] [Table 1]

[0082] When dynamic range mapping is performed on a video / image, several dynamic range mapping models may be used to perform dynamic range mapping on the pixels of the captured video / image. The dynamic range mapping models include, but are not limited to, a sigmoidal curve, a Bessel curve, etc. It should be understood that the dynamic range mapping model is a model for performing dynamic range mapping on a two-dimensional video / image.

[0083] Optionally, the method for performing dynamic range mapping on videos / images includes static dynamic range mapping and dynamic dynamic range mapping. Static dynamic range mapping is performing dynamic range mapping on videos / images by using the same dynamic range mapping curve for the same video / image content (which may relate to video / image content of multiple scenes) or video / image content in the same hard disk. Dynamic dynamic range mapping is performing dynamic range mapping on videos / images using different dynamic range mapping curves based on video / image content of different scenes or different frames.

[0084] The advantage of static dynamic range mapping is that it is a simple dynamic range mapping process and little information is carried to transmit dynamic mapping information (e.g., information including the dynamic range mapping curve). The disadvantage of static dynamic range mapping is that the mapping results obtained by performing dynamic range mapping on a video / image using a single dynamic range mapping curve may be poor, resulting in an insufficient display effect for the video / image obtained by dynamic range mapping. For example, if a dynamic range mapping curve is mainly used to perform dynamic range mapping for high brightness, the effect of performing dynamic range mapping on the bright areas of the image using the dynamic range mapping curve is good. However, in dark areas of the image, when the dynamic range mapping curve is used to perform dynamic range mapping, the pixel values ​​in the dark areas become too small, resulting in the loss of some information (e.g., detailed information) in the image obtained by dynamic range mapping.

[0085] The advantage of dynamic dynamic range mapping is that for different scenes or frames, different dynamic range mapping curves suitable for different scenes or frames are used to perform dynamic range mapping, and the mapping results are good, that is, the video / image display effect obtained by dynamic range mapping is good. The disadvantage of dynamic dynamic range mapping is the complex dynamic range mapping process and the large amount of information required to transmit the dynamic mapping information.

[0086] Panoramic Video: A panoramic video is a video containing spherical scene content covering 360 degrees horizontally and 180 degrees vertically. A panoramic video includes a three-dimensional (3D) panoramic image of multiple frames, and the 3D panoramic image can be represented using a 3D sphere. Therefore, the 3D panoramic image of a panoramic video is sometimes referred to as a 3D spherical representation panorama of the panoramic video.

[0087] Compared with typical two-dimensional videos (which cover a range of 33 degrees horizontally and 18 degrees vertically), panoramic videos cover a wider range, providing users with an immersive visual experience. Users watch panoramic videos using a wearable assistive device (e.g., VR glasses). Specifically, the user's head may be used as the center point and the head may be rotated 360 degrees horizontally and 180 degrees vertically. The user can freely select a designated direction for viewing the window by adjusting the horizontal and vertical angles of the field of view, and the window is the user's field of view range. It can be understood that if the panoramic video is simulated as a spherical surface of a sphere and the user's head is located at the center of the sphere, the window can be continuously adjusted by rotating the head and / or looking up / down to view the complete panoramic video.

[0088] In an embodiment of the present application, panoramic video has a larger dynamic range than general 2D video due to its larger brightness coverage range. Specifically, 2D video is captured in a narrow range, and the lighting conditions in the narrow range are similar, i.e., the light change range is small and the dynamic range is also small. Panoramic video is captured in a wide range, and the light change range is usually large. For example, a panoramic video captured during the day may include areas of the sun in the sky, which are bright (i.e., high brightness values), and may also include areas inside buildings or shadow areas, which are dark (i.e., low brightness values). Therefore, the large dynamic range of panoramic video poses significant challenges in describing and displaying panoramic video.

[0089] Currently, the technology for performing dynamic range mapping on 2D videos / images is mature. However, with the emergence of panoramic videos, there is an urgent need for related technology for performing dynamic range mapping on panoramic videos. In view of this, an embodiment of the present application provides a dynamic range mapping method and apparatus for panoramic videos. For any frame of a 3D panoramic image in a panoramic video, an electronic device performs dynamic range mapping on a 2D panoramic projection image corresponding to the 3D panoramic image, and converts the 2D panoramic projection image obtained through dynamic range mapping into a 3D panoramic image, thereby performing dynamic range mapping on the panoramic video.

[0090] The specific solution of the embodiments of the present application is as follows: The electronic device divides a 2D panoramic projection image corresponding to a 3D panoramic image to obtain multiple regions of the 2D panoramic projection image, each of which satisfies a preset condition (the preset condition will be described in detail in the following embodiments). The electronic device acquires metadata of the 2D panoramic projection image, the metadata including metadata information elements respectively corresponding to multiple regions of the 2D panoramic projection image, and each metadata information element corresponding to a region includes dynamic mapping information for that region. The electronic device then determines dynamic mapping information for each pixel within a current field of view range based on the dynamic mapping information for each of the multiple regions, and performs dynamic range mapping for each pixel within the current field of view range (the field of view range of the user currently viewing the panoramic video) based on the dynamic mapping information for each pixel within the current field of view range to obtain a 2D panoramic projection image within the current field of view range, which is used for display or subsequent processing. In the dynamic range mapping method for panoramic video provided in the embodiments of the present application, in relation to the large dynamic range of panoramic video, the electronic device divides the 2D panoramic projection image corresponding to the 3D panoramic image to obtain regions, and performs dynamic range mapping for the different regions. This can effectively perform dynamic range mapping on the panoramic video, thereby improving the display effect of the panoramic video.

[0091] It should be noted that in the embodiment of the present application, the format of the images (sometimes referred to as source data) included in the panoramic video is not limited, and the images may be in YUV format or RGB format. In addition, in the embodiment of the present application, the bit width of the source data is limited. figure ,The bit width of the source data may be 8 bits (bit), 10 bits, or 12 bits.

[0092] It should be understood that for a panoramic video, the processing process of the panoramic video generally includes generating, encoding, decoding, and displaying the panoramic video. The generating and encoding process of the panoramic video may be completed at a front end (sometimes referred to as a generating end), and the decoding and displaying process of the panoramic video may be completed at a back end (sometimes referred to as a display end). See FIG. 1. At the generating end, generating the panoramic video includes performing simulated dynamic range mapping for each frame of a 3D panoramic image in the panoramic video, determining dynamic mapping information (e.g., the dynamic mapping information includes information of a curve used for dynamic range mapping), and generating metadata including the dynamic mapping information. Encoding the panoramic video includes encoding each frame of the 3D panoramic image in the panoramic video (e.g., including prediction, transformation, quantization, and entropy coding). For the image encoding process, please refer to the prior art. The panoramic video encoding method is not limited in this application. The generating end further writes the generated metadata including the dynamic mapping information into a bitstream and transmits the bitstream to the display end. At the display end, the bitstream is parsed to obtain metadata and image decoding data (including syntax elements used for image decoding and image data), and the image data is decoded (e.g., including prediction, inverse quantization, and inverse transform) based on the syntax elements used for image decoding to obtain a 3D panoramic image. For the image decoding process, please refer to the prior art (the decoding method corresponds to the encoding method). The panoramic video decoding method is not limited in this application. Then, the 3D panoramic image is converted into a 2D panoramic projection image, and dynamic range mapping is performed on the 2D panoramic projection image based on the metadata obtained by parsing the bitstream, and the panoramic video obtained by dynamic range mapping is displayed.

[0093] In the embodiment of the present application, the dynamic range mapping process of the panoramic video is described from the perspective of the display end. For example, the product forms of electronic devices used as the display end include, but are not limited to, virtual reality (VR) devices (e.g., VR glasses), display devices (e.g., mobile phones), video application conversion devices (e.g., transcoders), and live streaming devices. It should be understood that in VR glasses and display devices, the dynamic range mapping is implemented based on hardware chips, and in live streaming devices and video conversion devices, the dynamic range mapping is implemented based on software program code.

[0094] For example, in an embodiment of the present application, the display terminal is a mobile phone. Figure 2 is a diagram of the hardware structure of a mobile phone 200 according to an embodiment of the present application. The mobile phone 200 includes a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, a subscriber identification module (SIM) card interface 295, etc.

[0095] It will be understood that the exemplary structures in the embodiments of the present application do not constitute specific limitations on the mobile phone 200. In some other embodiments of the present application, the mobile phone 200 may include more or fewer components than those shown in the figures, or some components may be combined, or some components may be divided, or components may be arranged in a different manner. The components shown in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0096] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The various processing units may be independent components or integrated into one or more processors. The controller may be the nerve center and command center of the mobile phone 200. The controller may generate operation control signals based on instruction operation codes and time-series signals to complete the control of instruction reading and instruction execution.

[0097] A memory may also be disposed in the processor 210 and configured to store instructions and data. In some embodiments, the memory in the processor 210 is a cache. The memory may store instructions or data that have just been used or that are periodically used by the processor 210. When the processor 210 needs to use the instructions or data again, the processor 210 may retrieve the instructions or data directly from the memory. This avoids repeated accesses and reduces the latency of the processor 210, thereby improving system efficiency.

[0098] The charging management module 240 is configured to receive a charging input from a charger. While charging the battery 242, the charging management module 240 may also power the electronic device using the power management module 241.

[0099] Power management module 241 is configured to connect battery 242, charge management module 240, and processor 210. Power management module 241 receives input from battery 242 and / or input from charge management module 240 and provides power to processor 210, internal memory 221, external memory, display 294, camera 293, wireless communication module 260, etc. Power management module 241 may be further configured to monitor parameters such as battery capacity, battery cycle count, and battery health (leakage and impedance). In some other embodiments, power management module 241 may be alternatively located in processor 210. In some other embodiments, power management module 241 and charge management module 240 may be alternatively located in the same component.

[0100] The wireless communication function of the mobile phone 200 may be implemented using Antenna 1, Antenna 2, mobile communication module 250, wireless communication module 260, a modem processor, a baseband processor, etc. Antenna 1 and Antenna 2 are configured to transmit and receive electromagnetic wave signals.

[0101] Mobile communication module 250 may provide wireless communication solutions applied to mobile phone 200, including 2G / 3G / 4G / 5G, etc. Mobile communication module 250 may receive electromagnetic waves via antenna 1, perform processing such as filtering or amplification on the received electromagnetic waves, and transmit the processed electromagnetic waves to a modem processor for demodulation. Mobile communication module 250 may further amplify signals modulated by the modem processor and convert the amplified signals into electromagnetic waves for emission via antenna 1. In some embodiments, at least some functional modules in mobile communication module 250 may be located within processor 210. In some embodiments, at least some functional modules in mobile communication module 250 may be located within the same component as at least some modules in processor 210.

[0102] The wireless communication module 260 may provide wireless communication solutions applied to the mobile phone 200, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC) technology, or infrared (IR) technology. The wireless communication module 260 may be one or more components integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 210. The wireless communication module 260 may further receive signals to be transmitted from the processor 210, perform frequency modulation and amplification on the signals, and convert the signals into electromagnetic waves for emission via the antenna 2.

[0103] In some embodiments, in mobile phone 200, antenna 1 is coupled to mobile communication module 250 and antenna 2 is coupled to wireless communication module 260, such that mobile phone 200 can communicate with a network and other devices using wireless communication techniques.

[0104] The mobile phone 400 realizes display functions by using a GPU, a display 294, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 294 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0105] The display 294 is configured to display images, videos, etc. The display 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 200 may include one or N displays 294, where N is a positive integer greater than 1.

[0106] The mobile phone 200 may perform a photography function using an ISP, a camera 293, a video codec, a GPU, a display 294, an application processor, etc. The ISP is configured to process data fed back by the camera 293. For example, during photography, a shutter is pressed, light is transmitted through a lens to the camera's photosensitive elements, the light signal is converted into an electrical signal, and the camera's photosensitive elements transmit the electrical signal to the ISP for processing, which converts the electrical signal into a visible image. The ISP may further perform algorithmic optimization for image noise, brightness, and skin tone. The ISP may further optimize parameters such as exposure and color temperature of the photographed scene. In some embodiments, the ISP may be located in the camera 293.

[0107] The camera 293 is configured to capture still images or video. An optical image of an object is generated through a lens and projected onto a light-receiving element. The light-receiving element may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light-sensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. In some embodiments, the mobile phone 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0108] The digital signal processor is configured to process digital signals, and may process other digital signals in addition to digital image signals (eg, audio signals).

[0109] A video codec is configured to compress or decompress digital video. Mobile phone 200 may support one or more video codecs. Thus, mobile phone 200 can play or record video in multiple encoding formats, such as Moving Picture Experts Group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4.

[0110] The external memory interface 220 may be configured to connect to an external memory card, such as a Micro SD card, to expand the storage capabilities of the mobile phone 200. The external memory card communicates with the processor 210 through the external memory interface 220 to perform data storage functions. For example, files such as music and videos are stored on the external memory card.

[0111] The internal memory 221 may be configured to store computer-executable program code, which includes instructions. The processor 210 executes the instructions stored in the internal memory 221 to implement various functional applications and data processing of the mobile phone 200. The internal memory 221 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function or image playback function), etc. The data storage area may store data created during use of the mobile phone 200 (e.g., audio data or an address book), etc. Additionally, the internal memory 221 may include high-speed random access memory and non-volatile memory, such as at least one magnetic disk storage device, flash memory, or universal flash storage (UFS).

[0112] The mobile phone 200 may use an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset interface 270D, an application processor, etc. to perform audio functions such as playing or recording music.

[0113] The audio module 270 is configured to convert digital audio information into analog audio signals for output, and is also configured to convert analog audio input into digital audio signals. The audio module 270 may be further configured to encode and decode audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0114] Speaker 270A, also called a "horn," is configured to convert audio electrical signals into acoustic signals. Mobile phone 200 may use speaker 270A to listen to music or answer hands-free calls.

[0115] Receiver 270B, also referred to as an "earpiece," is configured to convert an audio electrical signal into an acoustic signal. When a call is answered or voice information is received by using mobile phone 200, receiver 270B may be brought close to a person's ear to listen to the voice.

[0116] The microphone 270C, also referred to as a "mike" or "mic," is configured to convert acoustic signals into electrical signals. When making a phone call or transmitting audio information, a user may place their mouth close to the microphone 270C and speak to input the acoustic signal into the microphone 270C. At least one microphone 270C may be disposed on the mobile phone 200. In some other embodiments, two microphones 270C may be disposed on the mobile phone 200 to capture acoustic signals and further perform noise reduction functions. In some other embodiments, three, four, or more microphones 270C may alternatively be disposed on the mobile phone 200 to capture acoustic signals, reduce noise, further identify sound sources, perform directional recording functions, etc.

[0117] Headset jack 270D is configured to connect to a wired headset.

[0118] The buttons 290 include a power button, a volume button, etc. The mobile phone 200 may receive button inputs and generate button signal inputs related to user settings and function control of the mobile phone 200.

[0119] The motor 291 may generate a vibration prompt. The motor 291 may be configured to provide an incoming vibration prompt and touch vibration feedback.

[0120] The indicator 292 may be an indicator light and may be configured to indicate charging status and power source changes, or may be configured to indicate messages, missed calls, notifications, and the like.

[0121] The SIM card interface 295 is configured to connect to a SIM card, which can be inserted into or removed from the SIM card interface 295 to effect contact with or separation from the mobile phone 200.

[0122] It will be understood that in the embodiments of the present application, the mobile phone 200 may perform some or all of the steps of the embodiments of the present application. These steps or operations are merely examples. The mobile phone 200 may further perform other operations or variations of various operations. In addition, the steps may be performed in an order different from that presented in the embodiments of the present application, and not all operations in the embodiments of the present application need be performed. The embodiments of the present application may be implemented separately or in any combination. This is not a limitation of the present application.

[0123]

[0023] Referring to the foregoing, an embodiment of the present application provides a dynamic range mapping method for panoramic video. As shown in Figure 3, the method includes steps 301 to 305.

[0124] Step 301: An electronic device (i.e., a display end) projects any frame of a 3D panoramic image in a panoramic video onto the surface of a polyhedron to obtain a 2D panoramic projection image, and extends the polyhedron into a 2D plane.

[0125] It should be understood that after obtaining the bitstream of the panoramic video (for example, after receiving the bitstream of the panoramic video from the generating end), the displaying end performs video decoding to obtain the panoramic video. Each frame of the 3D panoramic image of the panoramic video can also be called a 3D spherical representation panorama of the panoramic video.

[0126] Optionally, the polyhedron onto which the three-dimensional panoramic image is projected as the two-dimensional panoramic projection image comprises at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0127] In an embodiment of the present application, a process for projecting a 3D panoramic image onto a 2D panoramic projection image using a polyhedron includes the steps of: placing a sphere capable of representing the 3D panoramic image (the 3D panoramic image is represented by a 3D sphere) on the polyhedron so that the sphere is an inscribed sphere of the polyhedron (it should be understood that the center of the polyhedron coincides with the spherical center of the sphere); connecting the sphere center to any point (pixel point) on the sphere and extending a connecting line to intersect with the surface of the polyhedron, where the point obtained by projecting the sphere point onto the surface of the polyhedron is the intersection point of the extension of the connecting line connecting the sphere center to the sphere point and the surface of the polyhedron; or connecting a point on the surface of the polyhedron to the sphere center, where the connecting line intersects with the sphere at a point, and the intersection point on the sphere and the point on the polyhedron have a mutual projection relationship. It should be understood that the pixel value of the point on the sphere is the pixel value of the intersection point on the surface of the polyhedron. Thus, in the aforementioned projection process, all pixels on the 3D sphere are projected to obtain projected pixels on each surface of the polyhedron (interpolation is performed for projected pixels on the 2D plane based on different cases). Finally, the polyhedron is expanded onto a 2D plane according to certain rules to obtain a 2D panoramic projection image.

[0128] See FIG. 4. An example in which the polyhedron is a regular hexahedron is used, and the regular hexahedron is denoted as ABCDEFGH. As shown in FIG. 4(a), the sphere representing the 3D panoramic image is a sphere inscribed in the regular hexahedron ABCDEFGH. For point M' on plane ABCD on plane A'B'C'D' (plane ABCD is the base of the regular hexahedron ABCDEFGH, and plane A'B'C'D' is the projection plane of plane ABCD), point M' is connected to the sphere center O, and the connecting line intersects with the sphere at point M. Thus, the pixel value of point M' is the pixel value of point M. Similarly, pixel values ​​of all points on plane ABCD can be obtained, and the pixel points on plane ABCD form a surface image of plane ABCD. According to the projection method onto plane ABCD, the pixel points of the 3D panoramic image are projected onto other surfaces of the regular hexahedron to obtain other surface images. As shown in FIG. 4(b), after the 3D panoramic image is projected, a surface image of the top surface EFGH, a surface image of the bottom surface ABCD, a surface image of the front surface HGCD, a surface image of the back surface EFBA, a surface image of the left surface EHDA, and a surface image of the right surface GFBC can be obtained. Furthermore, to obtain the 2D panoramic image shown in FIG. 4(c), FIG. 4(b) is expanded onto a 2D plane. It can be seen that the 2D panoramic projection image has an irregular shape. The surface image of the surface of the polyhedron may be referred to as an image of a region within the 2D panoramic projection image or a sub-image of the 2D panoramic projection image.

[0129] Fig. 5 shows the projection effect when using a regular tetrahedron, a regular hexahedron, a regular octahedron, a regular dodecahedron, and a regular decahedron. After projecting a 3D panoramic image using the regular polyhedrons shown in the first row of Fig. 5, a corresponding 2D panoramic projection image (the projection result shown in the second row of Fig. 5) is obtained.

[0130] In an embodiment of the present application, during subsequent processing, the irregular projection image shown in the second line of FIG. 5 can be processed directly, or the irregular image can be converted into a regular image for processing. In one embodiment, a rectangular region that surrounds the irregular image and has the smallest area can be processed (see the third line of FIG. 5, where the rectangle shown in the dashed box is the rectangle that surrounds the irregular image and has the smallest area). Optionally, other regions within the rectangular region other than the projection region can be filled, for example, with a preset pixel value. In another embodiment, the surface image within the irregular projection image can be stitched into a projection image with a regular shape (e.g., a rectangle). In this way, pixel filling does not need to be performed.

[0131] It should be noted that in the embodiment of the present application, the polyhedron is not limited to a regular polyhedron. In other words, a two-dimensional panoramic projection of a three-dimensional panoramic image may include several polygons of different sizes.

[0132] Optionally, in an embodiment of the present application, the electronic device can use another method to convert a 3D panoramic image into a 2D panoramic projection image. For example, the 3D panoramic image (spherical) is directly expanded. A 3D panoramic image of the Earth is used as an example. The 3D panoramic image of the Earth is expanded into a 2D panoramic image. Specifically, to obtain the content of the N and S pole regions in the 2D panoramic image, the areas adjacent to the N and S poles in the 3D panorama are expanded.

[0133] Step 302: The electronic device divides the 2D panoramic projection image corresponding to any frame of the 3D panoramic image in the panoramic video to obtain multiple regions of the 2D panoramic projection image.

[0134] Each of the plurality of regions satisfies a preset condition, and the preset condition includes at least one of the following condition 1 and condition 2.

[0135] Condition 1: Pixels that are adjacent in the 2D panoramic projection image and not adjacent in the 3D panoramic image are not located in the same region, adjacent pixel points in the 2D panoramic projection image include pixels on at least two adjacent projection planes of the 3D panoramic image, and non-adjacent pixels in the 3D panoramic image include pixels that are in the 3D panoramic image and correspond to at least two non-adjacent projection planes.

[0136] An example will be used in which the polyhedron onto which the 3D panoramic image is projected is a regular hexahedron. As shown in FIG. 6, the six faces of the regular hexahedron are numbered 0, 1, 2, 3, 4, 5, and 6, in the order of front, back, top, bottom, left, and right. In the 2D panoramic projection image obtained by projecting the 3D panoramic image of FIG. 6, the arrows indicated by the edges of the squares in each surface image (square) indicate edges corresponding to edges in the polyhedron. In FIG. 6, edges corresponding to arrows marked with the same letter are the same edge in the polyhedron. The edge corresponding to the arrow marked with the letter a in surface image 0 (which may be referred to as projection surface 0) and the edge corresponding to the arrow marked with the letter a in surface image 3 are the same edge in the regular hexahedron, i.e., the edge where surface 0 intersects with bottom surface 3.

[0137] In Fig. 6, for two adjacent projection planes of a 3D panoramic image, for example, projection plane 0 and projection plane 1, in the 2D panoramic projection image, projection plane 0 is adjacent to projection plane 1, and some pixels of projection plane 0 are adjacent to some pixels of projection plane 1. For example, the pixels on the edge corresponding to the arrow marked with the letter a on projection plane 0 are adjacent to the pixels on the edge corresponding to the arrow marked with the letter g on projection plane 1. The pixels of the 3D panoramic image corresponding to projection plane 0 and the pixels of the 3D panoramic image corresponding to projection plane 1 are not adjacent pixels in the 3D panoramic image.

[0138] When the 2D panoramic projection image obtained in FIG. 6 is divided, it is necessary to ensure that adjacent pixels in the 2D panoramic projection image and non-adjacent pixels in the 3D panoramic image are not located in the same region. For example, in FIG. 6, the pixels on the edge corresponding to the arrow marked with the letter a on projection plane 0 and the pixels on the edge corresponding to the arrow marked with the letter g on projection plane 1 are adjacent pixels in the 2D panoramic projection image. However, the pixels on the edge corresponding to the arrow marked with the letter a on projection plane 0 and the pixels on the edge corresponding to the arrow marked with the letter g on projection plane 1 are not adjacent pixels in the 3D panoramic image. Therefore, it is necessary to ensure that the pixels on the edge corresponding to the arrow marked with the letter a on projection plane 0 and the pixels on the edge corresponding to the arrow marked with the letter g on projection plane 1 are not divided into the same region. In other words, when the 2D panoramic projection image is divided horizontally, the division cannot be performed across the line where projection plane 0 and projection plane 1 intersect; that is, the pixels on both sides of the line cannot be divided into a single region. Similarly, the pixels on both sides of the line where projection plane 4 and projection plane 3 intersect cannot be divided into the same area, and the pixels on both sides of the line where projection plane 5 and projection plane 2 intersect cannot be divided into the same area.

[0139] Condition 2: Non-adjacent pixels in the 3D panoramic image are not located in the same region, and the non-adjacent pixels in the 3D panoramic image include pixels corresponding to at least two projection planes that are not adjacent in the 3D panoramic image.

[0140] See an example of Condition 1. In FIG. 6, the pixel of the 3D panoramic image corresponding to projection plane 0 and the pixel of the 3D panoramic image corresponding to projection plane 1 are non-adjacent pixels in the 3D panoramic image. When a 2D panoramic projection image is divided, it is necessary to ensure that the pixels of projection plane 0 and the pixels of projection plane 1 are not divided into the same region. For example, the pixels on the edge corresponding to the arrow marked with the letter "a" on projection plane 0 and the pixels on the edge corresponding to the arrow marked with the letter "g" on projection plane 1 are not divided into the same region. In other words, when a 2D panoramic projection image is divided horizontally, the division cannot be performed across the line where projection plane 0 and projection plane 1 intersect; that is, the pixels on both sides of the line cannot be divided into a single region. Similarly, the pixels on both sides of the line where projection plane 4 and projection plane 3 intersect cannot be divided into the same region, and the pixels on both sides of the line where projection plane 5 and projection plane 2 intersect cannot be divided into the same region.

[0141] In the embodiment of the present application, in the process of dividing a 2D panoramic projection image, when each region obtained by division meets a preset condition, it can ensure that pixels with similar brightness (similar brightness indicates similar dynamic range) are divided into the same region, and pixels with large brightness differences (large brightness difference indicates large dynamic range difference) are not divided into the same region. For example, in the 3D panoramic projection image of Figure 6, the pixel corresponding to projection plane 2 can be an empty pixel, and the pixel corresponding to projection plane 2 can be an empty pixel. 5The pixels corresponding to may be ground pixels. Because the brightness of sky pixels is significantly different from that of ground pixels, the pixels on projection plane 2 and the pixels on projection plane 5 are not located in the same region during the process of dividing the 2D panoramic projection image. Because the dynamic ranges of pixels in the same region are similar, dynamic range mapping can be performed for pixels in the same region based on the same dynamic range mapping information. Because the dynamic ranges of pixels in different regions may be significantly different, dynamic range mapping can be performed for pixels in different regions based on different dynamic range mapping information. Therefore, after the 2D panoramic projection image is divided into multiple regions using this division method, subsequent processing (i.e., dynamic range mapping) is performed for each region. The characteristics of the pixels in each region can be adaptively adapted to the process to improve the processing effect.

[0142] Optionally, the electronic device may divide the two-dimensional panoramic projection image using different division methods. In the embodiment of the present application, the electronic device may divide the two-dimensional panoramic projection image by using any of step 3021, step 3022, or step 3023 and step 3024.

[0143] Referring to FIG. 3, as shown in FIG. 7, the method for dividing a two-dimensional panoramic projection image provided in an embodiment of the present application particularly includes step 3021 (i.e., step 302 is implemented using step 3021).

[0144] Step 3021: The electronic device divides the two-dimensional panoramic projection image using a first division method to obtain multiple regions of the two-dimensional panoramic projection image, where the first division method is a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions.

[0145] The regular hexahedron shown in FIG. 6 is still used as an example. See FIG. 8. After the 3D panoramic projection image is projected using the regular hexahedron, the resulting 2D panoramic projection image includes six projection planes. The first division method is a method of dividing pixels belonging to the same surface of the hexahedron into regions in the 2D panoramic projection image, i.e., each projection plane of the 2D panoramic projection image is used as a region. As shown in FIG. 8, the 2D panoramic projection image may be divided into six regions, i.e., Region A, Region B, Region C, Region D, Region E, and Region F in FIG. 8, using the first division method. See FIG. 8. Each of the multiple regions obtained by dividing the 2D panoramic projection image using the first division method satisfies a preset condition (including Condition 1 or Condition 2), i.e., pixels that are adjacent in the 2D panoramic projection image but not adjacent in the 3D panoramic image are not located in the same region.

[0146] Referring to FIG. 3, as shown in FIG. 9, another method for dividing a two-dimensional panoramic projection image provided in an embodiment of the present application specifically includes step 3022 (i.e., step 302 is implemented using step 3022).

[0147] Step 3022: The electronic device divides the two-dimensional panoramic projection image using a second division method to obtain multiple regions of the two-dimensional panoramic projection image, the second division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron (used for projection) into regions to obtain multiple intermediate regions, and further dividing at least one of the multiple intermediate regions.

[0148] The regular hexahedron shown in FIG. 6 is still used as an example. See FIG. 10. After the 3D panoramic projection image is projected using the regular hexahedron, the resulting 2D panoramic projection image includes six projection planes. The second division method divides pixels in the 2D panoramic projection image belonging to the same surface of the polyhedron (used for projection) into regions to obtain multiple intermediate regions, and further divides at least one of the multiple intermediate regions. In other words, the 2D panoramic projection image is first divided using the first division method to obtain multiple regions as the multiple intermediate regions, and then one or more of the multiple intermediate regions are divided to obtain multiple regions as the final division result. As shown in FIG. 10, after the 2D panoramic projection image is divided using the first division method, six intermediate regions, namely intermediate region a, intermediate region b, intermediate region c, intermediate region d, intermediate region e, and intermediate region f, are obtained. Then, each of the six intermediate regions is divided horizontally in half to finally obtain 12 regions. In other words, the 2D panoramic projection image can be divided into 12 regions, namely, region A, region B, region C, region D, region E, region F, region G, region H, region I, region J, region K, and region L in Fig. 10, using the second division scheme. See Fig. 10. Each of the multiple regions obtained by dividing the 2D panoramic projection image using the second division scheme satisfies a preset condition (including condition 1 or condition 2), i.e., pixels that are adjacent in the 2D panoramic projection image and not adjacent in the 3D panoramic image are not located in the same region.

[0149] Referring to FIG. 3, as shown in FIG. 11, yet another method for dividing a two-dimensional panoramic projection image provided in an embodiment of the present application particularly includes step 3023 and step 3024 (i.e., step 302 is implemented using step 3023 and step 3024).

[0150] Step 3023: The electronic device divides the two-dimensional panoramic projection image using a third division method to obtain multiple intermediate regions of the two-dimensional panoramic projection image, where the third division method is a method of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region.

[0151] In an embodiment of the present application, the process of clustering all pixels in a two-dimensional panoramic projection image in a third division manner includes:

[0152] A clustering center is first determined, and the clustering center is one pixel. Optionally, the 2D panoramic projection image may be first divided into multiple regions using a preset division method, and the pixel value at the center of each region is determined, and the pixel value at the center of each region is used as the clustering center. For example, assume that the 2D panoramic projection image is an image obtained by projection using a regular hexahedron as shown in FIG. 6. If the 2D panoramic projection image is divided into six regions, the 2D panoramic projection image may be divided into six regions using a first division method, and the pixel value at the center of each region is determined to obtain six clustering centers. If the 2D panoramic projection image is divided into 12 regions, the 2D panoramic projection image may be divided into 12 regions using a second division method, and the pixel value at the center of each region is determined to obtain 12 clustering centers.

[0153] Then, according to the preset clustering algorithm and the determined clustering centers, a calculation is performed for the distance between every pixel point and each clustering center (the distance is calculated based on the coordinates of the pixel and the coordinates of the pixel corresponding to the clustering center), and the distance is D iThen, a calculation is performed on the difference between the luminance values ​​of all pixel points and the luminance values ​​of pixel points corresponding to the clustering centers (a calculation is performed on the difference between the luminance values ​​of all pixel points and the average luminance value of the pixel points in the category corresponding to each clustering center (i.e., the category obtained by dividing using a preset dividing method), a calculation is performed on the difference between the color values ​​of all pixel points and the color values ​​of pixel points corresponding to the clustering centers, or a calculation is performed on the difference between the color values ​​of all pixel points and the average color value of the pixel points in the category corresponding to each clustering center (i.e., the category obtained by dividing using a preset dividing method)), and the difference is E i Each pixel is denoted by a distance D i and E i According to the weighted values ​​of the weights, the pixel is divided into categories corresponding to the corresponding clustering centers to obtain the clustering result. The category with the smallest weighted value is the category corresponding to the pixel, and the pixel points of each category in the clustering result form a region.

[0154] It should be noted that dividing the 2D panoramic projection image according to the aforementioned clustering method in step 3023 cannot guarantee whether the obtained intermediate regions satisfy the preset conditions (including condition 1 or condition 2). Therefore, the intermediate regions obtained by clustering need to be determined to determine whether the intermediate regions satisfy the preset conditions. If all of the intermediate regions satisfy the preset conditions, the intermediate regions are used as the final division results, i.e., the intermediate regions are used as the regions obtained by dividing the 2D panoramic projection image. If there is an intermediate region among the intermediate regions that does not satisfy the preset conditions, the electronic device continues to perform step 3024.

[0155] Step 3024: To obtain a plurality of regions, the electronic device divides any intermediate region among the plurality of intermediate regions that does not satisfy the preset condition using a fourth division scheme. For any intermediate region that does not satisfy the preset condition, the fourth division scheme is a scheme of dividing the intermediate region along the boundary lines of at least two surfaces of the polyhedron included in the intermediate region.

[0156] We will continue to use an example in which the 2D panoramic projection image is an image obtained by projection onto a regular hexahedron as shown in FIG. 6 . See FIG. 12 . Assume that the number of categories set during clustering is seven. After all pixels of the 2D panoramic projection image are clustered, seven intermediate regions are obtained: intermediate region A, intermediate region B, intermediate region C, intermediate region D, intermediate region E, intermediate region F, and intermediate region G. Because intermediate region E does not satisfy the preset condition (the other intermediate regions do), it needs to be further divided. As shown in FIG. 12 , intermediate region E includes some pixels of two projection planes (projection plane 0 and projection plane 1), and it can be seen that intermediate region E may be divided into region I and region H along the line where projection plane 0 and projection plane 1 intersect in intermediate region E. In this way, the 2D panoramic projection image is finally divided into eight regions: region A, region B, region C, region I, region D, region H, region F, and region G.

[0157] In the embodiments of the present application, based on the actual requirements for dividing a two-dimensional panoramic projection image, an appropriate division method may be selected from the three division methods mentioned above, or another division method other than the three division methods mentioned above may be used to divide a two-dimensional panoramic projection image, provided that the division method can ensure that each region obtained by division meets the preset conditions.

[0158] Optionally, in an embodiment of the present application, the division scheme of the two-dimensional panoramic projection image may be preset in the electronic device (e.g., agreed upon according to a protocol), such as being set to a first division scheme, a second division scheme, or a third division scheme, or the division scheme of the two-dimensional panoramic projection image is transferred in a bitstream (specifically, metadata).

[0159] Referring to the description of the division method of the 2D panoramic projection image in the above embodiment, it can be seen that the first division method is related to the second division method, and the division result of the second division method is the subdivision result based on the first division method.

[0160] In one embodiment, when the candidate division schemes of the 2D panoramic projection image include a first division scheme and a second division scheme, whether further division needs to be performed after division using the first division scheme can be determined based on the content of the 2D panoramic projection image, i.e., whether to use the first division scheme or the second division scheme. Specifically, the following process is performed to determine whether further division needs to be performed on the division result of the first division scheme:

[0161] S1: Divide the two-dimensional panoramic projection image using the first division method to obtain multiple intermediate regions.

[0162] S2 through S5 are performed for each of the multiple intermediate regions to determine whether each intermediate region needs to be further divided.

[0163] S2: Calculate the histogram of the mid-region.

[0164] For example, the histogram of the middle region is denoted as HISA. In the embodiment of the present application, the histogram may be a gray scale histogram or a color histogram, which is not limited in the embodiment of the present application.

[0165] S3: Divide the intermediate region into multiple regions and calculate the histogram of each region.

[0166] Optionally, one intermediate region may be equally divided into four regions, each occupying one-fourth of the area of ​​the intermediate region. The four regions obtained by the division are denoted as Region 0, Region 1, Region 2, and Region 3, respectively, and the histograms of the four regions are denoted as HISB[0], HISB[1], HISB[2], and HISB[3], respectively.

[0167] S4: The sum of the difference values ​​between the histogram of the intermediate region and the histogram of the region obtained by dividing the intermediate region is calculated.

[0168] The difference values ​​between the histogram of the intermediate region and the histograms of each region obtained by division are first calculated, and the difference values ​​are the differences between the histogram of the intermediate region multiplied by n and the corresponding elements of the histogram of each region. All differences are summed up, where n is the number of regions obtained by dividing the intermediate region. For example, if the intermediate region is divided into four regions, n=4. Then, the difference values ​​are summed up.

[0169] The sum of the difference values ​​is denoted as DiffAB, DiffAB=DiffAB[0]+DiffAB[1]+DiffAB[2]+DiffAB[3] is.

[0170] DiffAB[0] indicates the difference value between the histogram of the intermediate region and the histogram of region 0, DiffAB[1] indicates the difference value between the histogram of the intermediate region and the histogram of region 1, DiffAB[2] indicates the difference value between the histogram of the intermediate region and the histogram of region 2, and DiffAB[3] indicates the difference value between the histogram of the intermediate region and the histogram of region 3.

[0171] S5: If DiffAB>T1, where T1 is a preset threshold, the intermediate region needs to be further divided, i.e., the two-dimensional panoramic projection image is divided by the second division method; or if DiffAB≦T1, the intermediate region does not need to be further divided, i.e., the two-dimensional panoramic projection image is divided by the first division method.

[0172] DiffAB>T 1 If , it indicates that the luminance difference between parts of the intermediate region is large and further segmentation needs to be performed.

[0173] Since the first division scheme is related to the second division scheme, in some cases, when the default division scheme is the first division scheme, a related identifier may be used in the bitstream to indicate whether further division is to be performed. When the identifier indicates that further division is not to be performed, the division result obtained by the first division scheme is the final result. When the identifier indicates that further division needs to be performed, the division result of the first division scheme is further divided.

[0174] In another embodiment, when the candidate division schemes for a 2D panoramic projection image include a first division scheme, a second division scheme, and a third division scheme, which division scheme to use may be determined by performing the following process.

[0175] S11: Divide the two-dimensional panoramic projection image into a first division method, a second division method, and a third division method.

[0176] S12: A histogram of each of the regions obtained by the first division method, the second division method, and the third division method, and a histogram of the two-dimensional panoramic projection image are calculated separately.

[0177] S13: The sum of the difference values ​​between the histograms of the multiple regions obtained by each division method and the histogram of the two-dimensional panoramic projection image is calculated.

[0178] For example, a 2D panoramic projection image is divided into M regions using a first division method, and the sum of the difference values ​​between the histograms of the multiple regions obtained by the first division method and the histogram of the 2D panoramic projection image is denoted as DiffHISM. The calculation of the sum of the difference values ​​is similar to the process in S4. The difference value between the histogram of each region and the histogram of the 2D panoramic projection image is first calculated, and the difference value is the difference between the histogram obtained by multiplying the histogram of each region by M and the corresponding element of the histogram of the 2D panoramic projection image. All the differences are summed up as follows: DiffHISM=DiffHISM[0]+DiffHISM[1]+…+DiffHISM[M]

[0179] DiffHISM[0] indicates the difference between the histogram of region 0 obtained by the first division method and the histogram of the 2D panoramic projection image, DiffHISM[M] indicates the difference between the histogram of region M obtained by the first division method and the histogram of the 2D panoramic projection image, and the remainder can be estimated by analogy.

[0180] The 2D panoramic projection image is divided into N regions using a second division method, and the sum of the difference values ​​between the histograms of the multiple regions obtained by the second division method and the histogram of the 2D panoramic projection image is denoted as DiffHISN. The calculation of the sum of the difference values ​​is the same as the process in S4. The difference value between the histogram of each region and the histogram of the 2D panoramic projection image is first calculated, and the difference value is the difference between the histogram obtained by multiplying the histogram of each region by N and the corresponding element of the histogram of the 2D panoramic projection image. All differences are summed up as follows: DiffHISN=DiffHISN[0]+DiffHISN[1]+…+DiffHISN[N]

[0181] DiffHISN[0] indicates the difference between the histogram of region 0 obtained by the second division method and the histogram of the 2D panoramic projection image, DiffHISN[N] indicates the difference between the histogram of region N obtained by the second division method and the histogram of the 2D panoramic projection image, and the remainder can be estimated by analogy.

[0182] The 2D panoramic projection image is divided into P regions using a third division method, and the sum of the difference values ​​between the histograms of the multiple regions obtained by the third division method and the histogram of the 2D panoramic projection image is denoted as DiffHISP. The calculation of the sum of the difference values ​​is the same as the process in S4. The difference value between the histogram of each region and the histogram of the 2D panoramic projection image is first calculated, and the difference value is the difference between the histogram obtained by multiplying the histogram of each region by P and the corresponding element of the histogram of the 2D panoramic projection image. All differences are summed up as follows: DiffHISP=DiffHISP[0]+DiffHISP[1]+…+DiffHISP[P]

[0183] DiffHIS P [0] indicates the difference value between the histogram of area 0 obtained by the third division method and the histogram of the 2D panoramic projection image, DiffHISP[P] indicates the difference value between the histogram of area P obtained by the third division method and the histogram of the 2D panoramic projection image, and the remainder can be estimated by analogy.

[0184] S14: The division method that maximizes the sum of the difference values ​​between the histograms is determined as the final division method.

[0185] The sums of the difference values ​​between the histograms, DiffHISM, DiffHISN, and DiffHISP, are compared, and the larger the sum of the difference values ​​between the histograms, the larger the luminance difference between the regions obtained by the division, indicating that the division is appropriate.

[0186] Optionally, the sizes of the metadata for the three division schemes may be further calculated, and the metadata includes dynamic mapping information corresponding to each region of the 2D panoramic projection image (details will be described in step 303). Then, which division scheme to use is determined based on a weighted value between the sum of the histogram difference values ​​corresponding to the division schemes described in S13 and the sizes of the metadata corresponding to the division schemes. For example, the size of the metadata corresponding to the first division scheme is denoted as Size1, the size of the metadata corresponding to the second division scheme is denoted as Size2, and the size of the metadata corresponding to the third division scheme is denoted as Size3. Calculations are performed for DiffHISM-Size1 (the difference is denoted as err1), DiffHISN-Size2 (the difference is denoted as err2), and DiffHISP[P]-Size3 (the difference is denoted as err3), and err1, err2, and err3 are compared to determine the division scheme with the largest difference as the final division scheme.

[0187] Step 303: The electronic device acquires metadata of the 2D panoramic projection image, where the metadata includes metadata information elements respectively corresponding to multiple regions of the 2D panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information of the region.

[0188] It should be understood that in the embodiments of the present application, the concept of performing dynamic range mapping on a panoramic video is to project all 3D panoramic images included in the panoramic video onto a 2D plane, perform dynamic range mapping on the obtained 2D panoramic projection images, and convert the 2D panoramic projection images obtained by dynamic range mapping into 3D panoramic images to obtain a panoramic video obtained by dynamic range mapping. In other words, the metadata of the obtained 2D panoramic projection images is used to perform dynamic range mapping of the 3D panoramic image (i.e., used to perform dynamic range mapping on the panoramic video), and the metadata of the 2D panoramic projection images may be considered as the metadata of the 3D panoramic image (or the metadata of the panoramic video). The metadata referred to in the following embodiments is the data used for dynamic range mapping.

[0189] Optionally, the method for the electronic device to obtain the metadata of the 2D panoramic projection image may be a step (end of generation) in which the electronic device receives the metadata of the 2D panoramic projection image from another electronic device. After generating the metadata, the generating end encodes the metadata into a bitstream and sends the bitstream to the display end, so that the display end receives the bitstream and analyzes the bitstream to obtain the metadata, as can be seen from the description of the above embodiment.

[0190] In the embodiment of the present application, the two-dimensional panoramic projection image is divided into multiple regions, that is, the two-dimensional panoramic projection image includes multiple regions, and each region corresponds to one metadata information element.

[0191] In an embodiment of the present application, the metadata information element includes dynamic mapping information, which includes dynamic mapping parameters, and dynamic range mapping can be performed on pixels based on the dynamic mapping parameters. Optionally, the format of the dynamic mapping information may be histogram information or dynamic mapping curve information of the ST2094-40 standard, or dynamic mapping curve information of the ST2094-10 standard. This is not limited in the embodiment of the present application.

[0192] It can be seen from the description of the foregoing embodiments that the division scheme of the 2D panoramic projection image can be transmitted in a bitstream. Optionally, when the bitstream is used to transmit the division scheme of the 2D panoramic projection image, the metadata of the 2D panoramic projection image further includes division scheme indication information, and the division scheme indication information indicates that the division scheme of the 2D panoramic projection image is a first division scheme, a second division scheme, or a third division scheme.

[0193] Step 304: The electronic device determines dynamic mapping information for each pixel within the current field of view range based on the dynamic mapping information for each of the multiple regions.

[0194] It should be understood that when a user watches a panoramic video, the user (head) cannot see all the content of the panoramic video in the current pose, but can only see the content of the video in a certain range. The range can be understood as the user's field of view range. It can be seen that the user's field of view range is related to the user's current viewing pose. In an embodiment of the present application, performing dynamic range mapping on a panoramic video is performing dynamic range mapping on the content within the field of view range of the user who is currently watching the panoramic video.

[0195] In an embodiment of the present application, after obtaining the dynamic mapping information of the plurality of regions, the electronic device may determine the dynamic mapping information of each pixel within the current angle of view range based on the dynamic mapping information of the plurality of regions in one of four ways. Specifically, for one of the pixels, several ways of determining the dynamic mapping information of the pixel are described in detail.

[0196] Method 1: The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a plurality of regions within a first preset range, and the first preset range is a range centered on the pixel to be processed.

[0197] For example, it is assumed that the first preset range is a 3-pixel by 3-pixel range centered on the pixel to be processed, the first preset range covers pixels in three regions (i.e., the pixels in the first preset range belong to three regions), and the dynamic mapping information of all the three regions is used as the dynamic mapping information of the pixel to be processed.

[0198] Method 2: The dynamic mapping information of the pixel to be processed within the current angle of view range is the dynamic mapping information of the region to which the pixel belongs in the 2D panoramic projection image, which corresponds to the viewpoint center of the current angle of view range.

[0199] It should be understood that when a user watches a panoramic video, the user's current field of view range corresponds to the viewpoint center. Assuming that the viewpoint center is X and the pixel corresponding to the viewpoint center X in the 2D projection image is X', the dynamic mapping information of the region to which X' belongs is used as the dynamic mapping information of the target pixel.

[0200] Method 3: The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of multiple areas within a second preset range, and the second preset range is a range centered on a pixel in the 2D panoramic projection image that corresponds to the viewpoint center of the current angle of view range.

[0201] Referring to Method 2, the example where the viewpoint center of the current view angle range is X is still used. The pixel corresponding to the viewpoint center X in the 2D projection image is X'. For example, the second preset range may be a 3 pixel x 3 pixel range centered at X'. The second preset range covers pixels in two regions (i.e., the first 2 (The pixels in the preset range of belong to two regions), and it is assumed that all the dynamic mapping information of the two regions is used as the dynamic mapping information of the pixel to be processed.

[0202] Method 4: The dynamic mapping information of one pixel in the current region is the dynamic mapping information of the current region.

[0203] In other words, the dynamic mapping information of the region to which the pixel to be processed belongs is used as the dynamic mapping information of the pixel to be processed.

[0204] Optionally, during actual application, the generating end may select an appropriate method from the four methods based on the actual requirements for determining dynamic mapping information of the pixel to be processed, and transfer the selected method carried in the bitstream (metadata) to the display end. Alternatively, the generating end and the display end may predetermine the method to be used. This is not limited in the embodiments of the present application.

[0205] Step 305: The electronic device performs dynamic range mapping for each pixel within the current angle of view based on the dynamic mapping information of each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, and the two-dimensional panoramic projection image within the current angle of view is used for display or subsequent processing.

[0206] Optionally, the dynamic mapping information in the embodiments of the present application is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range, or the dynamic mapping information is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a second dynamic range to the first dynamic range, where the first dynamic range is greater than the second dynamic range. In other words, the dynamic range mapping performed on the pixels may be a mapping from a high dynamic range to a low dynamic range or a mapping from a low dynamic range to a high dynamic range, and may include, for example, some cases shown in Table 1 of the above embodiment.

[0207] For example, the dynamic mapping information is a dynamic mapping curve. In the embodiment of the present application, the dynamic mapping curve may include, but is not limited to, a sigmoidal curve, a cubic spline curve, a Bezier curve, etc.

[0208] A sigmoidal curve is used as an example: the curve is L'=F(L), where L denotes the pixel before dynamic range mapping and L' denotes the pixel after dynamic range mapping.

[0209] The formula for F(L) is

number

[0210] Optionally, L and L' are normalized values. The normalization may be linear space normalization or nonlinear space normalization. In addition, the range of normalization is not limited in the embodiments of the present application. For example, the range of normalization may be 0 to 10,000 nits or 0.001 to 100,000 nits.

[0211] Optionally, after dynamic range mapping is performed on each pixel within the current angle of view, the 2D panoramic projection image may be displayed to obtain a 2D panoramic projection image within the current angle of view. In some cases, inverse normalization may be performed on the pixels (L') of the 2D panoramic projection image based on the display capabilities (maximum and minimum display values) of the display device to normalize the pixels to the range of the display capabilities of the display device.

[0212] In an embodiment of the present application, after the dynamic mapping information of each pixel within the current angle of view range is determined using step 305, the pixel may correspond to multiple pieces of dynamic mapping information (e.g., multiple dynamic mapping curves). After dynamic range mapping is performed on the pixel in step 305, one dynamic mapping result may be obtained for each piece of dynamic mapping information. In this case, the multiple dynamic mapping results may be processed to obtain a final dynamic mapping result. The processing manner may include any one of the following:

[0213] Processing method 1: The median value of multiple dynamic mapping results is used as the final dynamic mapping result.

[0214] Processing method 2: A weighted average is performed on multiple dynamic mapping results to obtain the final dynamic mapping result.

[0215] Assume that the pixel to be processed includes n pieces of dynamic mapping information. n dynamic mapping results Result_1, Result_2, ..., and Result_n may be obtained, and the weighting coefficients corresponding to the n dynamic mapping results are W1, W2, ..., and W n and the final mapping result R is as follows:

number

[0216] In an embodiment of the present application, when multiple dynamic mapping results are processed by processing method 2, the weighting coefficient corresponding to each dynamic mapping result may be a preset weighting coefficient, or may be a weighting coefficient determined based on the current pixel to be processed.

[0217] Determining the weighting coefficient based on the current pixel to be processed specifically includes calculating the distance between the pixel to be processed and the pixel at the center of the region corresponding to each dynamic mapping information, and calculating the weighting coefficient based on each distance. For example, the current pixel to be processed is Y, and the dynamic mapping information for Y includes dynamic mapping information corresponding to three regions, with the pixels at the center of the three regions being Y1', Y2', and Y3', respectively. Calculations are performed on the distance D1 between Y and Y1', the distance D2 between Y and Y2', and the distance D3 between Y and Y3', and the weighting coefficient is determined based on D1, D2, and D3. Optionally, in an embodiment of the present application, the smaller the distance between the pixel to be processed and the pixel at the center of the region corresponding to the dynamic mapping information, the larger the corresponding weighting coefficient.

[0218] For example, the method for determining the weighting factors W1, W2, and W3 based on D1, D2, and D3 is as follows.

number

[0219] Optionally, referring to FIG. 3, as shown in FIG. 13, the current Angle of view range Before the dynamic mapping information of each pixel in the plurality of regions is determined based on the dynamic mapping information of each of the plurality of regions (i.e., before step 304), the dynamic range mapping method for panoramic video provided in the embodiment of the present application further includes step 306 and step 307.

[0220] Step 306: The electronic device obtains the correspondence between the region and the metadata information element.

[0221] Step 307: The electronic device determines metadata information elements corresponding to the multiple regions separately from the metadata according to the correspondence between the regions and the metadata information elements, so as to determine dynamic mapping information for each of the multiple regions.

[0222] Optionally, the correspondence between the regions and the metadata information elements may be a one-to-one correspondence or a many-to-one correspondence. In other words, one region of the 2D panoramic projection image corresponds to one metadata information element, or multiple regions of the 2D panoramic projection image correspond to one metadata information element. Optionally, when the brightness difference between multiple regions of the 2D panoramic projection image is large and one type of dynamic mapping information is used for each region, the correspondence between the regions and the metadata information elements is a one-to-one correspondence. When the brightness of some regions in the multiple regions is similar, some regions may use the same type of dynamic mapping information, and the correspondence between the regions and the metadata information elements is a many-to-one correspondence.

[0223] In an embodiment of the present application, the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and an index of the region, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata element and the coordinates of one or more pixels within the region.

[0224] For example, assume that a two-dimensional panoramic projection image is divided into six regions (see FIG. 6). Table 2 shows an example of the correspondence between the indexes of metadata information elements and the indexes of regions.

[0225] [Table 2]

[0226] It can be seen from Table 2 that the correspondence between the indexes of the metadata information elements and the indexes of the regions is essentially a traversal order relationship. See Figure 6. The order of the six regions in the 2D panoramic projection image is, from left to right and top to bottom, Region 4, Region 0, Region 5, Region 3, Region 1, and Region 2. Therefore, in the table of correspondence between regions and metadata information elements shown in Table 2, the six regions in the 2D panoramic projection image are traversed sequentially from left to right and top to bottom, and the metadata information elements corresponding to the six regions separately can be determined sequentially in the metadata.

[0227] In one embodiment, the traversal order of the regions may be indicated in the bitstream. For example, metadata may carry an indication of the traversal order. The traversal order is not limited to the above-described left-to-right and top-to-bottom order; the traversal order is specifically related to the location of each region within the 2D panoramic projection image.

[0228] In another example, assume that a two-dimensional panoramic projection image is divided into six regions. Table 3 shows an example of the correspondence between the index of a metadata information element and the coordinates of one or more pixels within a region.

[0229] [Table 3]

[0230] Referring to Table 3, in one embodiment, the coordinates of pixels in the regions in Table 3 may also be carried in the metadata. Specifically, the coordinates of pixels in each region are carried in a metadata information element corresponding to the region. After dividing the 2D panoramic projection image to obtain multiple regions, for one of the regions, the display device determines the pixels whose coordinates belong to the region according to the correspondence shown in Table 3, and uses the metadata information element corresponding to the coordinate as the metadata information element for the region. For example, for region 1 of the six regions, the relationship between the pixels corresponding to each coordinate in Table 3 and region 1 is sequentially determined. It is determined that the pixel corresponding to the coordinates (x4, y4) belongs to region 1. Therefore, the metadata information element whose index is 4 and corresponds to the coordinates (x4, y4) is used as the metadata information element for region 1.

[0231] Optionally, in an embodiment of the present application, the traversal order of multiple regions may be a preset order. Alternatively, the generating end determines a traversal order from the candidate traversal orders according to a related algorithm, adds an identifier to the metadata of the bitstream, and forwards the bitstream to the generating end to indicate the traversal order. For example, which traversal order is used may be determined using the following method:

[0232] S111: A histogram for each of the multiple regions is calculated.

[0233] S112: For each candidate traversal order, calculate the sum of the difference values ​​between the histograms of adjacent regions.

[0234] Assume there are four candidate traversal orders: traversal order Z1, traversal order Z2, traversal order Z3, and traversal order Z4.

[0235] An example will still be used in which a 2D panoramic projection image is divided into six regions. For a candidate traversal order (denoted as Z1) from left to right and top to bottom, assume the six regions are, in order, Region 4, Region 0, Region 5, Region 3, Region 1, and Region 2. Calculations are performed on the difference between the histogram of Region 4 and the histogram of Region 0 (denoted as DiffHIS_1), the difference between the histogram of Region 0 and the histogram of Region 5 (denoted as DiffHIS_2), the difference between the histogram of Region 5 and the histogram of Region 3 (denoted as DiffHIS_3), the difference between the histogram of Region 3 and the histogram of Region 1 (denoted as DiffHIS_4), and the difference between the histogram of Region 1 and the histogram of Region 2 (denoted as DiffHIS_5). The sum of the differences is calculated to obtain DiffHIS_Z1. Similarly, DiffHIS_Z2, DiffHIS_Z3, and DiffHIS_Z4 can be obtained by calculation.

[0236] S113: In the candidate traversal order, the traversal that minimizes the sum of the differences between histograms order The final traversal order It is determined as follows.

[0237] For example, the values ​​of DiffHIS_Z1, DiffHIS_Z2, DiffHIS_Z3, and DiffHIS_Z4 are compared, and the traversal order that minimizes the sum of the differences between the histograms is the final traversal order. order is determined as follows.

[0238] Optionally, in the embodiment of the present application, a preset algorithm may be used to determine whether the correspondence between multiple regions in a 2D panoramic image and metadata information elements is a one-to-one relationship or a many-to-one relationship. For example, whether to use a one-to-one relationship or a many-to-one relationship may be determined using the following method:

[0239] S1111: Cluster multiple regions.

[0240] The number of clustered categories may be set based on requirements, for example, half the number of regions or a quarter of the number of regions. For example, if the total number of regions is 8, the 8 regions may be divided into 2 categories, which corresponds to obtaining 2 large regions, or the 8 regions may be divided into 4 categories, which corresponds to obtaining 4 large regions.

[0241] S1112: The sum of the differences between the histogram of the region corresponding to the category and the histogram of the 2D panoramic projection image is calculated, and the sum of the differences between the histograms of the multiple original regions and the histogram of the 2D panoramic image is calculated.

[0242] For example, in the clustering results, the sum of the differences between the histogram of the region corresponding to each category and the histogram of the 2D panoramic projection image is denoted as DiffHIS_X, and the sum of the differences between the histograms of multiple original regions and the histogram of the 2D panoramic image is denoted as DiffHIS_org.

[0243] S1113: Determine whether the correspondence between the multiple regions and the metadata information elements is a one-to-one relationship or a many-to-one relationship based on two sums of the differences between the histograms.

[0244] For example, when eight regions are clustered to obtain two categories of regions, the sum of the differences between the histograms of the regions corresponding to each category of the clustering result and the histogram of the 2D panoramic projection image is denoted as DiffHIS_X2, and the sum of the differences between the histograms of multiple original regions and the histogram of the 2D panoramic image is denoted as DiffHIS_org. The determination method is as follows: if DiffHIS_org×(1-T2)>DiffHIS_X2, the correspondence between the regions and the metadata information elements is a many-to-one relationship, or if DiffHIS_org×(1-T2)≦DiffHIS_X2, the correspondence between the regions and the metadata information elements is a one-to-one relationship, where T2 is a preset threshold.

[0245] Optionally, referring to FIG. 13, after step 305, the dynamic range mapping method for panoramic video provided in the embodiment of the present application further includes step 308, as shown in FIG.

[0246] Step 308: The electronic device performs subsequent processing on the 2D panoramic projection image within the current angle of view range, the subsequent processing including performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range, and the 3D panoramic image is used for display.

[0247] In an embodiment of the present application, the 2D panoramic projection image obtained in step 305 can be used for display. Furthermore, the 2D panoramic projection image can be further used for subsequent processing, for example, the 2D panoramic projection image can be converted into a 3D panoramic image for display. Furthermore, after all the 3D panoramic images of the panoramic video are obtained, the panoramic video can be played.

[0248] In conclusion, an embodiment of the present application provides a dynamic range mapping method for panoramic video. An electronic device divides a 2D panoramic projection image corresponding to a 3D panoramic image to obtain multiple regions of the 2D panoramic projection image. Each of the multiple regions satisfies a preset condition, including at least one of the following: pixels that are adjacent in the 2D panoramic projection image but not adjacent in the 3D panoramic image are not located in the same region; and pixels that are not adjacent in the 3D panoramic image are not located in the same region. In addition, the electronic device obtains metadata for the 2D panoramic projection image, the metadata including metadata information elements corresponding to the multiple regions of the 2D panoramic projection image, and each metadata information element corresponding to a region includes dynamic mapping information for that region. The electronic device then determines dynamic mapping information for each pixel within a current angle of view based on the dynamic mapping information for each of the multiple regions, and performs dynamic range mapping for each pixel within the current angle of view (the angle of view of the user currently viewing the panoramic video) based on the dynamic mapping information for each pixel within the current angle of view to obtain a 2D panoramic projection image within the current angle of view. The 2D panoramic projection image within the current angle of view is used for display or subsequent processing. In the dynamic range mapping method for panoramic video provided in the embodiment of the present application, in relation to the large dynamic range of the panoramic video, the electronic device divides a 2D panoramic projection image corresponding to a 3D panoramic image to obtain regions, and performs dynamic range mapping for different regions, thereby effectively implementing dynamic range mapping on the panoramic video and thereby improving the display effect of the panoramic video.

[0249] Correspondingly, an embodiment of the present application provides a dynamic range mapping apparatus for panoramic video. The apparatus may be used in an electronic device. In the embodiment of the present application, the apparatus may be divided into functional modules based on the above-mentioned method example. For example, functional modules corresponding to functions may be obtained by division, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. application It should be noted that in this embodiment, the division into modules is an example and is merely a division of logical functions. In actual implementation, other division methods may be used.

[0250] When each functional module is obtained by division based on each corresponding function, FIG. 15 is a diagram of a possible structure of a dynamic range mapping device for panoramic video in the aforementioned embodiment. As shown in FIG. 15, the device includes a division module 1501, an acquisition module 1502, a determination module 1503, and a processing module 1504. The division module 1501 is configured to divide a 2D panoramic projection image to obtain multiple regions of the 2D panoramic projection image, for example, to perform step 302 in the method embodiment. The 2D panoramic projection image is a 2D panoramic projection image corresponding to any frame of a 3D panoramic image in the panoramic video, and each of the multiple regions satisfies a preset condition, which includes at least one of the following conditions: adjacent pixels in the 2D panoramic projection image and non-adjacent pixels in the 3D panoramic image are not located in the same region; and non-adjacent pixels in the 3D panoramic image are not located in the same region. The adjacent pixel points in the two-dimensional panoramic projection image include pixels on at least two adjacent projection planes of the three-dimensional panoramic image, and the non-adjacent pixels in the three-dimensional panoramic image include pixels in the three-dimensional panoramic image that correspond to at least two non-adjacent projection planes. The acquisition module 1502 acquires metadata of the two-dimensional panoramic projection image, for example, Rusu The metadata includes metadata information elements corresponding to a plurality of regions of the 2D panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information for that region. The determination module 1503 is configured to determine dynamic mapping information for each pixel within the current angle of view range based on the dynamic mapping information for each of the plurality of regions, for example, by performing step 304 in the method embodiment. The processing module 1504 is configured to perform dynamic range mapping for each pixel within the current angle of view range based on the dynamic mapping information for each pixel within the current angle of view range to obtain a 2D panoramic projection image within the current angle of view range, for example, by performing step 305 in the method embodiment. The 2D panoramic projection image within the current angle of view range is used for display or subsequent processing.

[0251] Optionally, the dynamic range mapping device for panoramic video provided in the embodiment of the present application further includes a transformation module 1505. The transformation module 1505 is configured to project the three-dimensional panoramic image onto a surface of a polyhedron, and expand the polyhedron onto a two-dimensional plane to obtain a two-dimensional panoramic projection image, for example, perform step 301 in the method embodiment. The polyhedron includes at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0252] Optionally, the division module 1501 is particularly configured to divide the two-dimensional panoramic projection image in a first division manner, for example, to perform step 3021 in the method embodiment, to obtain multiple regions of the two-dimensional panoramic projection image, wherein the first division manner is a manner of dividing pixels belonging to the same surface of a polyhedron into regions in the two-dimensional panoramic projection image.

[0253] Optionally, the division module 1501 is particularly configured to divide the 2D panoramic projection image in a second division manner, for example, to perform step 3022 in the method embodiment, to obtain multiple regions of the 2D panoramic projection image, wherein the second division manner is a manner of dividing pixels in the 2D panoramic projection image belonging to the same surface of a polyhedron into regions, and further dividing at least one of the multiple intermediate regions to obtain multiple intermediate regions.

[0254] Optionally, the division module 1501 is particularly configured to: divide the 2D panoramic projection image by a third division scheme to obtain multiple intermediate regions of the 2D panoramic projection image, where the third division scheme is a scheme of clustering all pixels in the 2D panoramic projection image and dividing pixels belonging to the same category into the same intermediate region; and to divide intermediate regions that do not satisfy the preset condition among the multiple intermediate regions by a fourth division scheme to obtain multiple intermediate regions, for example, perform steps 3023 and 3024 in the method embodiment. For any intermediate region that does not satisfy the preset condition, the fourth division scheme is a scheme of dividing the intermediate region along the boundary lines of at least two surfaces of a polyhedron included in the intermediate region.

[0255] Optionally, the acquiring module 1502 is further configured to acquire the correspondence between the regions and the metadata information elements, for example, perform step 306 in the method embodiment. The determining module 1503 is further configured to determine metadata information elements separately corresponding to the multiple regions from the metadata according to the correspondence between the regions and the metadata information elements, so as to determine dynamic mapping information for each of the multiple regions.

[0256] Optionally, the acquisition module 1502 is specifically configured to receive metadata of the two-dimensional panoramic projection image from another electronic device.

[0257] Optionally, the transformation module 1505 is further configured to perform subsequent processing on the 2D panoramic projection image within the current angle of view range, for example, to perform step 308 in the method embodiment. The subsequent processing includes performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range. The 3D panoramic image is used for display.

[0258] The modules of the dynamic range mapping device for panoramic video may be further configured to perform other operations in the method embodiments. Any relevant content of the steps in the method embodiments may be cited in the function description of the corresponding functional module. Details will not be repeated here.

[0259] When an integrated unit is used, FIG. 16 is a diagram of another possible structure of the dynamic range mapping apparatus for panoramic videos in the aforementioned embodiment. As shown in FIG. 16, the dynamic range mapping apparatus for panoramic videos provided in the embodiment of the present application may include a processing module 1601 and a communication module 1602. The processing module 1601 may be configured to control and manage the operation of the apparatus. For example, the processing module 1601 may be configured to support the apparatus in performing steps 301 to 305, 306, 307, and 308 in the method embodiment and / or in performing other processes of the techniques described herein. The communication module 1602 is configured to support the apparatus in communicating with another network entity, for example, communicating with another electronic device (generator). Optionally, as shown in FIG. 16, the dynamic range mapping apparatus for panoramic videos may further include a storage module 1603 configured to store program code and data of the apparatus, for example, metadata.

[0260] The processing module 1601 may be a processor or a controller (e.g., the processor 210 shown in FIG. 2). The processor may alternatively be a combination of processors that perform computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor. The communication module 1602 may be a transceiver, a transceiver circuit, a communication interface, etc. (e.g., the mobile communication module 250 or the wireless communication module 260 shown in FIG. 2). The storage module 1603 may be a memory (e.g., the internal memory 221 shown in FIG. 2).

[0261] When the processing module 1601 is a processor, the communication module 1602 is a transceiver, and the storage module 1603 is a memory, the processor, the transceiver, and the memory may be connected via a bus. The bus may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus may be classified into an address bus, a data bus, a control bus, etc.

[0262] For further details of implementing the aforementioned functions by modules included in the dynamic range mapping device for panoramic video, please refer to the description of the method embodiment, and the details will not be repeated here.

[0263] All embodiments herein are described in stages, with reference to the embodiments for the same or similar parts, and each embodiment highlighting the differences from the other embodiments.

[0264] All or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When a software program is used to implement the embodiments, the embodiments may be fully or partially implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the procedures or functions according to the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored on a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio waves, or microwave) transmission. The computer-readable storage medium may be any available medium accessible by a computer or may be a data storage device, such as a server or data center, that integrates one or more available media. The usable media may be magnetic media (e.g., floppy disks, magnetic disks, or magnetic tapes), optical media (e.g., digital video discs (DVDs)), semiconductor media (e.g., solid state drives (SSDs)), and the like.

[0265] The above description of the embodiments allows those skilled in the art to understand that the above division into functional modules is used as an example for easy explanation. In actual application, the above functions can be allocated and implemented into different modules based on requirements, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above functions. For detailed operation processes of the above systems, devices, and units, please refer to the corresponding processes in the method embodiments, and details will not be described again in this specification.

[0266] It should be understood that in some embodiments provided in the present application, the disclosed systems, devices, and methods may be implemented in other manners. For example, the described device embodiments are merely examples. For example, the division into modules or units is merely a logical division of function, and other divisions may be used in actual implementation. For example, multiple units or assemblies may be combined or integrated into other systems, or some functions may be omitted or not performed. In addition, the shown or described mutual couplings or direct couplings or communication connections may be implemented via some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0267] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, and may be located in one location or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0268] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0269] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may essentially, or a portion that contributes to the prior art, or all or a portion of the technical solution may be embodied in the form of a software product. The software product is stored in a storage medium and includes some instructions for instructing a computer device or processor (which may be a personal computer, a server, or a network device) to perform all or a portion of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium capable of storing program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0270] The above description is merely a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications or replacements within the technical scope disclosed in the present application shall fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. [Explanation of symbols]

[0271] 0 area 1 Antenna, dynamic range, processing method, area 2 Antenna, dynamic range, processing method, area 3 methods, areas 4 methods, areas 5 areas 200 mobile phones 210 processors 220 External Memory Interface 221 internal memory 230 Interface 240 Charging Management Module 241 Power Management Module 242 Battery 250 Mobile Communication Module 260 Wireless Communication Module 270 Audio Module 270A speaker 270B receiver 270C Microphone 270D Headset jack, headset interface 280 Sensor Module 290 Button 291 Motor 292 indicator 293 Camera 294 Display 295 SIM card interface 400 mobile phones 1501 Split Module 1502 Acquisition Module 1503 Decision Module 1504 Processing Module 1505 Conversion Module 1601 Processing Module 1602 communication module 1603 Storage Module

Claims

1. 1. A dynamic range mapping method for panoramic video applied to an electronic device, the method comprising: dividing the 2D panoramic projection image to obtain a plurality of regions of the 2D panoramic projection image, the 2D panoramic projection image being a 2D panoramic projection image corresponding to any frame of a 3D panoramic image in a panoramic video, each of the plurality of regions satisfying a preset condition, the preset condition including at least one of the following conditions: adjacent pixels in the 2D panoramic projection image and non-adjacent pixels in the 3D panoramic image are not located in the same region; non-adjacent pixels in the 3D panoramic image are not located in the same region; the adjacent pixels in the 2D panoramic projection image include pixels on at least two adjacent projection planes of the 3D panoramic image; and the non-adjacent pixels in the 3D panoramic image include pixels in the 3D panoramic image and corresponding to at least two non-adjacent projection planes; acquiring metadata of the 2D panoramic projection image, the metadata including metadata information elements corresponding to the plurality of regions of the 2D panoramic projection image separately, wherein a metadata information element corresponding to one region includes dynamic mapping information of the region; determining dynamic mapping information for each pixel within a current angle of view based on the dynamic mapping information for each of the plurality of regions; performing dynamic range mapping for each pixel within the current angle of view based on the dynamic mapping information of each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, wherein the two-dimensional panoramic projection image within the current angle of view is used for display or subsequent processing; A method comprising:

2. The method comprises: projecting the three-dimensional panoramic image onto a surface of a polyhedron to expand the polyhedron into a two-dimensional plane to obtain the two-dimensional panoramic projection image, wherein the polyhedron comprises at least one of the following: a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron. The method of claim 1 further comprising:

3. said step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image comprises: a step of dividing the two-dimensional panoramic projection image by a first division method to obtain the plurality of regions of the two-dimensional panoramic projection image, the first division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions; 2. The method of claim 1, comprising:

4. said step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image comprises: Dividing the two-dimensional panoramic projection image by a second division method to obtain the plurality of regions of the two-dimensional panoramic projection image, the second division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions and further dividing at least one of the plurality of intermediate regions to obtain a plurality of intermediate regions.

2. The method of claim 1, comprising:

5. said step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image comprises: Dividing the two-dimensional panoramic projection image by a third division method to obtain a plurality of intermediate regions of the two-dimensional panoramic projection image, the third division method being a method of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region; a step of dividing an intermediate region that does not satisfy the preset condition among the plurality of intermediate regions by a fourth division method to obtain the plurality of regions, wherein for any intermediate region that does not satisfy the preset condition, the fourth division method is a method of dividing the intermediate region along boundaries of at least two surfaces of a polyhedron included in the intermediate region; 2. The method of claim 1, comprising:

6. The first division method, the second division method, or the third division method is preset in the electronic device.

6. The method according to any one of claims 3 to 5.

7. the metadata of the two-dimensional panoramic projection image further includes division method instruction information, and the division method instruction information indicates a division method of the two-dimensional panoramic projection image; The method of claim 1.

8. Before the step of determining dynamic mapping information for each pixel within a current angle of view based on the dynamic mapping information for each of the plurality of regions, the method further comprises: obtaining a correspondence between regions and metadata information elements; determining metadata information elements corresponding to the plurality of regions separately from the metadata according to the correspondence between regions and metadata information elements to determine the dynamic mapping information for each of the plurality of regions; The method of claim 1 further comprising:

9. The correspondence between an area and a metadata information element is a correspondence between an index of the metadata information element and an index of the area, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and coordinates of one or more pixels within the region; The method of claim 8.

10. the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a plurality of regions within a first preset range, and the first preset range is a range centered on the pixel to be processed; the dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of an area to which a pixel in the two-dimensional panoramic projection image belongs, the area corresponding to the viewpoint center of the current angle of view range. The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a plurality of regions within a second preset range, and the second preset range is a range centered on a pixel in the 2D panoramic projection image that corresponds to the viewpoint center of the current angle of view range, or The dynamic mapping information of one pixel in a region is the dynamic mapping information of the region; The method of claim 1.

11. the dynamic mapping information is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range; or the dynamic mapping information is used to map a dynamic range of the pixels of the two-dimensional panoramic projection image from a second dynamic range to a first dynamic range, the first dynamic range being greater than the second dynamic range; The method of claim 1.

12. said step of obtaining metadata of said two-dimensional panoramic projection image further comprising: receiving the metadata for the two-dimensional panoramic projection image from another electronic device; 2. The method of claim 1, comprising:

13. The method comprises: performing a subsequent processing on the 2D panoramic projection image within the current angle of view range, the subsequent processing including performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range, the 3D panoramic image being used for display; The method of claim 1 further comprising:

14. 14. An electronic device comprising a memory and at least one processor connected to the memory, wherein the memory is configured to store instructions, and wherein the method of any one of claims 1 to 5, 7 to 13 is executed after the instructions are read by the at least one processor.

15. A computer program product causing an electronic device to perform the method of any one of claims 1 to 5, 7 to 13.

Images