Dynamic range mapping method and apparatus for panoramic video - Patents.com

The method segments panoramic videos into regions based on dynamic range conditions and adapts pixel mapping, addressing the lack of effective dynamic range mapping for panoramic videos and improving display quality.

JP2024545517AActive Publication Date: 2024-12-09HUAWEI TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2024536014
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-17
Filing Date
2022-11-22
Publication Date
2024-12-09
Estimated Expiration
2042-11-22

AI Technical Summary

Technical Problem

Current technologies lack effective dynamic range mapping techniques for panoramic videos, which are essential for ensuring smooth display on devices with varying dynamic ranges.

Method used

A method and apparatus for dynamic range mapping in panoramic videos involve segmenting a two-dimensional panoramic projection image into regions based on preset conditions, obtaining metadata for these regions, and determining dynamic mapping information for each pixel within the current viewing angle range to adaptively map the dynamic range.

Benefits of technology

This approach effectively implements dynamic range mapping on panoramic videos, improving their display quality by ensuring that pixels with similar brightness are grouped together and those with significant brightness differences are separated, thereby enhancing the overall display effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024545517000001_ABST
    Figure 2024545517000001_ABST
Patent Text Reader

Abstract

A dynamic range mapping method and apparatus for panoramic video related to the field of video image processing is provided. The method includes the steps of: dividing a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in a panoramic video to obtain a plurality of regions of the two-dimensional panoramic projection image; obtaining metadata of the two-dimensional panoramic projection image, the metadata including metadata information elements corresponding to the plurality of regions separately, and the metadata information element corresponding to one region including dynamic mapping information of the region; determining dynamic mapping information of each pixel within a current angle of view based on the dynamic mapping information of each of the plurality of regions; and performing dynamic range mapping for each pixel based on the dynamic mapping information of each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, the two-dimensional panoramic projection image being used for display or subsequent processing. According to the method, the dynamic range mapping of the panoramic video can be effectively implemented to improve the display effect of the panoramic video.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202111555880.5, entitled “DYNAMIC RANGE MAPPING METHOD AND APPARATUS FOR PANORAMIC VIDEO,” filed with the China National Intellectual Property Office on December 17, 2021, which is incorporated by reference in its entirety.

[0002] FIELD OF THE DISCLOSURE Embodiments of the present application relate to the field of video image processing, and in particular to a dynamic range mapping method and apparatus for panoramic videos. [Background technology]

[0003] In the field of video / image processing, dynamic range is the ratio of the maximum grayscale value (also called luminance value) of a pixel in an image to the minimum grayscale value of the pixel in the image.

[0004] After capturing the video / image, the electronic device may display the video / image or transmit the video / image to another display device for display. Usually, the dynamic range of pixels that can be displayed by different display devices is different. When the display device displays the video / image, the dynamic range of the captured video / image may be different from the dynamic range supported by the display device. Therefore, the dynamic range of the captured video / image needs to be adjusted to the dynamic range supported by the display device to smoothly display the video / image on the display device. The above process of adjusting the dynamic range of the video / image is dynamic range mapping.

[0005] At present, for two-dimensional videos / images (i.e., captured videos / images), there are several mature dynamic range mapping methods. However, with the development of technology, three-dimensional videos / images (e.g., panoramic videos) are becoming more and more popular, and there is no dynamic range mapping technique that is effective for three-dimensional videos / images. Summary of the Invention

[0006] The embodiments of the present application provide a dynamic range mapping method and apparatus for panoramic video, to effectively implement dynamic range mapping on panoramic video, thereby improving the display effect of the panoramic video. [Means for solving the problem]

[0007] In order to achieve the aforementioned objectives, the following technical solutions are used in the embodiments of this application.

[0008] According to a first aspect, an embodiment of the present application provides a dynamic range mapping method for a panoramic video, the method being applied to an electronic device. The method includes a step in which the electronic device divides the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image. The two-dimensional panoramic projection image is a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in the panoramic video, and each of the plurality of regions satisfies a preset condition, the preset condition including at least one of the following conditions: pixels that are adjacent in the two-dimensional panoramic projection image and are not adjacent in the three-dimensional panoramic image are not located in the same region, and pixels that are not adjacent in the three-dimensional panoramic image are not located in the same region. The pixel points that are adjacent in the two-dimensional panoramic projection image include pixels on at least two adjacent projection planes of the three-dimensional panoramic image, and the pixels that are not adjacent in the three-dimensional panoramic image include pixels that are in the three-dimensional panoramic image and correspond to at least two non-adjacent projection planes. Then, the electronic device obtains metadata of the two-dimensional panoramic projection image. The metadata includes metadata information elements that respectively correspond to a plurality of regions of the two-dimensional panoramic projection image, and the metadata information element that corresponds to one region includes dynamic mapping information of the region. The electronic device determines dynamic mapping information for each pixel within the current angle of view based on the dynamic mapping information for each of the multiple regions, and performs dynamic range mapping for each pixel within the current angle of view based on the dynamic mapping information for each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, which is used for display or subsequent processing.

[0009] In this embodiment of the present application, in the process of dividing a two-dimensional panoramic projection image, when each region obtained by division meets a preset condition, it can be ensured that pixels with similar brightness (similar brightness indicates similar dynamic range) are divided into the same region, and pixels with large brightness differences (large brightness difference indicates large dynamic range difference) are not divided into the same region. In a three-dimensional panoramic projection image, pixel 1 may be a sky pixel, and pixel 2 may be a ground pixel. Because the brightness of the sky pixel is significantly different from the brightness of the ground pixel, in the process of dividing a two-dimensional panoramic projection image, pixel 1 and pixel 2 are not located in the same region. Because the dynamic ranges of pixels in the same region are similar, dynamic range mapping can be performed for pixels in the same region based on the same dynamic range mapping information. Because the dynamic ranges of pixels in different regions may be significantly different, dynamic range mapping can be performed for pixels in different regions based on different dynamic range mapping information. Therefore, after the two-dimensional panoramic projection image is divided into multiple regions by this division method, subsequent processing (i.e., dynamic range mapping) is performed for each region. The characteristics of the pixels within each region can be adaptively adapted in the process to improve processing efficiency.

[0010] In this embodiment of the present application, in relation to the characteristic of the large dynamic range of the panoramic video, the electronic device divides the 2D panoramic projection image corresponding to the 3D panoramic image to obtain regions, and performs dynamic range mapping for different regions, so that the dynamic range mapping can be effectively implemented on the panoramic video, thereby improving the display effect of the panoramic video.

[0011] In one possible implementation, the dynamic range mapping method for panoramic videos provided in this embodiment of the present application further includes a step of projecting the three-dimensional panoramic image onto a surface of a polyhedron to obtain a two-dimensional panoramic projection image, the polyhedron including at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0012] In this embodiment of the present application, the process of using a polyhedron to project a 3D panoramic image into a 2D panoramic projection image includes the steps of: placing a sphere (the 3D panoramic image is represented by a 3D sphere) capable of representing the 3D panoramic image on the polyhedron, so that the sphere is an inscribed sphere of the polyhedron (it should be understood that the center of the polyhedron coincides with the spherical center of the sphere); connecting the sphere center to any point (pixel point) on the sphere and extending a connecting line to intersect with the surface of the polyhedron, where the point obtained by projecting the point on the sphere onto the surface of the polyhedron is the intersection point of the extension of the connecting line connecting the sphere center to the point on the sphere and the surface of the polyhedron; or connecting a point on the surface of the polyhedron to the sphere center, where the connecting line intersects with the sphere at a point, and the intersection point on the sphere and the point on the polyhedron are in a mutual projection relationship. It should be understood that the pixel value of the point on the sphere is the pixel value of the intersection point on the surface of the polyhedron. Thus, in the aforementioned projection process, all pixels on the 3D sphere are projected to obtain the projected pixels on each surface of the polyhedron (interpolation is performed for the projected pixels on the 2D plane based on different cases). Finally, the polyhedron is expanded to the 2D plane according to certain rules to obtain a 2D panoramic projection image.

[0013] The two-dimensional panoramic projection image obtained by expanding a polyhedron into a two-dimensional plane may be an irregular projection image. In the subsequent processing process, the irregular projection image may be processed directly, or the irregular image may be converted into a regular image for processing, optionally. In one embodiment, a rectangular region that encloses the irregular image and has a minimum area may be processed, and another region other than the projection region in the rectangular region may be filled, for example, with a preset pixel value. In another embodiment, the surface image in the irregular projection image may be stitched into a projection image of a regular shape (e.g., a rectangle). In this way, pixel filling does not need to be performed.

[0014] In this embodiment of the present application, the polyhedron is not limited to a regular polyhedron, in other words, a 2D panoramic projection of a 3D panoramic image may contain several polygons of different sizes.

[0015] In one possible embodiment, the step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image particularly includes a step of dividing the two-dimensional panoramic projection image in a first division scheme to obtain a plurality of regions of the two-dimensional panoramic projection image, the first division scheme being a scheme for dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions.

[0016] For example, the polyhedron used for projection is a regular hexahedron. After the 3D panoramic projection image is projected using the regular hexahedron, the obtained 2D panoramic projection image includes six projection surfaces. The first division method is a method of dividing pixels belonging to the same surface of the polyhedron into regions in the 2D panoramic projection image, that is, each projection surface of the 2D panoramic projection image is used as a region. For example, the 2D panoramic projection image may be divided into six regions, region A, region B, region C, region D, region E, and region F in FIG. 8, by the first division method. Each of the multiple regions obtained by dividing the 2D panoramic projection image by the first division method satisfies a preset condition (including condition 1 or condition 2), that is, pixels that are adjacent in the 2D panoramic projection image and not adjacent in the 3D panoramic image are not located in the same region.

[0017] In one possible embodiment, the step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image specifically includes a step of dividing the two-dimensional panoramic projection image using a second division scheme to obtain a plurality of regions of the two-dimensional panoramic projection image, the second division scheme being a scheme for dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions to obtain a plurality of intermediate regions, and further dividing at least one of the plurality of intermediate regions.

[0018] For example, the polyhedron used for projection is still a regular hexahedron. After the 3D panoramic projection image is projected using the regular hexahedron, the obtained 2D panoramic projection image includes six projection surfaces. The second division scheme is a scheme in which the pixels in the 2D panoramic projection image belonging to the same surface of the polyhedron are divided into regions to obtain multiple intermediate regions, and at least one of the multiple intermediate regions is further divided. In other words, the 2D panoramic projection image is first divided by the first division scheme to obtain multiple regions as multiple intermediate regions, and then one or more of the multiple intermediate regions are divided to obtain multiple regions as the final division result. Each of the multiple regions obtained by dividing the 2D panoramic projection image by the second division scheme satisfies a preset condition (including condition 1 or condition 2), i.e., pixels that are adjacent in the 2D panoramic projection image and not adjacent in the 3D panoramic image are not located in the same region.

[0019] In one possible embodiment, the step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image includes a step of dividing the two-dimensional panoramic projection image by a third division scheme to obtain a plurality of intermediate regions of the two-dimensional panoramic projection image, the third division scheme being a scheme of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region, and a step of dividing intermediate regions among the plurality of intermediate regions that do not satisfy a preset condition by a fourth division scheme to obtain a plurality of regions, the fourth division scheme being a scheme of dividing the intermediate region along the boundaries of at least two surfaces of a polyhedron included in the intermediate region, for any intermediate region that does not satisfy the preset condition.

[0020] In this embodiment of the present application, the process of clustering all pixels in a two-dimensional panoramic projection image with a third division scheme includes:

[0021] A clustering center is first determined, and the clustering center is one pixel. Optionally, the two-dimensional panoramic projection image may be first divided into a plurality of regions by a preset division method, and a pixel value at the center position of each region is determined, and the pixel value at the center position of each region is used as a clustering center. For example, assume that the two-dimensional panoramic projection image is an image obtained by projection using a regular hexahedron. If the two-dimensional panoramic projection image is divided into six regions, the two-dimensional panoramic projection image may be divided into six regions by a first division method, and a pixel value at the center position of each region is determined to obtain six clustering centers. If the two-dimensional panoramic projection image is divided into twelve regions, the two-dimensional panoramic projection image may be divided into twelve regions by a second division method, and a pixel value at the center position of each region is determined to obtain twelve clustering centers.

[0022] Then, according to the preset clustering algorithm and the determined clustering centers, a calculation is performed on the distance between all pixel points and each clustering center (the distance is calculated based on the coordinates of the pixel and the coordinates of the pixel corresponding to the clustering center), and the distance can be denoted as Di. Then, a calculation is performed on the difference between the luminance value of all pixel points and the luminance value of the pixel point corresponding to the clustering center (a calculation is performed on the difference between the luminance value of all pixel points and the average luminance value of the pixel points of the category corresponding to each clustering center (i.e., the category obtained by dividing with the preset division method), a calculation is performed on the difference between the color value of all pixel points and the color value of the pixel point corresponding to the clustering center, or a calculation is performed on the difference between the color value of all pixel points and the average color value of the pixel points of the category corresponding to each clustering center (i.e., the category obtained by dividing with the preset division method)), and the difference is denoted as Ei. Each pixel is divided into the category corresponding to the corresponding clustering center based on the weighted value of the distance Di and Ei to obtain a clustering result. The category with the smallest weighted value is the category corresponding to the pixel, and the pixel points of each category in the clustering result form an area.

[0023] Optionally, dividing the two-dimensional panoramic projection image according to the aforementioned clustering method cannot guarantee whether the obtained multiple intermediate regions meet the preset condition (including condition 1 or condition 2). Therefore, the multiple intermediate regions obtained by clustering need to be determined to determine whether the multiple intermediate regions meet the preset condition. If all of the multiple intermediate regions meet the preset condition, the multiple intermediate regions are used as the final division result, that is, the multiple intermediate regions are used as the multiple regions obtained by dividing the two-dimensional panoramic projection image. If there is an intermediate region among the multiple intermediate regions that does not meet the preset condition, the electronic device continues to divide the intermediate region among the multiple intermediate regions that does not meet the preset condition with a fourth division manner to obtain multiple regions.

[0024] In one possible implementation, the first division scheme, the second division scheme, or the third division scheme is pre-configured in the electronic device (eg, pre-configured according to a protocol).

[0025] In one possible embodiment, the metadata of the 2D panoramic projection image further includes division scheme indication information. The division scheme indication information indicates that the division scheme of the 2D panoramic projection image is a first division scheme, a second division scheme, or a third division scheme, that is, the division scheme indication information can be transmitted in the bitstream. After determining the division scheme of the 2D panoramic projection image, the generating end adds indication information indicating the division scheme (i.e., the division scheme indication information) to the bitstream. Thus, after receiving the bitstream, the displaying end can know the division scheme of the 2D panoramic projection image according to the division scheme indication information in the bitstream.

[0026] It will be understood that the first partitioning scheme is related to the second partitioning scheme, and the partitioning result of the second partitioning scheme is a sub-partitioning result based on the first partitioning scheme.

[0027] In one embodiment, when the candidate division schemes of a two-dimensional panoramic projection image include a first division scheme and a second division scheme, whether further division needs to be performed after division with the first division scheme may be determined based on the content of the two-dimensional panoramic projection image, i.e., to use the first division scheme or the second division scheme.

[0028] Specifically, whether the division result of the first division scheme needs to be further divided can be determined based on the following process: firstly dividing the two-dimensional panoramic projection image with the first division scheme to obtain a plurality of intermediate regions; for each of the plurality of intermediate regions, calculating the histogram of the intermediate region, dividing the intermediate region into a plurality of regions, calculating the histogram of each region, and calculating the sum of the difference values ​​between the histogram of the intermediate region and the histogram of the region obtained by dividing the intermediate region; determining whether the sum of the difference values ​​is greater than a preset threshold; and if the sum of the difference values ​​is greater than the preset threshold, further dividing the intermediate region, i.e., using a second division scheme to divide the two-dimensional panoramic projection image. If the sum of the difference values ​​is equal to or less than the preset threshold, the intermediate region does not need to be further divided, i.e., the two-dimensional panoramic projection image is divided with the first division scheme. It should be understood that when the sum of the difference values ​​is greater than the preset threshold, it indicates that the luminance difference between the parts in the intermediate region is large and further division needs to be performed.

[0029] Optionally, in some cases, when the default partitioning scheme is the first partitioning scheme, a related identifier may be used in the bitstream to indicate whether further partitioning is performed. When the identifier indicates that further partitioning is not performed, the partitioning result obtained in the first partitioning scheme is the final result. When the identifier indicates that further partitioning needs to be performed, the partitioning result of the first partitioning scheme is further partitioned.

[0030] In another embodiment, when the candidate division schemes of the two-dimensional panoramic projection image include a first division scheme, a second division scheme, and a third division scheme, which division scheme is used can be determined by performing the following process: firstly dividing the two-dimensional panoramic projection image with the first division scheme, the second division scheme, and the third division scheme; separately calculating histograms of the regions obtained by the first division scheme, the second division scheme, and the third division scheme and the histogram of the two-dimensional panoramic projection image; calculating the sum of difference values ​​between the histograms of the regions obtained by the division schemes and the histogram of the two-dimensional panoramic projection image; and determining the division scheme with the largest sum of difference values ​​between the histograms as the final division scheme. It should be understood that the larger the sum of difference values ​​between the histograms, the larger the luminance difference between the regions obtained by the division, and the more appropriate the division is.

[0031] Optionally, the size of the metadata in the three division schemes may be further calculated, and the metadata includes dynamic mapping information corresponding to each region of the two-dimensional panoramic projection image. Then, it is determined which division scheme is used based on a weighted value between the sum of the histogram difference values ​​corresponding to the division schemes and the size of the metadata corresponding to the division schemes.

[0032] In one possible implementation, before the step of determining dynamic mapping information of each pixel in the current region based on the dynamic mapping information of each of the multiple regions, the dynamic range mapping method for panoramic videos provided in this embodiment of the present application further includes the steps of obtaining a correspondence relationship between the region and metadata information elements, and determining metadata information elements corresponding to the multiple regions separately from the metadata according to the correspondence relationship between the region and the metadata information elements, so as to determine the dynamic mapping information of each of the multiple regions.

[0033] It should be understood that in this embodiment of the present application, the concept of performing dynamic range mapping on a panoramic video is to project all 3D panoramic images contained in the panoramic video onto a 2D plane, perform dynamic range mapping on the obtained 2D panoramic projection images, and convert the 2D panoramic projection images obtained by dynamic range mapping into 3D panoramic images to obtain a panoramic video obtained by dynamic range mapping. In other words, the metadata of the obtained 2D panoramic projection images is used to perform dynamic range mapping of the 3D panoramic image (i.e., used to perform dynamic range mapping on the panoramic video), and the metadata of the 2D panoramic projection images may be considered as the metadata of the 3D panoramic image (or the metadata of the panoramic video).

[0034] Optionally, the correspondence between the regions and the metadata information elements may be a one-to-one relationship or a many-to-one relationship. In other words, one region of the two-dimensional panoramic projection image corresponds to one metadata information element, or multiple regions of the two-dimensional panoramic projection image correspond to one metadata information element. When the luminance difference between multiple regions of the two-dimensional panoramic projection image is large and one type of dynamic mapping information is used for each region, the correspondence between the regions and the metadata information elements is a one-to-one relationship. When the luminance of some regions in the multiple regions is similar, some regions may use the same type of dynamic mapping information, and the correspondence between the regions and the metadata information elements is a many-to-one relationship.

[0035] In this embodiment of the present application, a preset algorithm may be used to determine whether the correspondence between the multiple regions in the two-dimensional panoramic image and the metadata information elements is a one-to-one relationship or a many-to-one relationship. For example, whether to use the one-to-one or many-to-one correspondence may be determined by using the following method, that is, the multiple regions are clustered, and the number of clustered categories may be set based on requirements. For example, the number of categories is set to half the number of regions or a quarter of the number of regions. For example, when the total number of regions is 8, the 8 regions may be divided into two categories, or the 8 regions may be divided into four categories, which corresponds to obtaining 4 large regions. Then, the sum of the differences between the histograms of the regions corresponding to the categories in the clustering result and the histogram of the two-dimensional panoramic projection image is calculated, and the sum of the differences between the histograms of the multiple original regions and the histogram of the two-dimensional panoramic image is calculated. Finally, based on the two sums of the differences between the histograms, it is determined whether the correspondence between the multiple regions and the metadata information elements is a one-to-one relationship or a many-to-one relationship. For example, when eight regions are clustered to obtain two categories of regions, the sum of the differences between the histograms of the regions corresponding to each category of the clustering result and the histogram of the two-dimensional panoramic projection image is denoted as DiffHIS_X2, and the sum of the differences between the histograms of multiple original regions and the histogram of the two-dimensional panoramic image is denoted as DiffHIS_org. The determination method is as follows: if DiffHIS_org×(1-T2)>DiffHIS_X2, the correspondence relationship between the regions and the metadata information elements is a many-to-one relationship, or if DiffHIS_org×(1-T2)≦DiffHIS_X2, the correspondence relationship between the regions and the metadata information elements is a one-to-one relationship, and T2 is a preset threshold.

[0036] In one possible implementation, the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and an index of the region, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata element and the coordinates of one or more pixels within the region.

[0037] In this embodiment of the present application, the correspondence between the index of the metadata information element and the index of the region is essentially a traversal order relationship. For example, in the case of a two-dimensional panoramic projection image including six regions, the six regions form a rectangle of 2×3 (i.e., two rows, each row including three regions), and are region 4, region 0, region 5, region 3, region 1, and region 2, from left to right and top to bottom, in order. Thus, in the table of correspondence between regions and metadata information elements, the six regions in the two-dimensional panoramic projection image are traversed from left to right and top to bottom in order, and the metadata information elements corresponding to the six regions separately can be determined in the metadata in order. In one embodiment, the traversal order of the multiple regions can be indicated in the bitstream. For example, instruction information indicating the traversal order may be carried in the metadata. The traversal order is not limited to the above order from left to right and top to bottom, and the traversal order is specifically related to the position of each region in the two-dimensional panoramic projection image.

[0038] Optionally, the traversal order of the multiple regions may be a preset order, or the generating end determines a traversal order from the candidate traversal orders according to a related algorithm, adds an identifier to the metadata of the bitstream, and forwards the bitstream to the generating end to indicate the traversal order.

[0039] In this embodiment of the application, which traversal order to use may be determined using the following method: computing a histogram for each of a plurality of regions; computing, for each candidate traversal order, the sum of the differences between the histograms of adjacent regions; and determining as the final traversal scheme the traversal method that results in the smallest sum of the differences between the histograms in the candidate traversal orders.

[0040] In this embodiment of the present application, the coordinates of the pixels of the region can also be carried in the metadata. Specifically, the coordinates of the pixels of each region are carried in the metadata information element corresponding to each region. After dividing the 2D panoramic projection image to obtain multiple regions, for one of the regions, the display end determines the pixel whose coordinate belongs to the region according to the correspondence relationship, and uses the metadata information element corresponding to the coordinate as the metadata information element of the region.

[0041] In one possible embodiment, the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of a plurality of regions within a first preset range, the first preset range being a range centered on the pixel to be processed (method 1), the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of a region to which a pixel in a two-dimensional panoramic projection image belongs, corresponding to the viewpoint center of the current angle of view range (method 2), the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of a plurality of regions within a second preset range, the second preset range being a range centered on a pixel in a two-dimensional panoramic projection image, corresponding to the viewpoint center of the current angle of view range (method 3), or the dynamic mapping information of one pixel in the current region is dynamic mapping information of the current region (method 4).

[0042] It should be understood that when a user watches a panoramic video, the user (head) cannot see all the content of the panoramic video in the current pose, but can only see the content of the video in a certain range. The range can be understood as the user's angle of view range. It can be seen that the user's angle of view range is related to the user's current viewing pose. In this embodiment of the present application, performing dynamic range mapping on the panoramic video is to perform dynamic range mapping on the content within the angle of view range of the user who is currently watching the panoramic video. After obtaining the dynamic mapping information of the multiple regions, the electronic device can determine the dynamic mapping information of each pixel within the current angle of view range in any of four manners based on the dynamic mapping information of the multiple regions.

[0043] Optionally, during practical application, the generating end may select an appropriate scheme from the four schemes based on the practical requirements for determining the dynamic mapping information of the pixel to be processed, and transfer the selected scheme carried in the bitstream (metadata) to the display end. Alternatively, the generating end and the display end may predetermine the scheme to be used. This is not limited in this embodiment of the present application.

[0044] In one possible implementation, the dynamic mapping information in this embodiment of the present application is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range, or the dynamic mapping information is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from the second dynamic range to the first dynamic range, the first dynamic range being greater than the second dynamic range, i.e. the dynamic range mapping performed on the pixels may be a mapping from a high dynamic range to a low dynamic range or a mapping from a low dynamic range to a high dynamic range.

[0045] Correspondingly, after the dynamic mapping information of each pixel in the current angle of view range is determined using the method, the pixel may correspond to multiple dynamic mapping information (e.g., multiple dynamic mapping curves). After dynamic range mapping is performed on the pixel, one dynamic mapping result may be obtained for each dynamic mapping information. In this case, the multiple dynamic mapping results may be processed to obtain a final dynamic mapping result. The processing manner may include any one of using a median value of the multiple dynamic mapping results as the final dynamic mapping result, or performing a weighted average on the multiple dynamic mapping results to obtain the final dynamic mapping result.

[0046] In one possible embodiment, the step of obtaining the metadata of the 2D panoramic projection image specifically includes the step of receiving the metadata of the 2D panoramic projection image from another electronic device. Specifically, after generating the metadata, the generating end encodes the metadata into a bitstream and sends the bitstream to the displaying end, so that the displaying end receives the bitstream and analyzes the bitstream to obtain the metadata.

[0047] In one possible implementation, the dynamic range mapping method for panoramic video in this embodiment of the present application further includes performing a subsequent processing on the two-dimensional panoramic projection image within the current angle of view range, the subsequent processing includes performing a three-dimensional transformation on the two-dimensional panoramic projection image within the current angle of view range to obtain a three-dimensional panoramic image within the current angle of view range, and the three-dimensional panoramic image is used for display.

[0048] The obtained 2D panoramic projection image may be used for display. Furthermore, the 2D panoramic projection image may be further used for subsequent processing, for example, the 2D panoramic projection image is converted into a 3D panoramic image for display. Furthermore, after all the 3D panoramic images of the panoramic video are obtained, the panoramic video may be played.

[0049] According to a second aspect, an embodiment of the present application provides a dynamic range mapping device for panoramic video, including a division module, an acquisition module, a determination module, and a processing module. The division module is configured to divide a two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image. The two-dimensional panoramic projection image is a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in the panoramic video, and each of the plurality of regions satisfies a preset condition, the preset condition including at least one of the following conditions: pixels that are adjacent in the two-dimensional panoramic projection image and are not adjacent in the three-dimensional panoramic image are not located in the same region, and pixels that are not adjacent in the three-dimensional panoramic image are not located in the same region. The pixel points that are adjacent in the two-dimensional panoramic projection image include pixels on at least two adjacent projection planes of the three-dimensional panoramic image, and the pixels that are not adjacent in the three-dimensional panoramic image include pixels that are in the three-dimensional panoramic image and correspond to at least two non-adjacent projection planes. The acquisition module is configured to acquire metadata of the two-dimensional panoramic projection image, the metadata including metadata information elements respectively corresponding to a plurality of regions of the two-dimensional panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information of the region. The determination module is configured to determine dynamic mapping information of each pixel within the current angle of view range based on the dynamic mapping information of each of the plurality of regions. The processing module is configured to perform dynamic range mapping for each pixel within the current angle of view range based on the dynamic mapping information of each pixel within the current angle of view range to obtain a two-dimensional panoramic projection image within the current angle of view range, and the two-dimensional panoramic projection image within the current angle of view range is used for display or subsequent processing.

[0050] In one possible implementation, the dynamic range mapping device for panoramic video provided in this embodiment of the present application further includes a transformation module. The transformation module is configured to project the three-dimensional panoramic image onto a surface of a polyhedron and expand the polyhedron into a two-dimensional plane to obtain a two-dimensional panoramic projection image. The polyhedron includes at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0051] In one possible embodiment, the division module is particularly configured to divide the two-dimensional panoramic projection image in a first division scheme to obtain multiple regions of the two-dimensional panoramic projection image, the first division scheme being a scheme for dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions.

[0052] In one possible embodiment, the division module is particularly configured to divide the two-dimensional panoramic projection image in a second division manner to obtain multiple regions of the two-dimensional panoramic projection image, the second division manner being a manner of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions to obtain multiple intermediate regions, and further dividing at least one of the multiple intermediate regions.

[0053] In one possible embodiment, the division module is particularly configured to: divide the two-dimensional panoramic projection image using a third division scheme to obtain multiple intermediate regions of the two-dimensional panoramic projection image, the third division scheme being a scheme of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region; and divide an intermediate region among the multiple intermediate regions that does not satisfy a preset condition using a fourth division scheme to obtain multiple regions, and for any intermediate region that does not satisfy the preset condition, the fourth division scheme being a scheme of dividing the intermediate region along the boundaries of at least two surfaces of a polyhedron included in the intermediate region.

[0054] In one possible embodiment, the first division scheme, the second division scheme or the third division scheme is pre-configured in the electronic device.

[0055] In one possible embodiment, the metadata of the 2D panoramic projection image further includes division scheme indication information, the division scheme indication information indicating that the division scheme of the 2D panoramic projection image is a first division scheme, a second division scheme, or a third division scheme.

[0056] In one possible implementation, the acquisition module is further configured to acquire a correspondence relationship between the regions and the metadata information elements. The determination module is further configured to determine metadata information elements corresponding to the multiple regions separately from the metadata according to the correspondence relationship between the regions and the metadata information elements, so as to determine dynamic mapping information for each of the multiple regions.

[0057] In one possible implementation, the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and an index of the region, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata element and the coordinates of one or more pixels within the region.

[0058] In one possible embodiment, the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of multiple regions within a first preset range, the first preset range being a range centered on the pixel to be processed; the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of a region to which a pixel in a two-dimensional panoramic projection image belongs, corresponding to the viewpoint center of the current angle of view; the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of multiple regions within a second preset range, the second preset range being a range centered on a pixel in a two-dimensional panoramic projection image, corresponding to the viewpoint center of the current angle of view; or the dynamic mapping information of one pixel in the current region is dynamic mapping information of the current region.

[0059] In one possible implementation, the dynamic mapping information in this embodiment of the present application is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range, or the dynamic mapping information is used to map the dynamic range of the pixels of the two-dimensional panoramic projection image from the second dynamic range to the first dynamic range, where the first dynamic range is greater than the second dynamic range.

[0060] In one possible implementation, the acquisition module is specifically configured to receive metadata of the 2D panoramic projection image from another electronic device.

[0061] In one possible embodiment, the transformation module is further configured to perform a subsequent processing on the 2D panoramic projection image within the current angle of view range, the subsequent processing including performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range, the 3D panoramic image being used for display.

[0062] According to a third aspect, an embodiment of the present application provides an electronic device including a memory and at least one processor connected to the memory, the memory being configured to store instructions, and the instructions being read by the at least one processor before the method of the first aspect and any one of the possible implementations of the first aspect is executed.

[0063] According to a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method of the first aspect and any one of the possible implementations of the first aspect.

[0064] According to a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which when executed on a computer, perform the method of the first aspect or any one of the possible implementations of the first aspect.

[0065] According to a sixth aspect, an embodiment of the present application provides a chip including a memory and a processor, the memory configured to store computer instructions, the processor configured to call the computer instructions from the memory and execute the computer instructions to perform a method according to the first aspect or any one of the possible implementations of the first aspect.

[0066] For the beneficial effects achieved by the technical solutions in the second to sixth aspects and the corresponding possible implementations in the embodiments of the present application, please refer to the aforementioned technical effects in the first aspect and the corresponding possible implementations, and the details will not be repeated here. [Brief description of the drawings]

[0067] [Figure 1] 1 is a schematic diagram of a framework of a video processing process according to an embodiment of the present application; [Diagram 2] 1 is a schematic diagram of the hardware of a mobile phone according to an embodiment of the present application. [Diagram 3] 1 is a schematic diagram of a dynamic range mapping method for panoramic video according to an embodiment of the present application; [Figure 4] 1 is a schematic diagram of a 3D panoramic image projection process according to an embodiment of the present application. [Diagram 5] 2 is a schematic diagram of a 3D panoramic image projection process according to an embodiment of the present application. [Figure 6] 3 is a schematic diagram of a 3D panoramic image projection process according to an embodiment of the present application. [Figure 7] 2 is a schematic diagram of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 8] 1 is a schematic diagram of a segmentation result of a 2D panoramic projection image according to an embodiment of the present application; [Figure 9] 3 is a schematic diagram of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 10] 2 is a schematic diagram of a segmentation result of a 2D panoramic projection image according to an embodiment of the present application; [Figure 11] 4 is a schematic diagram of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 12] 3 is a schematic diagram of a segmentation result of a two-dimensional panoramic projection image according to an embodiment of the present application; [Figure 13] 5 is a schematic diagram of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 14] 6 is a schematic diagram of a dynamic range mapping method for panoramic video according to an embodiment of the present application. [Figure 15] 1 is a schematic diagram of the structure of a dynamic range mapping apparatus for panoramic video according to an embodiment of the present application; [Figure 16] 2 is a schematic diagram of the structure of a dynamic range mapping apparatus for panoramic video according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0068] The term "and / or" in this specification only describes an association relationship for describing associated objects, and indicates that three relationships may exist. For example, A and / or B can represent three cases: when only A exists, when both A and B exist, and when only B exists.

[0069] In the specification and claims of the embodiments of the present application, the terms "first", "second", etc. are intended to distinguish different objects, but do not indicate a specific order of the objects. For example, a first division scheme, a second division scheme, etc. are used to distinguish different division schemes, but are not used to describe a specific order of the division schemes.

[0070] In the embodiments of the present application, the words "example" or "for example" are used to provide an example, illustration, or explanation. In the embodiments of the present application, any embodiment or design scheme described as an "example" or "for example" should not be described as being preferred or having more advantages over another embodiment or design scheme. Rather, words such as "example" or "for example" are intended to present the relevant concept in a particular manner.

[0071] In describing the embodiments of the present application, unless otherwise specified, "plurality" means two or more than two. For example, a plurality of regions means two or more regions.

[0072] In the following, we first explain some concepts related to the dynamic range mapping method and apparatus for panoramic video provided in the embodiments of the present application.

[0073] Dynamic range: In most fields, the dynamic range indicates the ratio between the maximum and minimum values ​​of a variable. In the digital image field, the dynamic range indicates the ratio between the maximum and minimum tonal values ​​of the pixels of the image. Tonal values ​​are also expressed as 1 nit = 1 cd / m 2 is nit or candela per square meter (cd / m 2 ) may be understood as a luminance value in units of luminance.

[0074] Typically, the dynamic range in nature is large, with the luminance of a night scene under the stars being approximately 0.001cd / m 2 and the luminosity of the sun is 10 9 cd / m2 Thus, the dynamic range of the natural world can reach 10 9 / 0.001=10 12 In a real natural environment scene, the brightness of the sun and the brightness of the stars cannot be obtained at the same time, so the dynamic range of the real natural environment scene is 10 12 In real natural scenes, the dynamic range is usually 10 -3 From 10 6 It is.

[0075] Dynamic range mapping: After a video / image is captured in a real natural environment, in the process of displaying the captured video / image on a display device (e.g., a TV or an iPad), the dynamic range of the displayed video / image supported by the display device may be different from the original dynamic range of the video / image captured from the real natural environment, so that if the video / image is directly displayed based on the original dynamic range of the video / image, the display device cannot display the video / image. Therefore, the dynamic range of the captured video / image needs to be adapted to the display device, i.e., the dynamic range of the captured video / image is adjusted to the dynamic range supported by the display device to display the captured video / image on the display device. The above process of adjusting the dynamic range of the video / image is dynamic range mapping (sometimes called tone mapping).

[0076] It will be appreciated that dynamic range can be classified as high dynamic range (HDR) or low dynamic range, with low dynamic range sometimes referred to as standard dynamic range (SDR). -3 From 10 6An image with a luminance range of 0 to 255 is called a high dynamic range image, and an image with a luminance range of 0 to 255 is called a low dynamic range image. Currently, in most color digital images, one byte (i.e., 8 bits) represents each pixel in the R, G, and B channels, i.e., the luminance range of a pixel in each channel is 0 to 255, with 0 to 255 being the standard dynamic range of an image.

[0077] In the embodiment of the present application, the dynamic range mapping includes mapping from a high dynamic range to a low dynamic range and mapping from a low dynamic range to a high dynamic range. It should be noted that "high" and "low" here are relative concepts, which are different from the above concepts of high dynamic range and low dynamic range. For example, the dynamic range is mapped from dynamic range 1 to dynamic range 2. The relationship between dynamic range 1 and dynamic range 2 may be such that dynamic range 1 is larger than dynamic range 2, or dynamic range 1 is smaller than dynamic range 2. However, dynamic range 1 and dynamic range 2 may both be high dynamic ranges or low dynamic ranges.

[0078] For example, the original dynamic range of a captured video / image is HDR of 1000, and the dynamic range supported by the display device is SDR of 100. In this case, the dynamic range mapping is from a high dynamic range to a low dynamic range, the dynamic range before the mapping is HDR, and the dynamic range after the mapping is SDR. In another example, the original dynamic range of a captured video / image is HDR of 4000, and the dynamic range supported by the display device is HDR of 500. In this case, the dynamic range mapping is also from a high dynamic range to a low dynamic range, but both the dynamic range before the mapping and the dynamic range after the mapping are HDR.

[0079] For example, the original dynamic range of a captured video / image is SDR of 100, and the dynamic range supported by the display device is HDR of 2000. In this case, the dynamic range mapping is from a low dynamic range to a high dynamic range, the dynamic range before the mapping is SDR, and the dynamic range after the mapping is HDR. In another example, the original dynamic range of a captured video / image is SDR of 100, and the dynamic range supported by the display device is SDR of 200. In this case, the dynamic range mapping is from a low dynamic range to a high dynamic range, but both the dynamic range before the mapping and the dynamic range after the mapping are SDR.

[0080] With reference to the previous examples, Table 1 shows some examples of dynamic range mapping.

[0081] [Table 1]

[0082] When dynamic range mapping is performed on a video / image, several dynamic range mapping models may be used to perform dynamic range mapping on the pixels of the captured video / image. The dynamic range mapping models include, but are not limited to, sigmoidal curves, Bessel curves, etc. It should be understood that the dynamic range mapping models are models for performing dynamic range mapping on two-dimensional videos / images.

[0083] Optionally, the method for performing dynamic range mapping on a video / image includes static dynamic range mapping and dynamic dynamic range mapping. Static dynamic range mapping is performing dynamic range mapping on a video / image by using the same dynamic range mapping curve for the same video / image content (which may relate to video / image content of multiple scenes) or video / image content in the same hard disk. Dynamic dynamic range mapping is performing dynamic range mapping on a video / image using different dynamic range mapping curves based on video / image content of different scenes or different frames.

[0084] The advantage of static dynamic range mapping is that it is a simple dynamic range mapping process, and little information is carried to transmit dynamic mapping information (e.g., information including dynamic range mapping curve). The disadvantage of static dynamic range mapping is that the mapping result obtained by performing dynamic range mapping on a video / image using a single dynamic range mapping curve may be poor, resulting in an insufficient display effect of the video / image obtained by dynamic range mapping. For example, when a dynamic range mapping curve is mainly used to perform dynamic range mapping for high brightness, the effect of performing dynamic range mapping on the bright area of ​​the image using the dynamic range mapping curve is good. However, in the dark area of ​​the image, when the dynamic range mapping curve is used to perform dynamic range mapping, the pixel value of the dark area becomes too small, and some information (e.g., detailed information) is lost in the image obtained by dynamic range mapping.

[0085] The advantage of dynamic dynamic range mapping is that for different scenes or different frames, different dynamic range mapping curves suitable for different scenes or different frames are used to perform dynamic range mapping, and the mapping results are good, that is, the display effect of the video / image obtained by dynamic range mapping is good. The disadvantage of dynamic dynamic range mapping is the complex dynamic range mapping process, and the large amount of information carried to transmit the dynamic mapping information.

[0086] Panoramic Video: A panoramic video is a video that includes spherical scene content with a horizontal extent of 360 degrees and a vertical extent of 180 degrees. A panoramic video includes a three-dimensional (3D) panoramic image of a plurality of frames, and the 3D panoramic image may be represented using a 3D sphere. Therefore, the 3D panoramic image of a panoramic video may be referred to as a 3D spherical representation panorama of the panoramic video.

[0087] Compared with common two-dimensional videos (covering a range of 33 degrees horizontally and 18 degrees vertically), panoramic videos cover a wider range and provide users with an immersive visual experience. The user watches the panoramic video using a wearable auxiliary device (e.g., VR glasses). Specifically, the user's head may be taken as the center point and the head may rotate 360 ​​degrees horizontally and 180 degrees vertically. The user is free to select a designated direction to view the window by adjusting the horizontal and vertical angles of the field of view, and the window is the user's field of view angle range. It can be understood that when the panoramic video is simulated as a spherical surface of a sphere and the user's head is located at the spherical center of the sphere, the window can be continuously adjusted by rotating the head and / or looking up / down to view the complete panoramic video.

[0088] In the embodiment of the present application, the panoramic video has a larger dynamic range compared with a general two-dimensional video because the luminance coverage range of the panoramic video is larger. Specifically, the two-dimensional video is obtained in a narrow range, and the light conditions in the narrow range are similar, i.e., the light change range is small and the dynamic range is also small. The panoramic video is obtained in a wide range, and the light change range is usually large in a wide range. For example, a panoramic video obtained in the daytime may include a sun area in the sky, which is a bright area (i.e., a high luminance value), and may also include an interior area or a shadow area of ​​a building, which is a dark area (i.e., a low luminance value). Therefore, the dynamic range of the panoramic video is large, and there are great challenges in describing and displaying the panoramic video.

[0089] At present, the technology for performing dynamic range mapping on 2D videos / images is mature. However, with the emergence of panoramic videos, there is an urgent need for related technologies to perform dynamic range mapping on panoramic videos. In view of this, the embodiments of the present application provide a dynamic range mapping method and apparatus for panoramic videos. For any frame of a 3D panoramic image in a panoramic video, an electronic device performs dynamic range mapping on a 2D panoramic projection image corresponding to the 3D panoramic image, and converts the 2D panoramic projection image obtained through dynamic range mapping into a 3D panoramic image to perform dynamic range mapping on the panoramic video.

[0090] The specific solutions of the embodiments of the present application are as follows: The electronic device divides a 2D panoramic projection image corresponding to a 3D panoramic image to obtain multiple regions of the 2D panoramic projection image, and each of the multiple regions satisfies a preset condition (the preset condition will be described in detail in the following embodiments); The electronic device obtains metadata of the 2D panoramic projection image, and the metadata includes metadata information elements respectively corresponding to the multiple regions of the 2D panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information of the region; Then, the electronic device determines dynamic mapping information of each pixel within a current angle of view range based on the dynamic mapping information of each of the multiple regions, and performs dynamic range mapping for each pixel within the current angle of view range based on the dynamic mapping information of each pixel within the current angle of view range (the angle of view range of the user currently viewing the panoramic video) to obtain a 2D panoramic projection image within the current angle of view range, and the 2D panoramic projection image within the current angle of view range is used for display or subsequent processing. In the dynamic range mapping method for panoramic video provided in the embodiments of the present application, in relation to the characteristic that the dynamic range of the panoramic video is large, the electronic device divides a 2D panoramic projection image corresponding to a 3D panoramic image to obtain regions, and performs dynamic range mapping for different regions. This makes it possible to effectively implement dynamic range mapping on the panoramic video, thereby improving the display effect of the panoramic video.

[0091] It should be noted that in the embodiment of the present application, the format of the image (sometimes called source data) included in the panoramic video is not limited, and the image may be in YUV format or RGB format. In addition, in the embodiment of the present application, the bit width of the source data is limited, and the bit width of the source data may be 8 bits, 10 bits, or 12 bits.

[0092] For a panoramic video, it can be understood that the processing process of the panoramic video generally mainly includes generating, encoding, decoding and displaying the panoramic video. The generating and encoding process of the panoramic video may be completed at a front end (sometimes called a generating end), and the decoding and displaying process of the panoramic video may be completed at a back end (sometimes called a displaying end). See FIG. 1. At the generating end, the generating of the panoramic video includes performing a simulated dynamic range mapping for each frame of a 3D panoramic image in the panoramic video, determining dynamic mapping information (e.g., the dynamic mapping information includes information of a curve used for dynamic range mapping), and generating metadata including the dynamic mapping information. The encoding of the panoramic video includes encoding each frame of the 3D panoramic image in the panoramic video (e.g., including prediction, transformation, quantization, and entropy encoding). For the image encoding process, please refer to the prior art. The panoramic video encoding method is not limited in this application. The generating end further writes the generated metadata including the dynamic mapping information into a bitstream, and transmits the bitstream to the displaying end. At the display end, the bitstream is parsed to obtain metadata and image decoding data (including syntax elements used for image decoding and image data), and the image data is decoded (including, for example, prediction, inverse quantization, and inverse transform) based on the syntax elements used for image decoding to obtain a 3D panoramic image. For the image decoding process, please refer to the prior art (the decoding method corresponds to the encoding method). The panoramic video decoding method is not limited in this application. Then, the 3D panoramic image is converted into a 2D panoramic projection image, and dynamic range mapping is performed on the 2D panoramic projection image based on the metadata obtained by parsing the bitstream, and the panoramic video obtained by dynamic range mapping is displayed.

[0093] In the embodiment of the present application, the dynamic range mapping process of panoramic video is described from the perspective of a display end. For example, the product forms of electronic devices used as display ends include, but are not limited to, virtual reality (VR) devices (e.g., VR glasses), display devices (e.g., mobile phones), video application conversion devices (e.g., transcoders), and live streaming devices. It should be understood that in VR glasses and display devices, dynamic range mapping is implemented based on hardware chips, and in live streaming devices and video conversion devices, dynamic range mapping is implemented based on software program codes.

[0094] For example, in an embodiment of the present application, the display end is a mobile phone. Figure 2 is a diagram of the hardware structure of a mobile phone 200 according to an embodiment of the present application. The mobile phone 200 includes a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display 294, a subscriber identification module (SIM) card interface 295, etc.

[0095] It will be understood that the exemplary structures in the embodiments of the present application do not constitute any particular limitation on the mobile phone 200. In some other embodiments of the present application, the mobile phone 200 may include more or fewer components than those shown in the figures, or some components may be combined, or some components may be divided, or components may be arranged in a different manner. The components shown in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0096] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The various processing units may be independent components or may be integrated into one or more processors. The controller may be the nerve center and command center of the mobile phone 200. The controller may generate an operation control signal based on the instruction operation code and the time series signal to complete the control of reading the instruction and executing the instruction.

[0097] A memory may further be disposed in the processor 210 and configured to store instructions and data. In some embodiments, the memory in the processor 210 is a cache. The memory may store instructions or data that have just been used or are used periodically by the processor 210. When the processor 210 needs to use the instructions or data again, the processor 210 may directly retrieve the instructions or data from the memory. This avoids repeated accesses and reduces the latency of the processor 210, thereby improving system efficiency.

[0098] The charging management module 240 is configured to receive a charging input from a charger. While charging the battery 242, the charging management module 240 may further power the electronic device using the power management module 241.

[0099] The power management module 241 is configured to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or input from the charging management module 240 and provides power to the processor 210, the internal memory 221, the external memory, the display 294, the camera 293, the wireless communication module 260, etc. The power management module 241 may be further configured to monitor parameters such as battery capacity, battery cycle count, and battery health (leakage and impedance). In some other embodiments, the power management module 241 may be alternatively located in the processor 210. In some other embodiments, the power management module 241 and the charging management module 240 may be alternatively located in the same component.

[0100] The wireless communication function of the mobile phone 200 may be implemented using an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, a modem processor, a baseband processor, etc. The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals.

[0101] The mobile communication module 250 may provide a wireless communication solution applied to the mobile phone 200, including 2G / 3G / 4G / 5G, etc. The mobile communication module 250 may receive electromagnetic waves via the antenna 1, perform processing such as filtering or amplification on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 250 may further amplify the signals modulated by the modem processor and convert the amplified signals to electromagnetic waves for emission via the antenna 1. In some embodiments, at least some of the functional modules in the mobile communication module 250 may be located within the processor 210. In some embodiments, at least some of the functional modules in the mobile communication module 250 may be located within the same component as at least some of the modules in the processor 210.

[0102] The wireless communication module 260 may provide wireless communication solutions applied to the mobile phone 200, including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, or infrared (IR) technology, etc. The wireless communication module 260 may be one or more components integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 210. The wireless communication module 260 may further receive signals to be transmitted from the processor 210, perform frequency modulation and amplification on the signals, and convert the signals into electromagnetic waves for radiation via the antenna 2.

[0103] In some embodiments, in the mobile phone 200, antenna 1 is coupled to the mobile communication module 250 and antenna 2 is coupled to the wireless communication module 260, such that the mobile phone 200 can communicate with a network and another device using wireless communication techniques.

[0104] The mobile phone 400 realizes display functions by using a GPU, a display 294, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display 294 and the application processor. The GPU is configured to perform mathematical and geometric calculations and render images. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0105] The display 294 is configured to display images, videos, etc. The display 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diode (QLED), etc. In some embodiments, the mobile phone 200 may include one or N displays 294, where N is a positive integer greater than 1.

[0106] The mobile phone 200 may implement a photography function using an ISP, a camera 293, a video codec, a GPU, a display 294, an application processor, etc. The ISP is configured to process data fed back by the camera 293. For example, during photography, a shutter is pressed, light is transmitted to the camera's photosensitive element through a lens, the light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting the electrical signal into a visible image. The ISP may further perform algorithmic optimization on noise, brightness, and skin tone of the image. The ISP may further optimize parameters such as exposure and color temperature of the photographed scene. In some embodiments, the ISP may be located in the camera 293.

[0107] The camera 293 is configured to capture still images or videos. An optical image of an object is generated through a lens and projected onto a light-receiving element. The light-receiving element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light-sensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. In some embodiments, the mobile phone 200 may include one or N cameras 293, where N is a positive integer greater than 1.

[0108] The digital signal processor is configured to process digital signals, and may process other digital signals in addition to digital image signals (eg audio signals).

[0109] A video codec is configured to compress or decompress digital video. Mobile phone 200 may support one or more video codecs. Thus, mobile phone 200 can play or record video in multiple encoding formats, e.g., moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, and MPEG-4.

[0110] The external memory interface 220 may be configured to connect to an external memory card, such as a Micro SD card, to expand the storage capabilities of the mobile phone 200. The external memory card communicates with the processor 210 through the external memory interface 220 to perform data storage functions. For example, files such as music and videos are stored on the external memory card.

[0111] The internal memory 221 may be configured to store computer executable program code, the executable program code including instructions. The processor 210 executes the instructions stored in the internal memory 221 to implement various functional applications and data processing of the mobile phone 200. The internal memory 221 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., a voice playback function and an image playback function), and the like. The data storage area may store data created during use of the mobile phone 200 (e.g., audio data and an address book), and the like. In addition, the internal memory 221 may include a high-speed random access memory and may include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory, or a universal flash storage (UFS).

[0112] The mobile phone 200 may use an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset interface 270D, an application processor, etc. to perform audio functions such as playing or recording music.

[0113] The audio module 270 is configured to convert digital audio information to analog audio signals for output, and is also configured to convert analog audio input to digital audio signals. The audio module 270 may be further configured to encode and decode audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0114] The speaker 270A, also called a "horn," is configured to convert audio electrical signals into acoustic signals. The mobile phone 200 may use the speaker 270A to listen to music or to answer hands-free calls.

[0115] Receiver 270B, also referred to as an "earpiece," is configured to convert an audio electrical signal into an acoustic signal. When a call is answered or voice information is received by using mobile phone 200, receiver 270B may be brought close to a person's ear to listen to the sound.

[0116] The microphone 270C, also referred to as a "mike" or "mic", is configured to convert an acoustic signal into an electrical signal. When making a call or transmitting voice information, a user may place a person's mouth close to the microphone 270C to speak and input an acoustic signal to the microphone 270C. At least one microphone 270C may be disposed on the mobile phone 200. In some other embodiments, two microphones 270C may be disposed on the mobile phone 200 to capture acoustic signals and further perform noise reduction functions. In some other embodiments, three, four, or more microphones 270C may alternatively be disposed on the mobile phone 200 to capture acoustic signals, reduce noise, further identify sound sources, perform directional recording functions, and the like.

[0117] Headset jack 270D is configured to connect to a wired headset.

[0118] The buttons 290 include a power button, a volume button, etc. The mobile phone 200 may receive button inputs and generate button signal inputs associated with user settings and function control of the mobile phone 200.

[0119] The motor 291 may generate vibration prompts. The motor 291 may be configured to provide incoming vibration prompts and touch vibration feedback.

[0120] The indicator 292 may be an indicator light and may be configured to indicate charging status and power source changes, or may be configured to indicate messages, missed calls, notifications, and the like.

[0121] The SIM card interface 295 is configured to connect to a SIM card, which may be inserted into or removed from the SIM card interface 295 to effect contact or separation with the mobile phone 200.

[0122] It will be understood that in the embodiments of the present application, the mobile phone 200 may perform some or all of the steps of the embodiments of the present application. These steps or operations are merely examples. The mobile phone 200 may further perform other operations or variations of various operations. In addition, the steps may be performed in a different order than the order presented in the embodiments of the present application, and not all operations in the embodiments of the present application need to be performed. The embodiments of the present application may be implemented separately or in any combination. This is not limited in the present application.

[0123] With reference to the foregoing, an embodiment of the present application provides a dynamic range mapping method for panoramic video. As shown in Figure 3, the method includes steps 301 to 305.

[0124] Step 301: An electronic device (i.e., a display end) projects any frame of a 3D panoramic image in a panoramic video onto the surface of a polyhedron to obtain a 2D panoramic projection image, and extends the polyhedron into a 2D plane.

[0125] It should be understood that after obtaining the bitstream of the panoramic video (for example, after receiving the bitstream of the panoramic video from the generating end), the displaying end performs video decoding to obtain the panoramic video. Each frame of the 3D panoramic image of the panoramic video can also be called the 3D spherical representation panorama of the panoramic video.

[0126] Optionally, the polyhedron onto which the three-dimensional panoramic image is projected as the two-dimensional panoramic projection image comprises at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0127] In an embodiment of the present application, the process of using a polyhedron to project a 3D panoramic image into a 2D panoramic projection image includes the steps of: placing a sphere (the 3D panoramic image is represented by a 3D sphere) capable of representing a 3D panoramic image on a polyhedron, so that the sphere is an inscribed sphere of the polyhedron (it should be understood that the center of the polyhedron coincides with the spherical center of the sphere); connecting the sphere center to any point (pixel point) on the sphere and extending a connecting line to intersect with the surface of the polyhedron, where the point obtained by projecting the point on the sphere onto the surface of the polyhedron is the intersection point of the extension of the connecting line connecting the sphere center to the point on the sphere and the surface of the polyhedron; or connecting a point on the surface of the polyhedron to the sphere center, where the connecting line intersects with the sphere at a point, and the intersection point on the sphere and the point on the polyhedron are in a mutual projection relationship. It should be understood that the pixel value of the point on the sphere is the pixel value of the intersection point on the surface of the polyhedron. Thus, in the aforementioned projection process, all pixels on the 3D sphere are projected to obtain the projected pixels on each surface of the polyhedron (interpolation is performed for the projected pixels on the 2D plane based on different cases). Finally, the polyhedron is expanded to the 2D plane according to certain rules to obtain a 2D panoramic projection image.

[0128] Please refer to FIG. 4. An example in which the polyhedron is a regular hexahedron is used, and the regular hexahedron is denoted as ABCDEFGH. As shown in FIG. 4(a), the sphere representing the 3D panoramic image is a sphere inscribed in the regular hexahedron ABCDEFGH. For a point M' on the plane ABCD on the plane A'B'C'D' (the plane ABCD is the base of the regular hexahedron ABCDEFGH, and the plane A'B'C'D' is the projection plane of the plane ABCD), the point M' is connected to the sphere center O, and the connecting line intersects with the sphere at the point M. Thus, the pixel value of the point M' is the pixel value of the point M. Similarly, the pixel values ​​of all points on the plane ABCD can be obtained, and the pixel points on the plane ABCD form a surface image of the plane ABCD. According to the projection method onto the plane ABCD, the pixel points of the 3D panoramic image are projected onto other faces of the regular hexahedron to obtain other surface images. As shown in FIG. 4(b), after the 3D panoramic image is projected, a surface image of the upper surface EFGH, a surface image of the lower surface ABCD, a surface image of the front surface HGCD, a surface image of the rear surface EFBA, a surface image of the left surface EHDA, and a surface image of the right surface GFBC can be obtained. Furthermore, FIG. 4(b) is extended to a 2D plane to obtain a 2D panoramic image shown in FIG. 4(c). It can be seen that the 2D panoramic projection image has an irregular shape. The surface image of the surface of the polyhedron may be called an image of a region in the 2D panoramic projection image, or a sub-image of the 2D panoramic projection image.

[0129] Fig. 5 shows the projection effect when using a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, and a regular decahedron. After projecting a 3D panoramic image using the regular polyhedrons shown in the first row of Fig. 5, a corresponding 2D panoramic projection image (projection result shown in the second row of Fig. 5) is obtained.

[0130] In the embodiment of the present application, during subsequent processing, the irregular projection image shown in the second line of FIG. 5 may be directly processed, or the irregular image may be converted into a regular image for processing. In one implementation, a rectangular region that surrounds the irregular image and has a minimum area may be processed (see the third line of FIG. 5, where the rectangle shown in the dashed box is the rectangle that surrounds the irregular image and has a minimum area). Optionally, another region other than the projection region in the rectangular region may be filled, for example, with a preset pixel value. In another implementation, the surface image in the irregular projection image may be stitched into a projection image of a regular shape (e.g., a rectangle). In this way, pixel filling does not need to be performed.

[0131] It should be noted that in the embodiment of the present application, the polyhedron is not limited to a regular polyhedron. In other words, a 2D panoramic projection of a 3D panoramic image may contain several polygons of different sizes.

[0132] Optionally, in the embodiment of the present application, the electronic device can use another method to convert the 3D panoramic image into a 2D panoramic projection image. For example, the 3D panoramic image (spherical) is expanded as it is. The 3D panoramic image of the earth is used as an example. The 3D panoramic image of the earth is expanded into a 2D panoramic image. Specifically, the area adjacent to the N pole and the S pole in the 3D panorama is expanded to obtain the content of the N pole and the S pole area in the 2D panoramic image.

[0133] Step 302: The electronic device divides a 2D panoramic projection image corresponding to any frame of the 3D panoramic image in the panoramic video to obtain multiple regions of the 2D panoramic projection image.

[0134] Each of the multiple regions satisfies a preset condition, and the preset condition includes at least one of the following condition 1 and condition 2.

[0135] Condition 1: Pixels that are adjacent in the 2D panoramic projection image and non-adjacent in the 3D panoramic image are not located in the same region, adjacent pixel points in the 2D panoramic projection image include pixels on at least two adjacent projection planes of the 3D panoramic image, and non-adjacent pixels in the 3D panoramic image include pixels in the 3D panoramic image that correspond to at least two non-adjacent projection planes.

[0136] An example is used in which the polyhedron onto which the 3D panoramic image is projected is a regular hexahedron. As shown in FIG. 6, the six faces of the regular hexahedron are numbered 0, 1, 2, 3, 4, 5, and 6 in the order of front, back, top, bottom, left, and right. In the 2D panoramic projection image obtained by projecting the 3D panoramic image of FIG. 6, in each surface image (square), the arrows indicated by the edges of the squares indicate edges corresponding to edges in the polyhedron. In FIG. 6, edges corresponding to arrows marked with the same letter are the same edge in the polyhedron. The edge corresponding to the arrow marked with the letter a in surface image 0 (which may be called projection surface 0) and the edge corresponding to the arrow marked with the letter a in surface image 3 are the same edge in the regular hexahedron, i.e., the edge where surface 0 and bottom surface 3 intersect.

[0137] 6, for two adjacent projection planes of a 3D panoramic image, for example, projection plane 0 and projection plane 1, in the 2D panoramic projection image, projection plane 0 is adjacent to projection plane 1, and some pixels of projection plane 0 are adjacent to some pixels of projection plane 1. For example, pixels on an edge corresponding to an arrow marked with the letter a on projection plane 0 are adjacent to pixels on an edge corresponding to an arrow marked with the letter g on projection plane 1. The pixels of the 3D panoramic image corresponding to projection plane 0 and the pixels of the 3D panoramic image corresponding to projection plane 1 are non-adjacent pixels in the 3D panoramic image.

[0138] When the two-dimensional panoramic projection image obtained in FIG. 6 is divided, it is necessary to ensure that adjacent pixels in the two-dimensional panoramic projection image and non-adjacent pixels in the three-dimensional panoramic image are not located in the same region. For example, the pixels on the edge corresponding to the arrow marked with the letter a on the projection plane 0 in FIG. 6 and the pixels on the edge corresponding to the arrow marked with the letter g on the projection plane 1 are adjacent pixels in the two-dimensional panoramic projection image. However, the pixels on the edge corresponding to the arrow marked with the letter a on the projection plane 0 and the pixels on the edge corresponding to the arrow marked with the letter g on the projection plane 1 are non-adjacent pixels in the three-dimensional panoramic image. Therefore, it is necessary to ensure that the pixels on the edge corresponding to the arrow marked with the letter a on the projection plane 0 and the pixels on the edge corresponding to the arrow marked with the letter g on the projection plane 1 are not divided into the same region. In other words, when the two-dimensional panoramic projection image is divided horizontally, the division cannot be performed across the line where the projection plane 0 and the projection plane 1 intersect, i.e., the pixels on both sides of the line cannot be divided into one region. Similarly, the pixels on either side of the line where projection plane 4 and projection plane 3 intersect cannot be divided into the same region, and the pixels on either side of the line where projection plane 5 and projection plane 2 intersect cannot be divided into the same region.

[0139] Condition 2: Non-adjacent pixels in the 3D panoramic image are not located in the same region, and the non-adjacent pixels in the 3D panoramic image include pixels corresponding to at least two non-adjacent projection planes in the 3D panoramic image.

[0140] Please refer to an example of condition 1. The pixel of the 3D panoramic image corresponding to the projection plane 0 in FIG. 6 and the pixel of the 3D panoramic image corresponding to the projection plane 1 are non-adjacent pixels in the 3D panoramic image. When the 2D panoramic projection image is divided, it is necessary to ensure that the pixel of the projection plane 0 and the pixel of the projection plane 1 are not divided into the same region. For example, the pixel on the edge corresponding to the arrow marked with the letter a on the projection plane 0 and the pixel on the edge corresponding to the arrow marked with the letter g on the projection plane 1 are not divided into the same region. In other words, when the 2D panoramic projection image is divided horizontally, the division cannot be performed across the line where the projection plane 0 and the projection plane 1 intersect, that is, the pixels on both sides of the line cannot be divided into one region. Similarly, the pixels on both sides of the line where the projection plane 4 and the projection plane 3 intersect cannot be divided into the same region, and the pixels on both sides of the line where the projection plane 5 and the projection plane 2 intersect cannot be divided into the same region.

[0141] In the embodiment of the present application, in the process of dividing a two-dimensional panoramic projection image, when each region obtained by division meets a preset condition, it can be ensured that pixels with similar luminance (similar luminance indicates similar dynamic range) are divided into the same region, and pixels with large luminance differences (large luminance difference indicates large dynamic range difference) are not divided into the same region. For example, in the three-dimensional panoramic projection image of FIG. 6, the pixel corresponding to projection plane 2 may be a sky pixel, and the pixel corresponding to projection plane 3 may be a ground pixel. Since the luminance of the sky pixel is significantly different from the luminance of the ground pixel, the pixel of projection plane 2 and the pixel of projection plane 5 are not located in the same region in the process of dividing a two-dimensional panoramic projection image. Since the dynamic ranges of pixels in the same region are similar, dynamic range mapping may be performed for pixels in the same region based on the same dynamic range mapping information. Since the dynamic ranges of pixels in different regions may be significantly different, dynamic range mapping may be performed for pixels in different regions based on different dynamic range mapping information. Therefore, after the two-dimensional panoramic projection image is divided into multiple regions by this division method, a subsequent process (i.e., dynamic range mapping) is performed for each region. The characteristics of the pixels within each region can be adaptively adapted in the process to improve processing efficiency.

[0142] Optionally, the electronic device may divide the two-dimensional panoramic projection image with different division methods. In the embodiment of the present application, the electronic device may divide the two-dimensional panoramic projection image by using any of step 3021, step 3022, or step 3023 and step 3024.

[0143] Referring to FIG. 3, as shown in FIG. 7, the method for dividing a two-dimensional panoramic projection image provided in an embodiment of the present application particularly includes step 3021 (i.e., step 302 is implemented using step 3021).

[0144] Step 3021: The electronic device divides the two-dimensional panoramic projection image using a first division method to obtain multiple regions of the two-dimensional panoramic projection image, where the first division method is a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron into regions.

[0145] The regular hexahedron shown in FIG. 6 is still used as an example. Please refer to FIG. 8. After the regular hexahedron is used to project the 3D panoramic projection image, the obtained 2D panoramic projection image includes six projection planes. The first division method is a method of dividing the pixels belonging to the same surface of the polyhedron into regions in the 2D panoramic projection image, that is, each projection plane of the 2D panoramic projection image is used as a region. As shown in FIG. 8, the 2D panoramic projection image may be divided into six regions, region A, region B, region C, region D, region E, and region F in FIG. 8, by the first division method. Please refer to FIG. 8. Each of the multiple regions obtained by dividing the 2D panoramic projection image by the first division method satisfies a preset condition (including condition 1 or condition 2), that is, pixels that are adjacent in the 2D panoramic projection image and not adjacent in the 3D panoramic image are not located in the same region.

[0146] Referring to FIG. 3, as shown in FIG. 9, another scheme for dividing a two-dimensional panoramic projection image provided in an embodiment of the present application specifically includes step 3022 (i.e., step 302 is implemented using step 3022).

[0147] Step 3022: The electronic device divides the two-dimensional panoramic projection image using a second division scheme to obtain multiple regions of the two-dimensional panoramic projection image, the second division scheme being a scheme of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of a polyhedron (used for projection) into regions to obtain multiple intermediate regions, and further dividing at least one of the multiple intermediate regions.

[0148] The regular hexahedron shown in FIG. 6 is still used as an example. See FIG. 10. After the regular hexahedron is used to project the 3D panoramic projection image, the obtained 2D panoramic projection image includes six projection planes. The second division scheme is a scheme in which the pixels in the 2D panoramic projection image belonging to the same surface of the polyhedron (used for projection) are divided into regions to obtain multiple intermediate regions, and at least one of the multiple intermediate regions is further divided. In other words, the 2D panoramic projection image is first divided in the first division scheme to obtain multiple regions as multiple intermediate regions, and then one or more of the multiple intermediate regions are divided to obtain multiple regions as the final division result. As shown in FIG. 10, after the 2D panoramic projection image is divided in the first division scheme, six intermediate regions are obtained, which are intermediate region a, intermediate region b, intermediate region c, intermediate region d, intermediate region e, and intermediate region f. Then, each of the six intermediate regions is divided in half horizontally to finally obtain 12 regions. In other words, the 2D panoramic projection image can be divided into 12 regions, including region A, region B, region C, region D, region E, region F, region G, region H, region I, region J, region K, and region L in Fig. 10, in the second division manner. See Fig. 10. Each of the multiple regions obtained by dividing the 2D panoramic projection image in the second division manner satisfies a preset condition (including condition 1 or condition 2), i.e., pixels that are adjacent in the 2D panoramic projection image and not adjacent in the 3D panoramic image are not located in the same region.

[0149] Referring to FIG. 3, as shown in FIG. 11, yet another scheme for dividing a two-dimensional panoramic projection image provided in an embodiment of the present application specifically includes steps 3023 and 3024 (i.e., step 302 is implemented using steps 3023 and 3024).

[0150] Step 3023: The electronic device divides the two-dimensional panoramic projection image using a third division scheme to obtain multiple intermediate regions of the two-dimensional panoramic projection image, where the third division scheme is a scheme of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region.

[0151] In an embodiment of the present application, the process of clustering all pixels in a two-dimensional panoramic projection image in a third division manner includes:

[0152] A clustering center is first determined, and the clustering center is one pixel. Optionally, the two-dimensional panoramic projection image may be first divided into a plurality of regions in a preset division manner, and a pixel value at the center position of each region is determined, and the pixel value at the center position of each region is used as a clustering center. For example, assume that the two-dimensional panoramic projection image is an image obtained by projection using a regular hexahedron shown in FIG. 6. If the two-dimensional panoramic projection image is divided into six regions, the two-dimensional panoramic projection image may be divided into six regions in a first division manner, and a pixel value at the center position of each region is determined to obtain six clustering centers. If the two-dimensional panoramic projection image is divided into twelve regions, the two-dimensional panoramic projection image may be divided into twelve regions in a second division manner, and a pixel value at the center position of each region is determined to obtain twelve clustering centers.

[0153] Then, according to the preset clustering algorithm and the determined clustering centers, a calculation is performed for the distance between every pixel point and each clustering center (the distance is calculated based on the coordinates of the pixel and the coordinates of the pixel corresponding to the clustering center), and the distance is D iThen, a calculation is performed on the difference between the luminance values ​​of all pixel points and the luminance values ​​of pixel points corresponding to the clustering centers (a calculation is performed on the difference between the luminance values ​​of all pixel points and the average luminance value of the pixel points of the category corresponding to each clustering center (i.e., the category obtained by dividing with the preset division method), a calculation is performed on the difference between the color values ​​of all pixel points and the color values ​​of pixel points corresponding to the clustering centers, or a calculation is performed on the difference between the color values ​​of all pixel points and the average color value of the pixel points of the category corresponding to each clustering center (i.e., the category obtained by dividing with the preset division method)), and the difference is E i Each pixel is denoted by a distance D i and E i According to the weighted values ​​of the weights, the corresponding clustering centers are divided into categories to obtain a clustering result. The category with the smallest weighted value is the category corresponding to the pixel, and the pixel points of each category in the clustering result form a region.

[0154] It should be noted that dividing the two-dimensional panoramic projection image according to the aforementioned clustering method in step 3023 cannot guarantee whether the obtained intermediate regions meet the preset condition (including condition 1 or condition 2). Therefore, the intermediate regions obtained by clustering need to be determined to determine whether the intermediate regions meet the preset condition. If all of the intermediate regions meet the preset condition, the intermediate regions are used as the final division result, that is, the intermediate regions are used as the regions obtained by dividing the two-dimensional panoramic projection image. If there is an intermediate region among the intermediate regions that does not meet the preset condition, the electronic device continues to execute step 3024.

[0155] Step 3024: The electronic device divides the intermediate regions that do not satisfy the preset condition among the multiple intermediate regions in a fourth division manner to obtain multiple regions. For any intermediate region that does not satisfy the preset condition, the fourth division manner is a manner of dividing the intermediate region along the boundary lines of at least two surfaces of a polyhedron included in the intermediate region.

[0156] An example is still used in which the two-dimensional panoramic projection image is an image obtained by projection onto a regular hexahedron shown in FIG. 6. See FIG. 12. Assume that the number of categories set during clustering is 7. After all the pixels of the two-dimensional panoramic projection image are clustered, seven intermediate regions are obtained, including intermediate region A, intermediate region B, intermediate region C, intermediate region D, intermediate region E, intermediate region F, and intermediate region G. Since intermediate region E does not satisfy the preset condition (other intermediate regions do satisfy the preset condition), intermediate region E needs to be further divided. As shown in FIG. 12, it can be seen that intermediate region E includes some pixels of two projection planes (projection plane 0 and projection plane 1), and intermediate region E may be divided into region I and region H along the line where projection plane 0 and projection plane 1 intersect in intermediate region E. In this way, the two-dimensional panoramic projection image is finally divided into eight regions, including region A, region B, region C, region I, region D, region H, region F, and region G.

[0157] In the embodiments of the present application, based on the actual requirements for dividing a two-dimensional panoramic projection image, an appropriate division method may be selected from the three division methods mentioned above, or another division method other than the three division methods mentioned above may be used to divide the two-dimensional panoramic projection image, provided that the division method can ensure that each region obtained by the division meets the preset conditions.

[0158] Optionally, in the embodiment of the present application, the division manner of the two-dimensional panoramic projection image may be pre-set in the electronic device (for example, agreed upon according to a protocol), for example, set to the first division manner, the second division manner, or the third division manner. Alternatively, the division manner of the two-dimensional panoramic projection image is transferred in the bitstream (specifically, metadata).

[0159] Referring to the description of the division method of the 2D panoramic projection image in the above embodiment, it can be seen that the first division method is related to the second division method, and the division result of the second division method is the sub-division result based on the first division method.

[0160] In one embodiment, when the candidate division schemes of the two-dimensional panoramic projection image include a first division scheme and a second division scheme, whether further division needs to be performed after the division by the first division scheme can be determined based on the content of the two-dimensional panoramic projection image, i.e., determine to use the first division scheme or the second division scheme. Specifically, the following process is performed to determine whether further division needs to be performed on the division result of the first division scheme:

[0161] S1: Divide the two-dimensional panoramic projection image using a first division method to obtain multiple intermediate regions.

[0162] S2 through S5 are performed for each of the multiple intermediate regions to determine whether each intermediate region needs to be further divided.

[0163] S2: Calculate the histogram of the mid-region.

[0164] For example, the histogram of the middle region is denoted as HISA. In the embodiment of the present application, the histogram can be a gray scale histogram or a color histogram, which is not limited in the embodiment of the present application.

[0165] S3: Divide the intermediate region into multiple regions and calculate the histogram of each region.

[0166] Optionally, one intermediate region may be equally divided into four regions, each occupying one-fourth of the area of ​​the intermediate region, and the four regions obtained by the division are denoted as Region 0, Region 1, Region 2, and Region 3, respectively, and the histograms of the four regions are denoted as HISB[0], HISB[1], HISB[2], and HISB[3], respectively.

[0167] S4: Calculate the sum of the difference values ​​between the histogram of the intermediate region and the histogram of the region obtained by dividing the intermediate region.

[0168] The difference values ​​between the histogram of the intermediate region and the histogram of each region obtained by division are first calculated, and the difference values ​​are the differences between the histogram of the intermediate region multiplied by n and the corresponding elements of the histogram of each region. All the differences are summed up, where n is the number of regions obtained by dividing the intermediate region. For example, if the intermediate region is divided into four regions, n=4. The difference values ​​are then summed up.

[0169] The sum of the difference values ​​is denoted as DiffAB, DiffAB=DiffAB[0]+DiffAB[1]+DiffAB[2]+DiffAB[3] It is.

[0170] DiffAB[0] indicates the difference value between the histogram of the intermediate region and the histogram of region 0, DiffAB[1] indicates the difference value between the histogram of the intermediate region and the histogram of region 1, DiffAB[2] indicates the difference value between the histogram of the intermediate region and the histogram of region 2, and DiffAB[3] indicates the difference value between the histogram of the intermediate region and the histogram of region 3.

[0171] S5: If DiffAB>T1, where T1 is a preset threshold, the intermediate region needs to be further divided, i.e., the two-dimensional panoramic projection image is divided by the second division manner; or if DiffAB≦T1, the intermediate region does not need to be further divided, i.e., the two-dimensional panoramic projection image is divided by the first division manner.

[0172] If DiffAB>T, it indicates that the luminance difference between parts of the intermediate region is large and further segmentation needs to be performed.

[0173] Since the first division scheme is related to the second division scheme, in some cases, when the default division scheme is the first division scheme, a related identifier may be used in the bitstream to indicate whether further division is to be performed. When the identifier indicates that further division is not to be performed, the division result obtained in the first division scheme is the final result. When the identifier indicates that further division needs to be performed, the division result of the first division scheme is further divided.

[0174] In another embodiment, when the candidate decomposition schemes of a 2D panoramic projection image include a first decomposition scheme, a second decomposition scheme, and a third decomposition scheme, which decomposition scheme to be used may be determined by performing the following process.

[0175] S11: Divide the two-dimensional panoramic projection image into a first division method, a second division method, and a third division method.

[0176] S12: A histogram of each of the regions obtained by the first division method, the second division method, and the third division method and a histogram of the 2D panoramic projection image are calculated separately.

[0177] S13: The sum of the difference values ​​between the histograms of the multiple regions obtained by each division method and the histogram of the two-dimensional panoramic projection image is calculated.

[0178] For example, a two-dimensional panoramic projection image is divided by a first division method to obtain M regions, and the sum of the difference values ​​between the histograms of the multiple regions obtained by the first division method and the histogram of the two-dimensional panoramic projection image is denoted as DiffHISM. The calculation of the sum of the difference values ​​is similar to the process of S4. The difference value between the histogram of each region and the histogram of the two-dimensional panoramic projection image is first calculated, and the difference value is the difference between the histogram of each region multiplied by M and the corresponding element of the histogram of the two-dimensional panoramic projection image. All the differences are summed up, and the difference values ​​are summed up as follows: DiffHISM=DiffHISM[0]+DiffHISM[1]+…+DiffHISM[M]

[0179] DiffHISM[0] indicates the difference value between the histogram of area 0 obtained by the first division method and the histogram of the 2D panoramic projection image, DiffHISM[M] indicates the difference value between the histogram of area M obtained by the first division method and the histogram of the 2D panoramic projection image, and the remainder can be estimated by analogy.

[0180] The 2D panoramic projection image is divided by a second division method to obtain N regions, and the sum of the difference values ​​between the histograms of the multiple regions obtained by the second division method and the histogram of the 2D panoramic projection image is denoted as DiffHISN. The calculation of the sum of the difference values ​​is the same as the process of S4. The difference value between the histogram of each region and the histogram of the 2D panoramic projection image is first calculated, and the difference value is the difference between the histogram of the histogram of each region multiplied by N and the corresponding element of the histogram of the 2D panoramic projection image. All the differences are summed up, and the difference values ​​are summed up as follows: DiffHISN=DiffHISN[0]+DiffHISN[1]+…+DiffHISN[N]

[0181] DiffHISN[0] indicates the difference value between the histogram of area 0 obtained by the second division method and the histogram of the 2D panoramic projection image, DiffHISN[N] indicates the difference value between the histogram of area N obtained by the second division method and the histogram of the 2D panoramic projection image, and the remainder can be estimated by analogy.

[0182] The 2D panoramic projection image is divided by a third division method to obtain P regions, and the sum of the difference values ​​between the histograms of the multiple regions obtained by the third division method and the histogram of the 2D panoramic projection image is denoted as DiffHISP. The calculation of the sum of the difference values ​​is the same as the process of S4. The difference value between the histogram of each region and the histogram of the 2D panoramic projection image is first calculated, and the difference value is the difference between the histogram of each region multiplied by P and the corresponding element of the histogram of the 2D panoramic projection image. All the differences are summed up, and the difference values ​​are summed up as follows: DiffHISP=DiffHISP[0]+DiffHISP[1]+…+DiffHISP[P]

[0183] DiffHISN[0] indicates the difference value between the histogram of region 0 obtained by the third division method and the histogram of the 2D panoramic projection image, DiffHISP[P] indicates the difference value between the histogram of region P obtained by the third division method and the histogram of the 2D panoramic projection image, and the remainder can be estimated by analogy.

[0184] S14: The division scheme that maximizes the sum of the difference values ​​between the histograms is determined as the final division scheme.

[0185] The sums of the difference values ​​between the histograms, DiffHISM, DiffHISN, and DiffHISP, are compared, and the larger the sum of the difference values ​​between the histograms, the larger the luminance difference between the regions obtained by the division, indicating that the division is appropriate.

[0186] Optionally, the size of metadata in three division schemes may be further calculated, and the metadata includes dynamic mapping information corresponding to each region of the 2D panoramic projection image (details will be described in step 303). Then, based on the weighted value of the sum of the histogram difference values ​​corresponding to the division schemes described in S13 and the size of the metadata corresponding to the division schemes, it is determined which division scheme is used. For example, the size of the metadata corresponding to the first division scheme is denoted as Size1, the size of the metadata corresponding to the second division scheme is denoted as Size2, and the size of the metadata corresponding to the third division scheme is denoted as Size3, and calculations are performed for DiffHISM-Size1 (difference is denoted as err1), DiffHISN-Size2 (difference is denoted as err2), and DiffHISP[P]-Size3 (difference is denoted as err3), and err1, err2, and err3 are compared, and the division scheme with the largest difference is determined as the final division scheme.

[0187] Step 303: The electronic device acquires metadata of the two-dimensional panoramic projection image, the metadata including metadata information elements respectively corresponding to multiple regions of the two-dimensional panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information of the region.

[0188] It should be understood that in the embodiment of the present application, the concept of performing dynamic range mapping on a panoramic video is to project all 3D panoramic images contained in the panoramic video onto a 2D plane, perform dynamic range mapping on the obtained 2D panoramic projection images, and convert the 2D panoramic projection images obtained by dynamic range mapping into 3D panoramic images to obtain a panoramic video obtained by dynamic range mapping. In other words, the metadata of the obtained 2D panoramic projection images is used to perform dynamic range mapping of the 3D panoramic image (i.e., used to perform dynamic range mapping on the panoramic video), and the metadata of the 2D panoramic projection images may be considered as the metadata of the 3D panoramic image (or the metadata of the panoramic video). The metadata referred to in the following embodiment is the data used for dynamic range mapping.

[0189] Optionally, the method for the electronic device to obtain the metadata of the 2D panoramic projection image may be a step (generation end) of the electronic device receiving the metadata of the 2D panoramic projection image from another electronic device. After generating the metadata, the generating end encodes the metadata into a bitstream and sends the bitstream to the display end, so that the display end receives the bitstream and analyzes the bitstream to obtain the metadata, as can be seen from the description of the above embodiment.

[0190] In the embodiment of the present application, the two-dimensional panoramic projection image is divided into multiple regions, ie, the two-dimensional panoramic projection image includes multiple regions, and each region corresponds to one metadata information element.

[0191] In an embodiment of the present application, the metadata information element includes dynamic mapping information, and the dynamic mapping information includes dynamic mapping parameters, and dynamic range mapping can be performed on the pixel based on the dynamic mapping parameters. Optionally, the format of the dynamic mapping information may be histogram information or dynamic mapping curve information of the ST2094-40 standard, or may be dynamic mapping curve information of the ST2094-10 standard. This is not limited in the embodiment of the present application.

[0192] It can be seen from the description of the above embodiment that the division manner of the 2D panoramic projection image can be transmitted in a bitstream. Optionally, when the bitstream is used to transmit the division manner of the 2D panoramic projection image, the metadata of the 2D panoramic projection image further includes division manner indication information, and the division manner indication information indicates that the division manner of the 2D panoramic projection image is a first division manner, a second division manner, or a third division manner.

[0193] Step 304: The electronic device determines dynamic mapping information for each pixel within the current angle of view based on the dynamic mapping information for each of the multiple regions.

[0194] It should be understood that when a user watches a panoramic video, the user (head) cannot see all the content of the panoramic video in the current pose, but can only see the content of the video in a certain range. The range can be understood as the user's angle of view range. It can be seen that the user's angle of view range is related to the user's current viewing pose. In an embodiment of the present application, performing dynamic range mapping on a panoramic video is performing dynamic range mapping on the content within the angle of view range of the user who is currently watching the panoramic video.

[0195] In an embodiment of the present application, after obtaining the dynamic mapping information of the multiple regions, the electronic device may determine the dynamic mapping information of each pixel in the current angle of view range in any one of four manners based on the dynamic mapping information of the multiple regions. Specifically, for one of the pixels, several manners for determining the dynamic mapping information of the pixel are described in detail.

[0196] Method 1: The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a plurality of regions within a first preset range, and the first preset range is a range centered on the pixel to be processed.

[0197] For example, the first preset range is a range of 3 pixels x 3 pixels centered on the pixel to be processed. It is assumed that the first preset range covers pixels in three regions (i.e., the pixels in the first preset range belong to the three regions), and all the dynamic mapping information of the three regions is used as the dynamic mapping information of the pixel to be processed.

[0198] Method 2: The dynamic mapping information of the pixel to be processed within the current angle of view is the dynamic mapping information of the region to which the pixel belongs in the 2D panoramic projection image, which corresponds to the viewpoint center of the current angle of view.

[0199] It should be understood that when a user watches a panoramic video, the user's current view angle range corresponds to the viewpoint center. Assuming that the viewpoint center is X and the pixel corresponding to the viewpoint center X in the two-dimensional projection image is X', the dynamic mapping information of the region to which X' belongs is used as the dynamic mapping information of the target pixel to be processed.

[0200] Method 3: The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of multiple regions within a second preset range, and the second preset range is a range centered on a pixel in the 2D panoramic projection image that corresponds to the viewpoint center of the current angle of view range.

[0201] Referring to Scheme 2, an example is still used in which the viewpoint center of the current view angle range is X. The pixel corresponding to the viewpoint center X in the two-dimensional projection image is X'. For example, the second preset range may be a range of 3 pixels x 3 pixels centered on X'. It is assumed that the second preset range covers the pixels in the two regions (i.e., the pixels in the first preset range belong to the two regions), and all the dynamic mapping information of the two regions is used as the dynamic mapping information of the pixel to be processed.

[0202] Method 4: The dynamic mapping information of one pixel in the current region is the dynamic mapping information of the current region.

[0203] In other words, the dynamic mapping information of the region to which the target pixel belongs is used as the dynamic mapping information of the target pixel.

[0204] Optionally, during practical application, the generating end may select an appropriate method from the four methods based on the actual requirements for determining the dynamic mapping information of the pixel to be processed, and transfer the selected method carried in the bitstream (metadata) to the display end. Alternatively, the generating end and the display end may predetermine the method to be used. This is not limited in the embodiment of the present application.

[0205] Step 305: The electronic device performs dynamic range mapping for each pixel within the current angle of view based on the dynamic mapping information of each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view range, and the two-dimensional panoramic projection image within the current angle of view range is used for display or subsequent processing.

[0206] Optionally, the dynamic mapping information in the embodiment of the present application is used to map the dynamic range of the pixel of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range, or the dynamic mapping information is used to map the dynamic range of the pixel of the two-dimensional panoramic projection image from the second dynamic range to the first dynamic range. The first dynamic range is greater than the second dynamic range. In other words, the dynamic range mapping performed on the pixel may be a mapping from a high dynamic range to a low dynamic range or a mapping from a low dynamic range to a high dynamic range, and may include, for example, some cases shown in Table 1 of the above embodiment.

[0207] For example, the dynamic mapping information is a dynamic mapping curve. In the embodiment of the present application, the dynamic mapping curve may include, but is not limited to, a sigmoidal curve, a cubic spline curve, a Bezier curve, etc.

[0208] A sigmoidal curve is used as an example: the curve is L'=F(L), where L denotes the pixel before dynamic range mapping and L' denotes the pixel after dynamic range mapping.

[0209] The formula for F(L) is:

number

[0210] Optionally, L and L' are normalized values. The normalization may be linear space normalization or nonlinear space normalization. In addition, the range of normalization is not limited in the embodiment of the present application. For example, the range of normalization may be 0 to 10000 nit, or 0.001 to 100000 nit.

[0211] Optionally, after dynamic range mapping is performed for each pixel within the current angle of view, the two-dimensional panoramic projection image may be displayed to obtain a two-dimensional panoramic projection image within the current angle of view. In some cases, inverse normalization may be performed on the pixels (L') of the two-dimensional panoramic projection image based on the display capability (maximum and minimum display values) of the display device to normalize the pixels to the range of the display capability of the display device.

[0212] In an embodiment of the present application, after the dynamic mapping information of each pixel in the current angle of view range is determined using step 305, the pixel may correspond to multiple dynamic mapping information (e.g., multiple dynamic mapping curves). After dynamic range mapping is performed on the pixel in step 305, one dynamic mapping result may be obtained for each dynamic mapping information. In this case, the multiple dynamic mapping results may be processed to obtain a final dynamic mapping result. The processing manner may include any one of the following:

[0213] Processing method 1: The median value of multiple dynamic mapping results is used as the final dynamic mapping result.

[0214] Processing method 2: A weighted average is performed on multiple dynamic mapping results to obtain the final dynamic mapping result.

[0215] Suppose the pixel to be processed includes n pieces of dynamic mapping information. n dynamic mapping results Result_1, Result_2, ..., and Result_n may be obtained, and the weighting coefficients corresponding to the n dynamic mapping results are W1, W2, ..., and W n and the final mapping result R is:

number

[0216] In an embodiment of the present application, when multiple dynamic mapping results are processed by processing method 2, the weighting coefficient corresponding to each dynamic mapping result may be a preset weighting coefficient, or may be a weighting coefficient determined based on the current pixel to be processed.

[0217] Determining the weighting factor based on the current pixel to be processed specifically includes calculating the distance between the pixel to be processed and the pixel at the center of the area corresponding to each dynamic mapping information, and calculating the weighting factor based on each distance. For example, the current pixel to be processed is Y, the dynamic mapping information of Y includes dynamic mapping information corresponding to three areas, and the pixels at the center of the three areas are Y1', Y2', and Y3', respectively. Calculation is performed on the distance D1 between Y and Y1', the distance D2 between Y and Y2', and the distance D3 between Y and Y3', and the weighting factor is determined based on D1, D2, and D3. Optionally, in the embodiment of the present application, it indicates that the smaller the distance between the pixel to be processed and the pixel at the center of the area corresponding to the dynamic mapping information, the larger the corresponding weighting factor.

[0218] For example, the method for determining weighting factors W1, W2, W3 based on D1, D2, D3 is as follows.

number

[0219] Optionally, referring to FIG. 3, as shown in FIG. 13, before the dynamic mapping information of each pixel in the current region is determined based on the dynamic mapping information of each of the multiple regions (i.e., before step 304), the dynamic range mapping method for panoramic video provided in the embodiment of the present application further includes step 306 and step 307.

[0220] Step 306: The electronic device obtains the correspondence between the region and the metadata information element.

[0221] Step 307: The electronic device determines metadata information elements corresponding to the multiple regions separately from the metadata according to the correspondence between the regions and the metadata information elements, so as to determine dynamic mapping information of each of the multiple regions.

[0222] Optionally, the correspondence between the regions and the metadata information elements may be a one-to-one relationship or a many-to-one relationship. In other words, one region of the two-dimensional panoramic projection image corresponds to one metadata information element, or multiple regions of the two-dimensional panoramic projection image correspond to one metadata information element. Optionally, when the luminance difference between multiple regions of the two-dimensional panoramic projection image is large and one type of dynamic mapping information is used for each region, the correspondence between the regions and the metadata information elements is a one-to-one relationship. When the luminance of some regions in the multiple regions is similar, some regions may use the same type of dynamic mapping information, and the correspondence between the regions and the metadata information elements is a many-to-one relationship.

[0223] In an embodiment of the present application, the correspondence between a region and a metadata information element is a correspondence between an index of the metadata information element and an index of the region, or the correspondence between a region and a metadata information element is a correspondence between an index of the metadata element and the coordinates of one or more pixels within the region.

[0224] For example, assume that a 2D panoramic projection image is divided into six regions (see FIG. 6). Table 2 shows an example of the correspondence between the indexes of metadata information elements and the indexes of regions.

[0225] [Table 2]

[0226] From Table 2, it can be seen that the correspondence relationship between the index of the metadata information element and the index of the region is essentially a traversal order relationship. See FIG. 6. The order of the six regions in the two-dimensional panoramic projection image is, from left to right and top to bottom, region 4, region 0, region 5, region 3, region 1, and region 2. Therefore, in the table of the correspondence relationship between the region and the metadata information element shown in Table 2, the six regions in the two-dimensional panoramic projection image are traversed from left to right and top to bottom in sequence, and the metadata information elements corresponding to the six regions separately can be determined in sequence in the metadata.

[0227] In one embodiment, the traversal order of the regions may be indicated in the bitstream. For example, an indication of the traversal order may be carried in metadata. The traversal order is not limited to the above order from left to right and top to bottom, but is specifically related to the location of each region in the 2D panoramic projection image.

[0228] In another example, assume that a two-dimensional panoramic projection image is divided into six regions. Table 3 shows an example of a correspondence between the index of a metadata information element and the coordinates of one or more pixels within a region.

[0229] [Table 3]

[0230] Referring to Table 3, in one embodiment, the coordinates of the pixels of the regions in Table 3 may also be carried in the metadata. Specifically, the coordinates of the pixels of each region are carried in the metadata information element corresponding to each region. After dividing the 2D panoramic projection image to obtain multiple regions, for one of the regions, the display end determines the pixels whose coordinates belong to the region according to the correspondence relationship shown in Table 3, and uses the metadata information element corresponding to the coordinate as the metadata information element of the region. For example, for region 1 of the six regions, the relationship between the pixels corresponding to each coordinate in Table 3 and region 1 is determined sequentially. It is determined that the pixel corresponding to the coordinate (x4, y4) belongs to region 1. Therefore, the metadata information element whose index is 4 and corresponding to the coordinate (x4, y4) is used as the metadata information element of region 1.

[0231] Optionally, in the embodiment of the present application, the traversal order of the multiple regions may be a preset order. Alternatively, the generating end determines a traversal order from the candidate traversal orders according to a related algorithm, adds an identifier to the metadata of the bitstream, and forwards the bitstream to the generating end to indicate the traversal order. For example, which traversal order is used may be determined using the following method:

[0232] S111: A histogram of each of the multiple regions is calculated.

[0233] S112: For each of the candidate traversal orders, calculate the sum of the difference values ​​between the histograms of adjacent regions.

[0234] Assume there are four candidate traversal orders: traversal order Z1, traversal order Z2, traversal order Z3, and traversal order Z4.

[0235] An example is still used in which a 2D panoramic projection image is divided into six regions. For a candidate traversal order (denoted as Z1) from left to right and top to bottom, the six regions are assumed to be, in sequence, region 4, region 0, region 5, region 3, region 1, and region 2. Calculations are performed on the difference between the histogram of region 4 and the histogram of region 0 (denoted as DiffHIS_1), the difference between the histogram of region 0 and the histogram of region 5 (denoted as DiffHIS_2), the difference between the histogram of region 5 and the histogram of region 3 (denoted as DiffHIS_3), the difference between the histogram of region 3 and the histogram of region 1 (denoted as DiffHIS_4), and the difference between the histogram of region 1 and the histogram of region 2 (denoted as DiffHIS_5). The sum of the differences is calculated to obtain DiffHIS_Z1. Similarly, DiffHIS_Z2, DiffHIS_Z3, and DiffHIS_Z4 can be obtained by calculation.

[0236] S113: In the candidate traversal order, the traversal scheme that minimizes the sum of the differences between the histograms is determined as the final traversal scheme.

[0237] For example, the values ​​of DiffHIS_Z1, DiffHIS_Z2, DiffHIS_Z3, and DiffHIS_Z4 are compared, and the traversal order that minimizes the sum of the differences between the histograms is determined as the final traversal scheme.

[0238] Optionally, in an embodiment of the present application, a preset algorithm may be used to determine whether the correspondence between multiple regions in a two-dimensional panoramic image and metadata information elements is a one-to-one relationship or a many-to-one relationship. For example, whether to use a one-to-one relationship or a many-to-one relationship may be determined using the following method:

[0239] S1111: Clustering multiple regions.

[0240] The number of clustered categories may be set based on requirements, for example, half the number of regions or a quarter of the number of regions. For example, if the total number of regions is 8, the 8 regions may be divided into 2 categories, which is equivalent to obtaining 2 large regions, or the 8 regions may be divided into 4 categories, which is equivalent to obtaining 4 large regions.

[0241] S1112: A sum of differences between the histogram of the region corresponding to the category and the histogram of the two-dimensional panoramic projection image is calculated, and a sum of differences between the histograms of the multiple original regions and the histogram of the two-dimensional panoramic image is calculated.

[0242] For example, in the clustering results, the sum of the differences between the histograms of the regions corresponding to each category and the histogram of the 2D panoramic projection image is denoted as DiffHIS_X, and the sum of the differences between the histograms of multiple original regions and the histogram of the 2D panoramic image is denoted as DiffHIS_org.

[0243] S1113: Determine whether the correspondence between the multiple regions and the metadata information elements is a one-to-one relationship or a many-to-one relationship based on two sums of the differences between the histograms.

[0244] For example, when eight regions are clustered to obtain two categories of regions, the sum of the differences between the histograms of the regions corresponding to each category of the clustering result and the histogram of the two-dimensional panoramic projection image is denoted as DiffHIS_X2, and the sum of the differences between the histograms of multiple original regions and the histogram of the two-dimensional panoramic image is denoted as DiffHIS_org. The determination method is as follows: if DiffHIS_org×(1-T2)>DiffHIS_X2, the correspondence relationship between the regions and the metadata information elements is a many-to-one relationship, or if DiffHIS_org×(1-T2)≦DiffHIS_X2, the correspondence relationship between the regions and the metadata information elements is a one-to-one relationship, and T2 is a preset threshold.

[0245] Optionally, referring to FIG. 13, after step 305, the dynamic range mapping method for panoramic video provided in the embodiment of the present application further includes step 308, as shown in FIG.

[0246] Step 308: The electronic device performs subsequent processing on the 2D panoramic projection image within the current angle of view range, the subsequent processing including performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range, and the 3D panoramic image is used for display.

[0247] In an embodiment of the present application, the 2D panoramic projection image obtained in step 305 can be used for display. Moreover, the 2D panoramic projection image can be further used for subsequent processing, for example, the 2D panoramic projection image is converted into a 3D panoramic image for display. Furthermore, after all the 3D panoramic images of the panoramic video are obtained, the panoramic video can be played.

[0248] In conclusion, an embodiment of the present application provides a dynamic range mapping method for panoramic video. An electronic device divides a two-dimensional panoramic projection image corresponding to a three-dimensional panoramic image to obtain a plurality of regions of the two-dimensional panoramic projection image. Each of the plurality of regions satisfies a preset condition, and the preset condition includes at least one of the following conditions: pixels that are adjacent in the two-dimensional panoramic projection image and are not adjacent in the three-dimensional panoramic image are not located in the same region, and pixels that are not adjacent in the three-dimensional panoramic image are not located in the same region. In addition, the electronic device obtains metadata of the two-dimensional panoramic projection image, and the metadata includes metadata information elements respectively corresponding to the plurality of regions of the two-dimensional panoramic projection image, and the metadata information element corresponding to one region includes dynamic mapping information of the region. Then, the electronic device determines dynamic mapping information of each pixel in a current angle of view range based on the dynamic mapping information of each of the plurality of regions, and performs dynamic range mapping for each pixel in the current angle of view range based on the dynamic mapping information of each pixel in the current angle of view range (the angle of view range of the user currently viewing the panoramic video) to obtain a two-dimensional panoramic projection image in the current angle of view range, and the two-dimensional panoramic projection image in the current angle of view range is used for display or subsequent processing. In the dynamic range mapping method for panoramic video provided in the embodiment of the present application, in relation to the characteristic of the large dynamic range of the panoramic video, the electronic device divides a 2D panoramic projection image corresponding to a 3D panoramic image to obtain regions, and performs dynamic range mapping for different regions, so that the dynamic range mapping can be effectively implemented on the panoramic video, thereby improving the display effect of the panoramic video.

[0249] Correspondingly, an embodiment of the present application provides a dynamic range mapping apparatus for panoramic video. The apparatus may be used in an electronic device. In an embodiment of the present application, the apparatus may be divided into functional modules based on the above-mentioned method example. For example, a functional module corresponding to a function may be obtained by division, or two or more functions may be integrated into one processing module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. It should be noted that in the embodiment of the present invention, the division of modules is an example and is merely a division of logical functions. In actual implementation, another division scheme may be used.

[0250] When each functional module is obtained by division based on each corresponding function, FIG. 15 is a diagram of a possible structure of a dynamic range mapping device for panoramic video in the above embodiment. As shown in FIG. 15, the device includes a division module 1501, an acquisition module 1502, a determination module 1503, and a processing module 1504. The division module 1501 is configured to divide a two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image, for example, to perform step 302 in the method embodiment. The two-dimensional panoramic projection image is a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in the panoramic video, and each of the plurality of regions satisfies a preset condition, and the preset condition includes at least one of the following conditions: adjacent pixels in the two-dimensional panoramic projection image and non-adjacent pixels in the three-dimensional panoramic image are not located in the same region, and non-adjacent pixels in the three-dimensional panoramic image are not located in the same region. The pixel points adjacent in the 2D panoramic projection image include pixels on at least two adjacent projection planes of the 3D panoramic image, and the pixels that are not adjacent in the 3D panoramic image include pixels that are in the 3D panoramic image and correspond to at least two non-adjacent projection planes. The acquisition module 1502 is configured to acquire metadata of the 2D panoramic projection image, for example, to perform step 302 and step 303 in the method embodiment. The metadata includes metadata information elements that respectively correspond to a plurality of regions of the 2D panoramic projection image, and the metadata information element that corresponds to one region includes dynamic mapping information of the region. The determination module 1503 is configured to determine dynamic mapping information of each pixel in the current angle of view range based on the dynamic mapping information of each of the plurality of regions, for example, to perform step 304 in the method embodiment. The processing module 1504 is configured to perform dynamic range mapping for each pixel in the current angle of view range based on the dynamic mapping information of each pixel in the current angle of view range to acquire a 2D panoramic projection image in the current angle of view range, for example, to perform step 305 in the method embodiment.The 2D panoramic projection image within the current field of view is used for display or further processing.

[0251] Optionally, the dynamic range mapping device for panoramic video provided in the embodiment of the present application further includes a transformation module 1505. The transformation module 1505 is configured to project the three-dimensional panoramic image onto a surface of a polyhedron, and extend the polyhedron into a two-dimensional plane to obtain a two-dimensional panoramic projection image, for example, to perform step 301 in the method embodiment. The polyhedron includes at least one of a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron.

[0252] Optionally, the division module 1501 is specifically configured to divide the two-dimensional panoramic projection image in a first division manner, for example, to perform step 3021 in the method embodiment, to obtain multiple regions of the two-dimensional panoramic projection image. The first division manner is a manner of dividing pixels belonging to the same surface of a polyhedron into regions in the two-dimensional panoramic projection image.

[0253] Optionally, the division module 1501 is specifically configured to divide the two-dimensional panoramic projection image in a second division manner, for example, to perform step 3022 in the method embodiment, to obtain multiple regions of the two-dimensional panoramic projection image, the second division manner being a manner of dividing pixels in the two-dimensional panoramic projection image belonging to the same surface of a polyhedron into regions, and further dividing at least one of the multiple intermediate regions to obtain multiple intermediate regions.

[0254] Optionally, the division module 1501 is specifically configured to divide the two-dimensional panoramic projection image in a third division manner to obtain a plurality of intermediate regions of the two-dimensional panoramic projection image, the third division manner being a manner of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region, and to divide the intermediate regions that do not satisfy the preset condition in the plurality of intermediate regions in a fourth division manner to obtain a plurality of intermediate regions, for example, performing step 3023 and step 3024 in the method embodiment. For any intermediate region that does not satisfy the preset condition, the fourth division manner is a manner of dividing the intermediate region along the boundary line of at least two surfaces of a polyhedron included in the intermediate region.

[0255] Optionally, the acquiring module 1502 is further configured to acquire a correspondence relationship between the region and the metadata information element, for example, performing step 306 in the method embodiment. The determining module 1503 is further configured to determine metadata information elements corresponding to the multiple regions separately from the metadata according to the correspondence relationship between the region and the metadata information element, for example, performing step 307 in the method embodiment, so as to determine dynamic mapping information of each of the multiple regions.

[0256] Optionally, the acquisition module 1502 is specifically configured to receive metadata of the two-dimensional panoramic projection image from another electronic device.

[0257] Optionally, the transformation module 1505 is further configured to perform subsequent processing on the 2D panoramic projection image within the current angle of view range, for example, performing step 308 in the method embodiment. The subsequent processing includes performing a 3D transformation on the 2D panoramic projection image within the current angle of view range to obtain a 3D panoramic image within the current angle of view range. The 3D panoramic image is used for display.

[0258] The modules of the dynamic range mapping device for panoramic video may be further configured to perform other operations in the method embodiments. Any relevant contents of the steps in the method embodiments may be cited in the functional description of the corresponding functional modules. Details are not repeated here.

[0259] When the integration unit is used, FIG. 16 is a diagram of another possible structure of the dynamic range mapping device for panoramic video in the aforementioned embodiment. As shown in FIG. 16, the dynamic range mapping device for panoramic video provided in the embodiment of the present application may include a processing module 1601 and a communication module 1602. The processing module 1601 may be configured to control and manage the operation of the device. For example, the processing module 1601 may be configured to support the device in performing steps 301 to 305, 306, 307, and 308 in the method embodiment and / or in performing another process of the technology described herein. The communication module 1602 is configured to support the device to communicate with another network entity, for example, to communicate with another electronic device (generator). Optionally, as shown in FIG. 16, the dynamic range mapping device for panoramic video may further include a storage module 1603 configured to store the program code and data of the device, for example, to store metadata.

[0260] The processing module 1601 may be a processor or a controller (e.g., the processor 210 shown in FIG. 2). The processor may alternatively be a combination of processors performing computing functions, such as a combination of one or more microprocessors or a combination of a DSP and a microprocessor. The communication module 1602 may be a transceiver, a transceiver circuit, a communication interface, etc. (e.g., the mobile communication module 250 or the wireless communication module 260 shown in FIG. 2). The storage module 1603 may be a memory (e.g., the internal memory 221 shown in FIG. 2).

[0261] When the processing module 1601 is a processor, the communication module 1602 is a transceiver, and the storage module 1603 is a memory, the processor, the transceiver, and the memory may be connected via a bus. The bus may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus may be classified into an address bus, a data bus, a control bus, etc.

[0262] For further details of implementing the aforementioned functions by modules included in the dynamic range mapping device for panoramic video, please refer to the description of the method embodiment, and the details will not be repeated here.

[0263] All embodiments herein are described step by step, with reference to these embodiments for the same or similar parts of the embodiments, and each embodiment focusing on the differences from the other embodiments.

[0264] All or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When a software program is used to implement the embodiments, the embodiments may be fully or partially implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the procedures or functions according to the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired (e.g., coaxial cable, optical fiber, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio wave, or microwave) manner. The computer-readable storage medium may be any available medium accessible by a computer, or may be a data storage device that integrates one or more available media, such as a server or a data center. The usable media may be magnetic media (e.g., a floppy disk, magnetic disk, or magnetic tape), optical media (e.g., a digital video disc (DVD)), semiconductor media (e.g., a solid state drive (SSD)), and the like.

[0265] The above description of the embodiment allows those skilled in the art to understand that the above division into functional modules is used as an example for illustration for the sake of convenience. In practical application, the above functions can be allocated and implemented into different modules based on requirements, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above functions. For detailed operation processes of the above systems, devices and units, please refer to the corresponding processes in the method embodiments, and details will not be described again in this specification.

[0266] In some embodiments provided in the present application, it should be understood that the disclosed system, device, and method may be implemented in other ways. For example, the described device embodiment is merely an example. For example, the division into modules or units is merely a logical division of functions, and may be other divisions in actual implementation. For example, multiple units or assemblies may be combined or integrated into other systems, or some functions may be ignored or not performed. In addition, the shown or described mutual couplings or direct couplings or communication connections may be implemented through some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0267] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, and may be located in one location or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0268] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0269] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present application may be essentially, or the part that contributes to the prior art, or all or part of the technical solution may be implemented in the form of a software product. The software product is stored in a storage medium and includes some instructions for instructing a computer device or processor (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes any medium that can store program code, such as a flash memory, a removable hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0270] The above description is merely a specific embodiment of the present application, and is not intended to limit the protection scope of the present application. Any variation or replacement within the technical scope disclosed in the present application shall fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims. [Explanation of symbols]

[0271] 0 area 1 Antenna, dynamic range, processing method, area 2 Antenna, dynamic range, processing method, area 3 methods, areas 4 methods, areas 5 areas 200 Mobile Phones 210 Processor 220 External Memory Interface 221 Internal Memory 230 Interface 240 Charging Management Module 241 Power Management Module 242 Battery 250 Mobile communication module 260 Wireless communication module 270 Audio Module 270A Speaker 270B Receiver 270C Microphone 270D Headset Jack, Headset Interface 280 Sensor Module 290 Button 291 Motor 292 Indicator 293 Camera 294 Display 295 SIM card interface 400 Mobile Phones 1501 Split Module 1502 Acquisition Module 1503 Decision Module 1504 Processing Module 1505 Conversion Module 1601 Processing Module 1602 Communication Module 1603 Memory Module

Claims

1. 1. A dynamic range mapping method for a panoramic video applied to an electronic device, the method comprising: a step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image, the two-dimensional panoramic projection image being a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in a panoramic video, each of the plurality of regions satisfying a preset condition, the preset condition including at least one of the following conditions: adjacent pixels in the two-dimensional panoramic projection image and non-adjacent pixels in the three-dimensional panoramic image are not located in the same region, non-adjacent pixels in the three-dimensional panoramic image are not located in the same region, the pixel points adjacent in the two-dimensional panoramic projection image include pixels on at least two adjacent projection planes of the three-dimensional panoramic image, and the pixels non-adjacent in the three-dimensional panoramic image include pixels in the three-dimensional panoramic image and corresponding to at least two non-adjacent projection planes; acquiring metadata of the 2D panoramic projection image, the metadata including metadata information elements corresponding to the regions of the 2D panoramic projection image separately, a metadata information element corresponding to a region including dynamic mapping information of the region; determining dynamic mapping information for each pixel within a current angle of view based on the dynamic mapping information for each of the plurality of regions; performing dynamic range mapping for each pixel within the current angle of view based on the dynamic mapping information of each pixel within the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, the two-dimensional panoramic projection image within the current angle of view being used for display or subsequent processing; A method comprising:

2. The method further comprising: projecting the 3D panoramic image onto a surface of a polyhedron to expand the polyhedron into a two-dimensional plane to obtain the 2D panoramic projection image, the polyhedron comprising at least one of the following: a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron; The method of claim 1, further comprising:

3. said step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image further comprising: A step of dividing the two-dimensional panoramic projection image by a first division method to obtain the plurality of regions of the two-dimensional panoramic projection image, the first division method being a method of dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of the polyhedron into regions.

3. The method of claim 1 or 2, comprising:

4. said step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image further comprising: Dividing the two-dimensional panoramic projection image by a second division scheme to obtain the plurality of regions of the two-dimensional panoramic projection image, the second division scheme being a scheme for dividing pixels in the two-dimensional panoramic projection image that belong to the same surface of the polyhedron into regions to obtain a plurality of intermediate regions, and further dividing at least one of the plurality of intermediate regions.

3. The method of claim 1 or 2, comprising:

5. said step of dividing the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image further comprising: Dividing the two-dimensional panoramic projection image by a third division method to obtain a plurality of intermediate regions of the two-dimensional panoramic projection image, the third division method being a method of clustering all pixels in the two-dimensional panoramic projection image and dividing pixels belonging to the same category into the same intermediate region; a step of dividing the intermediate regions that do not satisfy the preset condition among the plurality of intermediate regions by a fourth division scheme to obtain the plurality of regions, wherein for any intermediate region that does not satisfy the preset condition, the fourth division scheme is a scheme of dividing the intermediate region along boundaries of at least two surfaces of the polyhedron that are included in the intermediate region; 3. The method of claim 1 or 2, comprising:

6. the first division scheme, the second division scheme, or the third division scheme is preset in the electronic device; 6. The method according to any one of claims 3 to 5.

7. The metadata of the two-dimensional panoramic projection image further includes division scheme instruction information, and the division scheme instruction information indicates that the division scheme of the two-dimensional panoramic projection image is the first division scheme, the second division scheme, or the third division scheme.

6. The method according to any one of claims 3 to 5.

8. Prior to the step of determining dynamic mapping information for each pixel in a current region based on dynamic mapping information for each of the plurality of regions, the method further comprises: obtaining a correspondence between regions and metadata information elements; determining, from the metadata according to the correspondence between regions and metadata information elements, the metadata information elements corresponding to the plurality of regions separately to determine the dynamic mapping information for each of the plurality of regions; 8. The method of claim 1, further comprising:

9. the correspondence between an area and a metadata information element is a correspondence between an index of the metadata information element and an index of the area, or the correspondence between regions and metadata information elements is a correspondence between an index of the metadata element and coordinates of one or more pixels within the region. The method of claim 8.

10. The dynamic mapping information of the processing target pixel within the current angle of view range is dynamic mapping information of a plurality of regions within a first preset range, and the first preset range is a range centered on the processing target pixel. the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of a region to which a pixel in the two-dimensional panoramic projection image belongs, the pixel corresponding to the viewpoint center of the current angle of view; The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a plurality of regions within a second preset range, and the second preset range is a range centered on a pixel in the two-dimensional panoramic projection image that corresponds to a viewpoint center of the current angle of view range, or The dynamic mapping information of a pixel in the current region is the dynamic mapping information of the current region.

10. The method according to any one of claims 1 to 9.

11. the dynamic mapping information is used to map a dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range; or the dynamic mapping information is used to map a dynamic range of the pixels of the two-dimensional panoramic projection image from a second dynamic range to a first dynamic range, the first dynamic range being greater than the second dynamic range; 11. The method according to any one of claims 1 to 10.

12. said step of obtaining metadata of said two-dimensional panoramic projection image further comprising: receiving the metadata of the two-dimensional panoramic projection image from another electronic device.

12. The method of any one of claims 1 to 11, comprising:

13. The method further comprising: performing a subsequent process on the two-dimensional panoramic projection image within the current angle of view range, the subsequent process including performing a three-dimensional transformation on the two-dimensional panoramic projection image within the current angle of view range to obtain a three-dimensional panoramic image within the current angle of view range, the three-dimensional panoramic image being used for display; 13. The method of any one of claims 1 to 12, further comprising:

14. A dynamic range mapping apparatus for panoramic videos, comprising: a segmentation module, an acquisition module, a determination module, and a processing module, the segmentation module is configured to segment the two-dimensional panoramic projection image to obtain a plurality of regions of the two-dimensional panoramic projection image, the two-dimensional panoramic projection image being a two-dimensional panoramic projection image corresponding to any frame of a three-dimensional panoramic image in a panoramic video, each of the plurality of regions satisfying a preset condition, the preset condition including at least one of the following conditions: adjacent pixels in the two-dimensional panoramic projection image and non-adjacent pixels in the three-dimensional panoramic image are not located in the same region, non-adjacent pixels in the three-dimensional panoramic image are not located in the same region, the pixel points adjacent in the two-dimensional panoramic projection image include pixels on at least two adjacent projection planes of the three-dimensional panoramic image, and the pixels non-adjacent in the three-dimensional panoramic image include pixels in the three-dimensional panoramic image and corresponding to at least two non-adjacent projection planes; the acquiring module is configured to acquire metadata of the two-dimensional panoramic projection image, the metadata including metadata information elements corresponding to the regions of the two-dimensional panoramic projection image separately, the metadata information element corresponding to one region including dynamic mapping information of the region; The determination module is configured to determine dynamic mapping information for each pixel within a current angle of view based on the dynamic mapping information for each of the plurality of regions; the processing module is configured to perform dynamic range mapping for each pixel in the current angle of view based on the dynamic mapping information of each pixel in the current angle of view to obtain a two-dimensional panoramic projection image within the current angle of view, and the two-dimensional panoramic projection image within the current angle of view is used for display or subsequent processing; Dynamic range mapping device.

15. the electronic device further comprising a conversion module; the transformation module is configured to project the 3D panoramic image onto a surface of a polyhedron to expand the polyhedron into a two-dimensional plane to obtain the 2D panoramic projection image, the polyhedron comprising at least one of the following: a regular tetrahedron, a regular cube, a regular octahedron, a regular dodecahedron, or a regular icosahedron; 15. The apparatus of claim 14.

16. The segmentation module is particularly configured to segment the two-dimensional panoramic projection image in a first segmentation manner to obtain the plurality of regions of the two-dimensional panoramic projection image, the first segmentation manner being a manner of segmenting pixels in the two-dimensional panoramic projection image that belong to the same surface of the polyhedron into regions.

16. Apparatus according to claim 14 or 15.

17. The segmentation module is particularly configured to segment the two-dimensional panoramic projection image in a second segmentation manner to obtain the plurality of regions of the two-dimensional panoramic projection image, the second segmentation manner being a manner of segmenting pixels in the two-dimensional panoramic projection image belonging to the same surface of the polyhedron into regions to obtain a plurality of intermediate regions, and further segmenting at least one of the plurality of intermediate regions.

16. Apparatus according to claim 14 or 15.

18. The segmentation module is particularly configured to segment the two-dimensional panoramic projection image in a third segmentation manner to obtain a plurality of intermediate regions of the two-dimensional panoramic projection image, the third segmentation manner being a manner of clustering all pixels in the two-dimensional panoramic projection image and segmenting pixels belonging to the same category into the same intermediate region; and to obtain the plurality of regions, segment an intermediate region among the plurality of intermediate regions that does not satisfy the preset condition in a fourth segmentation manner, and for any intermediate region that does not satisfy the preset condition, the fourth segmentation manner is a manner of dividing the intermediate region along the boundary line of at least two surfaces of the polyhedron included in the intermediate region.

16. Apparatus according to claim 14 or 15.

19. the first division scheme, the second division scheme, or the third division scheme is preset in the electronic device; 19. Apparatus according to any one of claims 16 to 18.

20. The metadata of the two-dimensional panoramic projection image further includes division scheme instruction information, and the division scheme instruction information indicates that the division scheme of the two-dimensional panoramic projection image is the first division scheme, the second division scheme, or the third division scheme.

19. Apparatus according to any one of claims 16 to 18.

21. The acquiring module is further configured to acquire a correspondence between a region and a metadata information element; The determining module is further configured to determine metadata information elements corresponding to the plurality of regions separately from the metadata according to the correspondence between regions and metadata information elements, so as to determine the dynamic mapping information of each of the plurality of regions.

21. Apparatus according to any one of claims 14 to 20.

22. the correspondence between an area and a metadata information element is a correspondence between an index of the metadata information element and an index of the area, or the correspondence between regions and metadata information elements is a correspondence between an index of the metadata element and coordinates of one or more pixels within the region.

22. The apparatus of claim 21.

23. The dynamic mapping information of the processing target pixel within the current angle of view range is dynamic mapping information of a plurality of regions within a first preset range, and the first preset range is a range centered on the processing target pixel. the dynamic mapping information of the pixel to be processed within the current angle of view is dynamic mapping information of a region to which a pixel in the two-dimensional panoramic projection image belongs, the pixel corresponding to the viewpoint center of the current angle of view; The dynamic mapping information of the pixel to be processed within the current angle of view range is dynamic mapping information of a plurality of regions within a second preset range, and the second preset range is a range centered on a pixel in the two-dimensional panoramic projection image that corresponds to a viewpoint center of the current angle of view range, or The dynamic mapping information of a pixel in the current region is the dynamic mapping information of the current region.

23. Apparatus according to any one of claims 14 to 22.

24. the dynamic mapping information is used to map a dynamic range of the pixels of the two-dimensional panoramic projection image from a first dynamic range to a second dynamic range; or the dynamic mapping information is used to map a dynamic range of the pixels of the two-dimensional panoramic projection image from a second dynamic range to a first dynamic range, the first dynamic range being greater than the second dynamic range; 24. Apparatus according to any one of claims 14 to 23.

25. the acquisition module is specifically configured to receive the metadata of the two-dimensional panoramic projection image from another electronic device; 25. Apparatus according to any one of claims 14 to 24.

26. the transformation module is further configured to perform subsequent processing on the two-dimensional panoramic projection image within the current angle of view range, the subsequent processing including performing a three-dimensional transformation on the two-dimensional panoramic projection image within the current angle of view range to obtain a three-dimensional panoramic image within the current angle of view range, and the three-dimensional panoramic image is used for display; 26. Apparatus according to any one of claims 14 to 25.

27. 14. An electronic device comprising a memory and at least one processor connected to the memory, the memory configured to store instructions, and wherein the method of any one of claims 1 to 13 is executed after the instructions are read by the at least one processor.

28. A computer readable storage medium storing a computer program which, when executed by a processor, performs the method of any one of claims 1 to 13.

Citation Information

Patent Citations

  • Motion compensation prediction method and device

    EP3499894A1

  • Method and system for generating region nesting messages for video pictures

    JP2019515572A

  • Motion compensated prediction method and device - Patent Application 20070122967

    JP2019531674A