Video regionalization encoding method, video regionalization decoding and display method, and electronic device

Abstract

The application provides a video regionalization encoding method, a video regionalization decoding and display method and an electronic device. A source video is divided into multiple regional units, and an enhanced regional unit is determined from the regional units. A base code stream corresponding to each regional unit is generated, and an enhanced regional code stream of each enhanced regional unit is generated. The code rate of the enhanced regional code stream is higher than that of the base code stream. The base code stream and the enhanced regional code stream are sent to a playing end, and then the playing end decodes the base code stream and the enhanced regional code stream to obtain a base picture and an enhanced picture. The base picture is fully covered to a target display region, and the base picture at a position corresponding to the enhanced regional unit in the target display region is adjusted based on the enhanced picture. According to the scheme, different regions of a video picture can be intelligently identified, and encoding and transmission are performed at different resolutions, so that the bandwidth is saved, and the region that a viewer focuses on watching is always the clearest.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and more specifically, to a video regional encoding method, a video regional decoding and display method, and an electronic device. Background Technology

[0002] Currently, when we watch immersive videos (such as watching 360-degree panoramic videos with VR headsets or watching dome movies in dome theaters), the common practice of video service providers is to use ordinary H.265 encoders to compress the entire ultra-high resolution spherical image as a single planar video into a single bitstream, and then transmit it completely to the playback device.

[0003] Existing traditional methods have several drawbacks. Specifically, human vision is limited; when wearing a head-mounted display, a user can only see a range of approximately 90 to 110 degrees in front of them at a time. However, the system transmits areas behind and above the head—areas we don't even see—at the same high-definition quality, resulting in a significant waste of bitrate. For dome theaters, the audience's primary focus is on the center of the stage, with the peripheral areas only providing a sense of immersion and not requiring the same level of clarity as the central area. However, uniform encoding across the entire frame cannot differentiate between these varying clarity requirements. While some new technologies can achieve "transmit what you see," they often rely on new video coding standards or special encapsulation formats. Most existing H.265 hardware decoders on the market cannot be used directly, resulting in poor compatibility and high hardware, software, and promotion costs.

[0004] The aforementioned shortcomings of existing traditional methods directly result in immersive videos being either of poor overall quality and blurry, or experiencing frequent buffering, all within limited network bandwidth. Content providers also cannot efficiently utilize bandwidth, failing to allocate different quality levels to different areas of the dome screen. Furthermore, to achieve a better experience, users may be forced to replace their devices with incompatible new ones, incurring high costs. Summary of the Invention

[0005] The purpose of this invention is to provide a video regional encoding method, a video regional decoding and display method, and an electronic device for encoding and transmitting at different resolutions, thereby saving bandwidth while ensuring the clarity of the viewing area.

[0006] In a first aspect, the present invention provides a video regionalization coding method, the method comprising: Acquire the source video and obtain the region boundary and target display area of ​​the source video; The source video is divided into multiple region units within the region boundary, and enhancement region units are determined from the multiple region units; Generate a base bitstream corresponding to each of the aforementioned regional units, and generate an enhanced regional bitstream for each of the aforementioned enhanced regional units, wherein the bitrate of the enhanced regional bitstream is higher than the bitrate of the base bitstream; The base stream and the enhanced region stream are sent to the playback terminal so that after the playback terminal decodes the base image and the enhanced image based on the base stream and the enhanced region stream, it fully covers the target display area with the base image and adjusts the base image at the corresponding enhanced region unit position within the target display area based on the enhanced image.

[0007] In an optional implementation, the step of determining the enhancement region element from the plurality of region elements includes: The regional importance of each regional unit is calculated based on the unit information of each regional unit; Regions with a region importance higher than a preset threshold are identified as enhanced region units.

[0008] In an optional implementation, the method further includes acquiring a display scene, the display scene including a VR headset scene and a dome display scene; The unit information includes the probability of overlap between the region unit and the current viewport or the predicted viewport, the dome display weight of the region unit, the content salience of the region unit, the motion complexity of the region unit, and whether the region unit is located in the projection blending area or the seam sensitive area.

[0009] In an optional implementation, the step of generating the enhanced region bitstream for each of the enhanced region units includes: The corresponding coding parameters are calculated based on the regional importance of each of the enhanced region units; The source video of the enhanced region unit is encoded according to the encoding parameters to obtain the corresponding enhanced region bitstream.

[0010] In an optional implementation, the step of calculating the corresponding coding parameters based on the regional importance of each of the enhanced region units includes: For each of the enhanced region units, the regional importance of the enhanced region unit at the current time and the average regional importance of all region units at the current time are obtained; The encoding parameter offset is calculated based on the current regional importance, the average regional importance, and the set parameter offset magnitude. The corresponding encoding parameters are determined based on the encoding parameter offset.

[0011] In an optional implementation, the step of encoding the source video of the enhancement region unit according to the encoding parameters to obtain the corresponding enhancement region bitstream includes: Multiple offset parameters are obtained by offsetting the encoding parameters; The source video of the enhanced region is encoded based on the encoding parameters and each of the offset parameters to obtain multiple enhanced region bitstreams with different quality levels.

[0012] In an optional implementation, the method further includes: Obtain the regional attributes of each of the aforementioned regional units, wherein the regional attributes include the regional number; Obtain the coordinate information of each location point in the target display area; A correspondence is established between the regional attributes of each region unit and the coordinate information of the display area, and a region mapping table is generated. The region mapping table is used by the playback terminal to display the basic screen and the enhanced screen according to the correspondence.

[0013] In a second aspect, the present invention provides a video regional decoding and display method, the method comprising: Obtain the base bitstream and enhanced region bitstream to be displayed in the target display area; The base bitstream and the enhanced region bitstream are obtained by encoding the source video. The base bitstream corresponds to each region unit within the region boundary of the source video, and the enhanced region bitstream corresponds to the enhanced region unit therein. The bitrate of the enhanced region bitstream is higher than that of the base bitstream. The base bitstream and the enhanced region bitstream are decoded respectively to obtain the base image and the enhanced image; The base image is displayed over the entire target display area, and the base image at the corresponding position of the enhanced area unit within the target display area is adjusted based on the enhanced image.

[0014] In an optional implementation, the step of decoding the base bitstream and the enhanced region bitstream respectively to obtain the base image and the enhanced image includes: Obtain the current playback state within the target display area, and determine the target enhancement region unit from multiple enhancement region units based on the current playback state; The target enhanced region bitstream corresponding to the target enhanced region unit is determined from the obtained enhanced region bitstream; The base stream and the target enhancement region stream are decoded respectively to obtain the base image and the enhancement image.

[0015] In an optional implementation, each of the enhancement region units corresponds to multiple quality levels, and each quality level corresponds to a different bitrate; The step of determining the target enhancement region unit from multiple enhancement region units based on the current playback state includes: For each of the aforementioned enhancement region units, based on the current playback state, multiple benefit information of the enhancement region unit at multiple quality levels is obtained; The total revenue is obtained by summing up multiple revenue data points according to the set weights. Under constraints, a preset number of enhancement region units with higher total benefits are selected, and the selected enhancement region units are used as target enhancement region units to obtain the target quality level of the target enhancement region units.

[0016] In an optional implementation, the multiple benefit information includes the regional importance of the enhanced region unit, the quality gain of the selected quality level relative to the base bitrate, the switching cost of switching from the current quality level to the selected quality level, the bandwidth usage of the enhanced region unit, and the decoding resource usage.

[0017] In an optional implementation, the step of adjusting the base image at the location corresponding to the enhanced area unit within the target display area based on the enhanced image includes: Obtain a region mapping table, which stores the correspondence between the region attributes of each region unit and the coordinate information of each location point in the target display area; Based on the correspondence in the region mapping table, determine the coordinate information of the enhanced region unit in the target display region; The enhanced image is projected onto the location corresponding to the determined coordinate information in order to adjust the base image at the corresponding location.

[0018] In an optional implementation, the step of adjusting the base image at the corresponding location includes: The enhanced image can be used to cover the base image at the corresponding location, or the enhanced image and the base image at the corresponding location can be merged.

[0019] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method described in any of the foregoing embodiments.

[0020] This invention provides a video regionalization encoding method, a video regionalization decoding and display method, and an electronic device. The method divides the source video into multiple regional units and identifies enhanced regional units from among them. A base bitstream corresponding to each regional unit and an enhanced regional bitstream for each enhanced regional unit are generated, with the bitrate of the enhanced regional bitstream being higher than that of the base bitstream. The base and enhanced regional bitstreams are sent to the playback end, which then decodes them to obtain the base and enhanced images. The base image fully covers the target display area, and the base image at the corresponding enhanced regional unit position within the target display area is adjusted based on the enhanced image. This solution intelligently identifies different regions of the video image and encodes and transmits them with different resolutions, thereby saving bandwidth while ensuring that the areas the viewer focuses on are always the clearest. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a video regionalization coding method provided in an embodiment of the present invention; Figure 2 for Figure 1 A flowchart of the sub-steps included in S12; Figure 3 for Figure 1 A flowchart of the sub-steps included in S13; Figure 4 A flowchart of a video regional decoding and display method provided in an embodiment of the present invention; Figure 5 for Figure 4 A flowchart of the sub-steps included in S22; Figure 6 for Figure 4 A flowchart of the sub-steps included in S23; Figure 7 This is a schematic diagram illustrating the overall logic of video regional encoding, decoding, and display in an embodiment of the present invention; Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0024] This invention provides a video regionalization coding method, which is applied at an encoding end equipped with an encoder responsible for compressing video / audio into a bitstream. The encoding end can be a cloud server or computer equipment, etc.

[0025] Please see Figure 1 The above is a flowchart of a video regionalization coding method provided in an embodiment of the present invention. The video regionalization coding method includes the following steps: S11, acquire the source video, and obtain the region boundary and target display area of ​​the source video; S12, the source video is divided into multiple region units within the region boundary, and an enhancement region unit is determined from the multiple region units; S13, generate the base bitstream corresponding to each of the said regional units, and generate the enhanced regional bitstream of each of the said enhanced regional units, wherein the bitrate of the enhanced regional bitstream is higher than the bitrate of the base bitstream; S14, the basic bitstream and the enhanced region bitstream are sent to the playback end, so that after the playback end decodes the basic image and the enhanced image based on the basic bitstream and the enhanced region bitstream, the basic image is fully covered to the target display area, and the basic image at the corresponding enhanced region unit position in the target display area is adjusted based on the enhanced image.

[0026] The video regionalization coding method provided in this embodiment can be applied to immersive video scenarios, that is, videos that can provide an immersive experience, such as 360-degree panoramic videos in VR headsets or movies projected onto a huge dome in a dome theater. Their characteristics are that the screen is very large and very wide, enveloping the viewer.

[0027] Based on this, the video regionalization encoding method provided in this embodiment can be mainly applied to display scenarios including VR head-mounted displays and dome displays.

[0028] In this embodiment, the source video can be an ERP, a cube map, or a fisheye dome master. In a VR headset scenario, the target display area is the area of ​​the image that can be displayed by the VR headset device; in a dome display scenario, the target display area is the area of ​​the image that the dome can display.

[0029] In addition, the display parameters of the source video can be obtained. In VR headset scenarios, the main parameters obtained are spherical coordinates, field of view, user orientation, and relationship between adjacent viewports. In dome display scenarios, the main parameters obtained are dome radius or normalized dome coordinates, main viewing area, projector coverage area, overlapping and blending area, and invalid outer circle area.

[0030] In a VR headset scenario, the viewport refers to the small area that the eyes actually see when the head is turned in a certain direction while wearing the VR headset. It is a very small part of the overall image.

[0031] After obtaining the source video to be encoded, the regional boundaries of the source video are determined, which can be understood as determining the boundaries of a frame. Within the regional boundaries, the source video is divided into regional units according to the set division rules.

[0032] For example, in VR headset scenarios, region units can be divided based on ERP grids, cube facets, or viewport neighborhood segmentation. In dome display scenarios, region units can be divided using a combination of segmentation methods such as central circular area, annular sector, edge area, projector coverage area, and blending area.

[0033] Among them, the boundaries of the region units are preferably set at locations with low requirements for texture continuity, weak motion, or that can be smoothed by the fusion parameters.

[0034] Regional units can be non-overlapping, or they can overlap slightly at the boundaries of the regional units.

[0035] In VR headset scenarios, the number of region units can range from 6 to 48, while in dome display scenarios, the number of region units can range from 5 to 64. Too few region units will reduce scheduling accuracy, while too many will increase coding and switching overhead.

[0036] From the multiple defined region units, enhancement region units are identified. There can be one or more enhancement region units. Enhancement region units are of higher importance than other region units, which is reflected in the higher bitrate of the encoded bitstream of the enhancement region unit compared to the base bitstream of other region units. Consequently, the image at the corresponding location of the enhancement region unit is clearer after subsequent decoding.

[0037] It should be noted that in this embodiment, each region unit is encoded to obtain a corresponding basic bitstream, while the enhanced region unit is also encoded to obtain a corresponding enhanced region bitstream. That is, the enhanced region unit corresponds to two forms of bitstream: the basic bitstream and the enhanced region bitstream, while other non-enhanced region units only correspond to the basic bitstream.

[0038] Please see Figure 2 In this embodiment, the step of determining the enhanced region element from multiple region elements can be implemented in the following way: S121, calculate the regional importance of each regional unit based on the unit information of each regional unit; S122, the region units whose region importance is higher than a preset threshold are determined as enhanced region units.

[0039] In this embodiment, the unit information of each region unit includes the overlap probability between the region unit and the current viewport or the predicted viewport, the dome display weight of the region unit, the content salience of the region unit, the motion complexity of the region unit, and whether the region unit is located in the projection blending area or the seam sensitive area.

[0040] The overlap probability between a region cell and the current viewport or the predicted viewport represents the probability that the region cell will be seen by the user. The higher the overlap probability, the higher the probability that the region cell will be seen by the user.

[0041] The display weight of the dome in each area can be set based on the actual situation. For example, the display weight of the central area of ​​the dome is higher, while the display weight of the two side edge areas of the dome is lower.

[0042] Furthermore, the salience of a region unit can be set based on the content in the image. For example, regions containing main characters have higher salience, while regions containing background have lower salience. The motion complexity corresponding to a region unit represents the main dynamically changing areas in the image.

[0043] Based on this, in this embodiment, the regional importance of the i-th region unit at time t can be calculated using the following formula:

[0044] in, This represents the probability of overlap between region cell i and the current viewport or the predicted viewport. This indicates the weight displayed on the dome screen. Indicates content salience, Indicates motion complexity. This indicates whether region unit i is located in the projection blending area or the seam sensitive area; a, b, c, d, and e are non-negative weights, and a+b+c+d+e=1. In VR headset scenarios, a can take a larger value, such as 0.4-0.8, while in dome display scenarios, b and e can take larger values, such as 0.3-0.7 and 0.1-0.3, respectively.

[0045] The regional importance of a region is calculated based on the information of the region unit, and the region unit whose regional importance is higher than a preset threshold is identified as an enhanced region unit.

[0046] Based on this, a base bitstream for each regional unit and an enhanced regional bitstream for enhanced regional units are generated. When generating the base bitstream for each regional unit, all regional units can be encoded together to generate the overall base bitstream, or multiple adjacent regional units can be merged for encoding, thus reducing the number of decoding instances. Alternatively, each regional unit can be encoded separately to generate its corresponding base bitstream. In this embodiment, the base bitstream is a low bitrate or basic quality bitstream covering all regional units within the region boundary.

[0047] Please refer to Figure 3 In the process of generating the enhanced region bitstream for each enhanced region unit, the following methods can be used: S131, calculate the corresponding coding parameters based on the regional importance of each enhanced region unit; S132, the source video of the enhanced region unit is encoded according to the encoding parameters to obtain the corresponding enhanced region bitstream.

[0048] In this embodiment, when calculating the corresponding coding parameters based on the regional importance of the enhanced region unit, it can be specifically achieved in the following way: For each of the enhanced region units, the regional importance of the enhanced region unit at the current time and the average regional importance of all region units at the current time are obtained; the coding parameter offset is calculated based on the regional importance at the current time, the average regional importance, and the set parameter offset magnitude; the corresponding coding parameters are determined according to the coding parameter offset.

[0049] Specifically, the coding parameter offset DQ can be calculated using the following formula:

[0050] in, This indicates the regional importance of region element i at time t. Let be the average regional importance of all regional units at time t, K be the adjustment coefficient, and Qmax be the set parameter offset amplitude. The actual value can be set according to the encoder bitrate control strategy and subjective quality requirements. 'round' means converting the value to the nearest integer, and 'clip' means limiting the value to a specified minimum and maximum value. In this embodiment, it is limited to - and between.

[0051] Based on the calculated encoding parameter offset, the corresponding encoding parameter QP can be determined. QP represents the quantization parameter, which is used to control the quantization intensity and directly determines the quantization step size at each step, thus affecting the bit rate, image quality, and the degree of detail retention.

[0052] Based on the above formula, when the region importance is higher than the average region importance, the value of the coding parameter offset is negative, corresponding to a lower coding parameter QP. The smaller the QP, the finer the quantization, the more details are retained, and the higher the bit rate. Conversely, the larger the QP, the coarser the quantization, the lower the bit rate, but the greater the distortion.

[0053] In addition, to reduce screen flicker, the encoding parameter offsets calculated using the above methods can be further smoothed using the following techniques:

[0054] in, This represents the smoothing coefficient, which can range from 0.5 to 0.95.

[0055] After calculating the encoding parameters using the above method, the source video of the enhanced region unit is encoded according to the encoding parameters to obtain the corresponding enhanced region bitstream.

[0056] In order to adapt to different bandwidth and decoding resource conditions in actual scenarios, this embodiment can also generate multiple enhanced region bitstreams of different quality levels for each enhanced region unit, so that the appropriate quality level can be selected based on the actual scenario.

[0057] Based on this consideration, in this embodiment, when encoding the source video of the enhancement region unit according to the encoding parameters, it can be achieved in the following way: Multiple offset parameters are obtained by offsetting based on the encoding parameters; the source video of the enhanced region is encoded based on the encoding parameters and each of the offset parameters respectively to obtain multiple enhanced region bitstreams with different quality levels.

[0058] In this embodiment, multiple offset parameters can be obtained by offsetting the encoding parameters by a set offset amount. For example, an offset parameter after upward offset and an offset parameter after downward offset can be obtained.

[0059] Thus, by encoding the source video of the enhanced region based on the encoding parameters and offset parameters, enhanced region bitstreams of different quality levels can be obtained. For example, enhanced region bitstreams of three quality levels—low, medium, and high—can be obtained.

[0060] It should be noted that, apart from the enhanced regional unit, other regional units can also be encoded based on their regional importance and in accordance with the above method to obtain the basic bitstream, or they can be encoded according to the set basic bitrate to obtain the basic bitstream.

[0061] In this embodiment, each bitstream output by the encoder is a standard H.265 bitstream. The enhanced region bitstream can be encoded by padding the cropped region image, or it can be formed into an independently accessible region version using H.265 tile / slice constraints. If a tile / slice implementation is used, cross-region referencing should be restricted, random access points should be aligned, and boundary filtering should be disabled or compensated.

[0062] In addition, the regionalization encoding method provided in this embodiment may also include the following steps: Obtain the region attributes of each region unit, the region attributes including the region number; obtain the coordinate information of each position point of the target display area; construct the correspondence between the region attributes of each region unit and the coordinate information of the display area, and generate a region mapping table, the region mapping table being used by the playback terminal to display the basic screen and the enhanced screen according to the correspondence.

[0063] In this embodiment, a region mapping table is constructed to store the correspondence between the coordinate information of each region unit and each location point in the target display area. In addition, it also includes the source projection coordinates, bitstream address, quality level, timestamp, switching point, and fusion parameters of each region unit.

[0064] The constructed region mapping table is interpreted by the playback end and is not a required syntax for the H.265 decoder.

[0065] Based on this, the base stream, enhanced region streams, and region mapping tables are published as pullable media resources. DASH, HLS, RTSP, or proprietary low-latency transport protocols can be used; the specific transport protocol is not limited.

[0066] This invention also provides a video regional decoding and display method, which is applied to a playback terminal. The decoder is deployed on the playback terminal, and the playback terminal can call the decoder to perform decoding.

[0067] Please see Figure 4 The above is a flowchart of a video regional decoding and display method provided in an embodiment of the present invention. The video regional decoding and display method includes the following steps: S21, obtain the base bitstream and enhanced region bitstream to be displayed in the target display area; The base bitstream and the enhanced region bitstream are obtained by encoding the source video. The base bitstream corresponds to each region unit within the region boundary of the source video, and the enhanced region bitstream corresponds to the enhanced region unit therein. The bitrate of the enhanced region bitstream is higher than that of the base bitstream. S22, decode the base bitstream and the enhanced region bitstream respectively to obtain the base image and the enhanced image; S23, the basic image is fully covered to display the target display area, and the basic image at the corresponding position of the enhanced area unit in the target display area is adjusted based on the enhanced image.

[0068] The regional decoding and display method provided in this embodiment corresponds to the regional encoding method provided in the above embodiments, and can be used to decode and display the bitstream obtained under any implementation of the above embodiments.

[0069] Similarly, the regional decoding and display method provided in this embodiment can be applied to VR headset scenarios or dome screen display scenarios. The playback terminal acquires the base bitstream and enhanced region bitstream to be displayed in the target display area. The target display area can be the screen area that can be displayed by the VR headset device or the screen area that can be displayed by the dome screen. The base bitstream and enhanced region bitstream acquired by the playback terminal are bitstreams obtained under any of the above embodiments. For the specific acquisition method, please refer to the relevant descriptions in the above embodiments.

[0070] The playback device calls the decoder to decode the base stream and the enhanced region stream, obtaining the base image and the enhanced image respectively. The decoder can be a regular H.265 decoder.

[0071] The decoded base image fully covers the target display area, while the enhanced image is only reflected in the corresponding enhanced area unit position within the target display area. For example, the base image at that position is adjusted based on the enhanced image.

[0072] As described above, the encoding end determines the enhanced region units based on the regional importance of each region unit, and provides enhanced region bitstreams with higher bitrates for these enhanced region units. The playback end can select the target enhanced region bitstream from these enhanced region bitstreams according to the actual playback status to adapt to the current actual scenario.

[0073] Specifically, please refer to Figure 5 The steps of decoding the base stream and the enhanced region stream to obtain the base image and the enhanced image can be implemented in the following ways: S221, obtain the current playback state within the target display area, and determine the target enhancement area unit from multiple enhancement area units based on the current playback state; S222, determine the target enhanced region bitstream corresponding to the target enhanced region unit from the obtained enhanced region bitstream; S223, decode the base bitstream and the target enhancement region bitstream respectively to obtain the base image and the enhancement image.

[0074] In this embodiment, at the playback end, the region scheduler can select the target enhancement region unit based on the current playback state within the target display area. For example, it can select based on the current viewport or dome screen state, bandwidth, buffering, and decoding resources. For instance, in a VR headset scenario, the enhancement region units corresponding to the current viewport and adjacent viewports can be the target enhancement region units. In a dome screen display scenario, the enhancement region units corresponding to the main visual path, central area, projection blending area, and highly salient area can be the target enhancement region units.

[0075] As described above, each enhancement region unit corresponds to multiple quality levels, and each quality level corresponds to a different bitrate. Specifically, in this embodiment, the step of determining the target enhancement region unit from multiple enhancement region units based on the current playback state can be implemented in the following way: For each of the aforementioned enhancement region units, multiple benefit information of the enhancement region unit under multiple quality levels is obtained based on the current playback state; the multiple benefit information is accumulated according to the set weights to obtain the total benefit; under constraints, a preset number of enhancement region units with higher total benefits are selected, the selected enhancement region units are used as target enhancement region units, and the target quality level of the target enhancement region units is obtained.

[0076] In this embodiment, the revenue information of the augmented region unit can be evaluated from multiple dimensions, and then the total revenue can be obtained. The augmented region unit with the higher total revenue is selected as the target augmented region unit. Constraints are considered when selecting, including bandwidth limitations, buffer resource limitations, and decoding resource limitations.

[0077] In this embodiment, while determining the target enhancement region unit, the target quality level of the target enhancement region unit among multiple quality levels is also determined. That is, the total benefit evaluated includes the benefit contributed to the selected target quality level.

[0078] Specifically, the enhanced region unit's multiple benefit information obtained based on the current playback state includes the enhanced region unit's regional importance, the quality gain of the selected quality level relative to the base bitrate, the switching cost of switching from the current quality level to the selected quality level, the enhanced region unit's bandwidth usage, and the decoding resource usage.

[0079] In this embodiment, the total benefit of each enhanced region unit can be evaluated using the following formula:

[0080] in, The calculated regional importance of the i-th enhancement region element at time t; This represents the normalized quality gain of the bitrate of the q-th quality level of the selected enhancement region i relative to the base bitrate (the bitstream of the base region unit). The candidate quality level q can be given by the media list, for example, each enhancement region unit corresponds to low, medium, high or multiple target bitrate levels. This indicates the normalized bandwidth usage of the enhanced region unit; This indicates the normalized decoding resource usage of the enhanced region unit; This represents the normalized switching cost when switching from the current quality level to the selected quality level. The weights are non-negative.

[0081] Under the constraints of total bandwidth upper limit, buffer lower limit and decoding resources, the area scheduler selects a set of enhanced area units with higher total revenue.

[0082] The standard H.265 decoder decodes the base stream and the selected enhancement region stream separately to obtain the base image and the enhancement image. The display compositor then displays the base image and the enhancement image in the target display area.

[0083] In this embodiment, the base image fully covers the target display area, and the base image at the corresponding enhanced area unit position within the target display area is adjusted based on the enhanced image. For details, please refer to [link to relevant documentation]. Figure 6 This step can be achieved in the following ways: S231, obtain a region mapping table, wherein the region mapping table stores the correspondence between the region attributes of each region unit and the coordinate information of each position point of the target display area; S232, determine the coordinate information of the enhanced region unit in the target display area based on the correspondence in the region mapping table; S233, the enhanced image is projected onto the position corresponding to the determined coordinate information in order to adjust the basic image at the corresponding position.

[0084] As described above, the encoding end constructs a region mapping table, which is published along with the bitstream. The region mapping table stores the correspondence between the coordinates of each region unit and the target display area.

[0085] After selecting an enhancement region unit, the corresponding coordinates within the target display area can be determined based on the mapping in the region map table. This allows the decoded enhanced image to be projected onto the corresponding location.

[0086] In this embodiment, the enhanced image is used to adjust the base image at the corresponding location. This can be done by covering the base image at the corresponding location with the enhanced image, or by merging the enhanced image with the base image at the corresponding location. Other unenhanced areas retain their original base image.

[0087] The playback device continuously collects viewport, bandwidth, and buffer status, and updates the enhanced region bitstream set at random access points or allowed switching points, updating it cyclically.

[0088] Please refer to the following: Figure 7 This is a schematic diagram illustrating the overall implementation logic of the video regionalization encoding method and the video regionalization decoding and display method provided in the embodiments of the present invention.

[0089] At the encoding end, after obtaining the source video and target display parameters (including the target display area), the source video is divided into multiple region units, resulting in a set of region units. The regional importance of each region unit is calculated to determine the enhanced region units. Enhanced region coding parameters are calculated for the enhanced region units, while basic region coding parameters are generated for non-enhanced region units. Based on the basic region coding parameters, each region unit is encoded to generate a basic bitstream, and the enhanced region units are encoded based on the enhanced region coding parameters to obtain the enhanced region bitstream. Furthermore, a region mapping table is constructed, containing the correspondence between the regional attributes of each region unit and the coordinate information of each location point in the target display area. A media list is then published based on the basic bitstream, the enhanced region bitstream, and the region mapping table.

[0090] On the playback end, the playback status of the viewport or dome screen is captured, and the bandwidth, buffer, and decoding resources are estimated. The media list is retrieved, and the appropriate target augmented region unit is selected from the augmented region units. The H.265 decoder is invoked to decode the base video based on the base bitstream and to obtain the augmented video based on the augmented region bitstream of the selected target augmented region unit. According to the region mapping table, the base video is overlaid onto the target display area, and the augmented video is overlaid onto the corresponding augmented region unit to achieve blended display. Finally, an immersive video image is output.

[0091] The following examples illustrate the two main application scenarios of this embodiment.

[0092] In VR headset scenarios, the base stream covers the entire 360-degree view, while the enhanced region stream covers the current viewport and the predicted viewport neighborhood. The region scheduler can predict the viewport position 0.5 to 2 seconds later based on the headset's posture, angular velocity, and historical viewing trajectory, setting the region unit of the current viewport as the highest priority, the region units of adjacent viewports as the second highest priority, and the region units facing away as the lowest priority.

[0093] When bandwidth is sufficient, the playback device requests enhanced region bitstreams from the current viewport, adjacent viewports, and some highly salient areas; when bandwidth decreases, the playback device retains the current viewport as the enhanced region, reduces the quality of adjacent viewport regions, or reverts to the basic bitstream; when the user turns their head quickly, the playback device prioritizes requesting candidate regions in the direction of rotation and replaces them with enhanced regions at the switching point.

[0094] Taking a 7680x3840 ERP 360-degree video as an example, the system divides the sphere into 12 4x3 region units and additionally records the wrap-around relationship of the left and right boundaries. The encoding end generates a base stream covering the entire 360-degree image, with a resolution of 3840x1920; simultaneously, it generates 2 to 3 enhanced region streams with different quality levels for each enhanced region unit. The random access points of the base stream and the enhanced region streams are aligned by 1 second.

[0095] During playback, the head-mounted display reports its current orientation and angular velocity. The region scheduler calculates the current viewport coverage area and adjacent regions in the rotation direction. When bandwidth allows, it requests one high-quality enhancement stream for the current viewport and medium-quality enhancement streams for two to four adjacent viewport regions; when bandwidth is insufficient, it retains only the current viewport enhancement stream, and the remaining regions are displayed by the base stream. The display compositor pastes the enhanced region images back onto the sphere and performs a smooth transition at the boundaries according to the blending width in the region map table.

[0096] In dome display scenarios, viewers typically perceive a large area of ​​dome content, therefore it's unsuitable to transmit only a single viewport area. The base bitstream should cover the entire effective dome area, while the enhanced region bitstream should prioritize covering the main visual path, central area, highly salient content areas, projector overlap / blending areas, and seam-sensitive areas. Invalid images outside the circular effective area can be preprocessed into low-complexity backgrounds before encoding to reduce the invalid bitrate.

[0097] For multi-projector dome systems, region units can correspond to projector coverage areas or blending areas. The display compositor performs dome mesh mapping, projector geometry transformation warping, and boundary blending before output. If an enhancement region's bitstream is unavailable, the display compositor fills that region with the base bitstream to ensure continuous display across the entire dome.

[0098] Taking an 8192x8192 fisheye dome master as an example, the system divides the effective circular area into a central circular area, several annular sectors, edge annular sectors, and a projector overlap and blending area. Invalid areas outside the circle are filled with a low-complexity background before encoding. The encoder generates a complete dome base bitstream and generates enhanced region bitstreams for the central circular area, main visual path area, blending area, and highly salient area.

[0099] During playback, the zone scheduler does not use a single head-mounted display viewport. Instead, it reads the dome playback configuration, venue projector coverage parameters, and the main visual path in the program timeline. If network or decoding resources are limited, priority is given to preserving the central area, the main visual path, and the enhanced stream in the blended area; low-interest areas at the edges fall back to the base stream. The display compositor maps each decoded image onto the dome grid and then outputs the warped / blend image according to the projector configuration.

[0100] In summary, the video regional decoding and display method provided in this embodiment is compatible with existing H.265 decoders. Without modifying the internal syntax and decoding process of the H.265 decoder, all media resources can be decoded by ordinary H.265 hardware or software decoders, which facilitates the reuse of existing terminal and chip capabilities.

[0101] This solution is compatible with both VR headset scenarios and dome display scenarios. It unifies viewport probability and dome display weight into the region importance calculation model, which can support both the current viewport enhancement of VR headsets and the basic quality assurance of the main visual area, projection blending area and the entire dome of the dome.

[0102] Compared to solutions that only adjust ROI QP within the entire frame, this solution dynamically selects the base bitstream and several enhanced region bitstreams when network bandwidth, buffer status, and the number of decoder instances change. This ensures the quality of critical areas of the viewport or dome screen, while using the base bitstream to cover unenhanced areas. Even when enhanced regions are unavailable, continuous image display is still possible, preventing black screens when the user turns their head or at the edges of the dome screen. This truly reduces the transmission pressure on non-critical areas.

[0103] In this solution, by combining regional boundary constraints, GOP alignment (ensuring that multiple independent video streams remain consistent at the same time point, facilitating seamless switching between them and avoiding black screens or stuttering), QP time smoothing, boundary blending, and switching hysteresis, the seams, flickering, and switching jitter caused by partitioned encoding transmission are reduced.

[0104] Please see Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be the aforementioned encoding end or playback end. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0105] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0106] The processor is used to read / write data or programs stored in the memory and execute the video localization encoding method or the video localization decoding and display method provided in any embodiment of the present invention.

[0107] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0108] It should be understood that, Figure 8 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0109] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the video localization encoding method or the video localization decoding and display method provided in the above embodiments.

[0110] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, it can perform the aforementioned video localization encoding method or video localization decoding and display method. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0111] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0112] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0116] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video regional coding method, characterized in that, The method includes: Acquire the source video and obtain the region boundary and target display area of ​​the source video; The source video is divided into multiple region units within the region boundary, and enhancement region units are determined from the multiple region units; Generate a base bitstream corresponding to each of the aforementioned regional units, and generate an enhanced regional bitstream for each of the aforementioned enhanced regional units, wherein the bitrate of the enhanced regional bitstream is higher than the bitrate of the base bitstream, and each of the aforementioned enhanced regional units corresponds to a base bitstream and an enhanced regional bitstream; The base stream and the enhanced region stream are sent to the playback terminal so that after the playback terminal decodes the base image and the enhanced image based on the base stream and the enhanced region stream, the base image is fully covered to the target display area, and the base image at the corresponding enhanced region unit position in the target display area is adjusted based on the enhanced image. The step of determining the enhancement region element from the plurality of region elements includes: The regional importance of each regional unit is calculated based on the unit information of each regional unit; regional units with regional importance higher than a preset threshold are identified as enhanced regional units. The method further includes acquiring a display scene, which includes a VR headset scene and a dome display scene; The unit information includes the probability of overlap between the region unit and the current viewport or the predicted viewport, the dome display weight of the region unit, the content salience of the region unit, the motion complexity of the region unit, and whether the region unit is located in the projection blending area or the seam sensitive area.

2. The video regionalization coding method according to claim 1, characterized in that, The step of generating the enhanced region bitstream for each of the enhanced region units includes: The corresponding coding parameters are calculated based on the regional importance of each of the enhanced region units; The source video of the enhanced region unit is encoded according to the encoding parameters to obtain the corresponding enhanced region bitstream.

3. The video regionalization coding method according to claim 2, characterized in that, The step of calculating the corresponding coding parameters based on the regional importance of each of the enhanced region units includes: For each of the enhanced region units, the regional importance of the enhanced region unit at the current time and the average regional importance of all region units at the current time are obtained; The encoding parameter offset is calculated based on the current regional importance, the average regional importance, and the set parameter offset magnitude. The corresponding encoding parameters are determined based on the encoding parameter offset.

4. The video regionalization coding method according to claim 2, characterized in that, The step of encoding the source video of the enhanced region unit according to the encoding parameters to obtain the corresponding enhanced region bitstream includes: Multiple offset parameters are obtained by offsetting the encoding parameters; The source video of the enhanced region is encoded based on the encoding parameters and each of the offset parameters to obtain multiple enhanced region bitstreams with different quality levels.

5. The video regionalization coding method according to claim 1, characterized in that, The method further includes: Obtain the regional attributes of each of the aforementioned regional units, wherein the regional attributes include the regional number; Obtain the coordinate information of each location point in the target display area; A correspondence is established between the regional attributes of each region unit and the coordinate information of the display area, and a region mapping table is generated. The region mapping table is used by the playback terminal to display the basic screen and the enhanced screen according to the correspondence.

6. A video regional decoding and display method, characterized in that, The method includes: Obtain the base bitstream and enhanced region bitstream to be displayed in the target display area; Wherein, the base bitstream and the enhanced region bitstream are obtained by encoding based on the source video under the video regionalization coding method according to any one of claims 1-5, the base bitstream corresponds to each region unit within the region boundary of the source video, the enhanced region bitstream corresponds to the enhanced region unit therein, and the bitrate of the enhanced region bitstream is higher than that of the base bitstream. The base bitstream and the enhanced region bitstream are decoded respectively to obtain the base image and the enhanced image; The base image is displayed over the entire target display area, and the base image at the corresponding position of the enhanced area unit within the target display area is adjusted based on the enhanced image.

7. The video regional decoding and display method according to claim 6, characterized in that, The step of decoding the base bitstream and the enhanced region bitstream respectively to obtain the base image and the enhanced image includes: Obtain the current playback state within the target display area, and determine the target enhancement region unit from multiple enhancement region units based on the current playback state; The target enhanced region bitstream corresponding to the target enhanced region unit is determined from the obtained enhanced region bitstream; The base stream and the target enhancement region stream are decoded respectively to obtain the base image and the enhancement image.

8. The video regional decoding and display method according to claim 7, characterized in that, Each of the aforementioned enhancement region units has multiple quality levels, and each quality level corresponds to a different bitrate; The step of determining the target enhancement region unit from multiple enhancement region units based on the current playback state includes: For each of the aforementioned enhancement region units, based on the current playback state, multiple benefit information of the enhancement region unit at multiple quality levels is obtained; The total revenue is obtained by summing up multiple revenue data points according to the set weights. Under constraints, a preset number of enhancement region units with higher total benefits are selected, and the selected enhancement region units are used as target enhancement region units to obtain the target quality level of the target enhancement region units.

9. The video regional decoding and display method according to claim 8, characterized in that, The multiple benefit information includes the regional importance of the enhanced region unit, the quality gain of the selected quality level relative to the base bitrate, the switching cost of switching from the current quality level to the selected quality level, the bandwidth usage of the enhanced region unit, and the decoding resource usage.

10. The video regional decoding and display method according to claim 9, characterized in that, The step of adjusting the base image at the corresponding position of the enhanced area unit within the target display area based on the enhanced image includes: Obtain a region mapping table, which stores the correspondence between the region attributes of each region unit and the coordinate information of each location point in the target display area; Based on the correspondence in the region mapping table, determine the coordinate information of the enhanced region unit in the target display region; The enhanced image is projected onto the location corresponding to the determined coordinate information in order to adjust the base image at the corresponding location.

11. The video regional decoding and display method according to claim 10, characterized in that, The step of adjusting the base image at the corresponding location includes: The enhanced image can be used to cover the base image at the corresponding location, or the enhanced image and the base image at the corresponding location can be merged.

12. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1-5, or the method of any one of claims 6-11.

Images