Scalable audio-video encoding method and system thereof

Abstract

The application provides a scalable audio-video coding method and system. The method comprises the following steps: dividing an original picture collected into multiple regions; obtaining coding resolution according to the position and content of each region; coding the original picture according to the coding resolution to obtain a coded picture; and sending the coded picture to a receiving end. The scalable audio-video coding method and system can ensure the definition and accuracy of key content, significantly optimize the bandwidth utilization, improve the transmission efficiency and stability, enhance the user experience, and adapt to the performance of various devices.

Description

Technical Field

[0001] This invention relates to the field of video communication technology, and more specifically, to a scalable audio and video coding method and system. Background Technology

[0002] In today's digital age, video communication has become an indispensable part of people's daily lives and work. With the popularization of the internet and the continuous development of network technology, video communication technology is also constantly evolving to meet people's demands for high quality, real-time performance, and flexibility. Traditional video communication systems often face many challenges, such as network bandwidth limitations, differences in receiving equipment, and varying user requirements for video quality.

[0003] To address these issues, SVC (Scalable Video Coding) was developed. Unlike traditional AVC (Advanced Video Coding), SVC does not use a fixed encoding resolution and bitrate to compress and transmit video data. Instead, it divides the video into multiple different quality levels, each of which can be encoded and transmitted independently. Therefore, SVC allows for dynamic adjustment of video quality based on network conditions and device performance, thus better adapting to different network environments and receiving devices.

[0004] However, existing SVC technology still has some shortcomings in practical applications. First, existing SVC typically cannot make fine-grained adjustments based on the characteristics of different areas in the image. This one-size-fits-all encoding method can easily lead to a decline in overall video quality when there are large fluctuations in network bandwidth or significant differences in the performance of receiving devices, failing to guarantee the clarity and stability of critical areas.

[0005] Secondly, existing SVC technology has low encoding efficiency when processing complex video content. Due to the lack of a dynamic adjustment mechanism for the content, highly complex or important content (such as faces, text, etc.) may still appear blurry or unclear in low-bandwidth environments, affecting the user experience. Summary of the Invention

[0006] This invention provides a scalable audio and video encoding method and system. The original image is subdivided into multiple regions, each region being a part of the image. Then, the corresponding encoding resolution is determined based on the location and content level of each region. Specifically, this involves a comprehensive analysis of the region's location and content to personalize the encoding of different regions during the encoding process. For example, higher encoding resolutions are assigned to important content in the image, such as faces or text, to ensure clarity during transmission, while lower encoding resolutions are used for backgrounds or static content to save bandwidth and resources. This invention, through regional division and dynamic encoding based on region location and content, makes video encoding and transmission more flexible and intelligent, enabling finer-grained adjustments to encoding quality according to the actual content, thereby improving the efficiency and stability of video encoding and transmission.

[0007] In a first aspect, the present invention provides a scalable audio and video encoding method, characterized in that the method includes:

[0008] The captured raw images are divided into multiple regions;

[0009] The encoding resolution is determined based on the location and content of each region;

[0010] The original image is encoded according to the encoding resolution to obtain the encoded image;

[0011] The encoded image is sent to the receiving end.

[0012] Secondly, the present invention also provides a scalable audio and video encoding system, characterized in that the system includes a transmitting end and a receiving end; wherein the transmitting end includes:

[0013] A segmentation unit, which is used to divide the captured raw image into multiple regions;

[0014] A determining unit, which is used to obtain the encoding resolution based on the location and content of each region;

[0015] An encoding unit is used to encode the original image according to the encoding resolution to obtain an encoded image;

[0016] A sending unit is used to send the encoded image to the receiving end.

[0017] The scalable audio and video coding method and system provided by this invention, by using the importance of different content and different positions of the image as the basis for determining the coding resolution, can ensure the clarity and accuracy of key content compared with existing technologies under the premise of the same real-time bandwidth and receiver performance. On the other hand, by dividing the original image into multiple regions for fine-grained coding optimization and allocating different coding parameters to different regions, higher resolution can be given to important regions (such as faces, text, etc.) while the resolution of unimportant regions (such as background, etc.) can be reduced. This fine-grained adjustment method allocates coding resources in a targeted manner, avoiding unnecessary high-resolution coding of the entire image, reducing redundant calculations, saving computing resources and bandwidth, and the adaptive strategy improves the overall coding efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the scalable audio and video encoding method provided in an embodiment of the present invention;

[0020] Figure 2 This is a block diagram of a scalable audio and video encoding system provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Invention Overview

[0023] As mentioned above, this invention provides a scalable audio and video encoding method and system. By dividing the encoding into regions and using dynamic encoding based on region location and content, it can ensure the clarity and accuracy of key content, significantly optimize bandwidth utilization, improve transmission efficiency and stability, enhance user experience, and adapt to the performance of various devices.

[0024] Exemplary methods

[0025] Figure 1 This is a flowchart of a scalable audio and video encoding method provided in an embodiment of the present invention. This embodiment includes the following steps:

[0026] S101: Divide the captured raw image into multiple regions.

[0027] For example, an image segmentation algorithm (such as superpixel segmentation or other region partitioning methods) can be used to divide the original image into multiple rectangular regions of the same size (such as 3×3, 4×4, etc.) using a uniform grid.

[0028] By dividing the original image into multiple regions, each region can be processed and encoded independently. This supports applying different encoding strategies to different parts of the image, enabling a more rational allocation of encoding resources and thus improving the precision and accuracy of encoding.

[0029] S102: Obtain the encoding resolution based on the location and content of each region.

[0030] The encoding resolution can be either the regional resolution of each area or the overall resolution of the entire original image. The choice of resolution depends on the encoding method. If the encoding method is regional encoding, the encoding resolution is the regional resolution of each area, with each area encoded independently according to its own regional resolution. If the encoding method is overall encoding, the encoding resolution is the overall resolution of the entire original image, with the entire original image encoded using a uniform overall resolution. Each area can have one or more regional resolutions, and the overall resolution can also be one or more.

[0031] More specifically, the encoding method, whether regional encoding or overall encoding, is determined based on the encoding capabilities of the sending end, the performance of the receiving end, the real-time bandwidth, and / or the user interaction behavior of the receiving end.

[0032] The encoding capability of the sending end refers to whether the sending end has sufficient computing power and hardware configuration to perform complex regional encoding. The performance of the receiving end refers to the ability of the device receiving the video stream to decode and display the video, including but not limited to the receiving end's decoding capability (whether the receiving end's hardware can handle regional decoding), display capability, audio processing capability, and network performance. User interaction behaviors include but are not limited to eye tracking, mouse clicks, touch, swiping, and zooming. These user interaction behaviors are used to determine the areas of focus for the user.

[0033] For example, if sender A has regional encoding capability, and receiver a also has regional decoding capability, with sufficient real-time bandwidth, and the user at receiver a zooms in on the center area of ​​the video image, then sender A will choose regional encoding, using high-resolution encoding for the zoomed-in area and low-resolution encoding for the other undisplayed areas. Sender B, however, does not have regional encoding capability and will choose overall encoding.

[0034] Specifically, the encoding resolution can be calculated based on the positional and content weights of each region. For example, first define the positional weight of each region (e.g., high weight for the central region, low weight for the edge regions) and the weights for various content types (high weight for faces and text, low weight for backgrounds and static areas); then calculate the weighted sum of the positional and content weights for each region as the total weight for that region; based on the total weight and a preset resolution range, calculate the encoding resolution for each region; if the encoding method is overall encoding, then calculate the overall resolution by weighted averaging the regional resolutions of each region. There are various methods to obtain the overall resolution based on the regional resolutions of each region, including the weighted averaging method, the maximum resolution method, the minimum resolution method, the area-weighted averaging method, and the adaptive adjustment method, etc. The choice of method depends on the specific application scenario and requirements.

[0035] For a more specific example, the position weight of the central region is defined as 1.0, and the position weight of the edge regions is 0.5; the content weights are 1.0 for faces, 0.8 for text, 0.2 for background, and 0.3 for static areas. The weighted sum of the position weight and content weight is calculated for each region. Region 1: Located in the center, contains a face, total weight = position weight 1.0 + content weight 1.0 = 2.0; Region 2: Located in the center, contains text, total weight = position weight 1.0 + content weight 0.8 = 1.8; Region 3: Located on the edge, contains the background, total weight = position weight 0.5 + content weight 0.2 = 0.7; Region 4: Located on the edge, contains static areas, total weight = position weight 0.5 + content weight 0.3 = 0.8. The encoding resolution of each region is obtained based on its total weight and a preset resolution range: Region 1 with a total weight of 2.0: 1080p (high resolution); Region 2 with a total weight of 1.8: 720p (medium-high resolution); Region 3 with a total weight of 0.7: 360p (low resolution); Region 4 with a total weight of 0.8: 480p (medium-low resolution). If the encoding method is overall encoding, the weighted average of the region resolutions is calculated to obtain the overall resolution. Assuming that the four regions have equal proportions, the overall resolution is calculated by weighted averaging of the number of pixels: 1080p: 1920*1080 = 2,073,600 pixels, 720p: 1280*720 = 921,600 pixels, 360p: 640*360 = 230,400 pixels, 480p: 854*480 = 409,920 pixels. Calculate the weighted average number of pixels: (2,073,600*0.25)+(921,600*0.25)+(230,400*0.25)+(409,920*0.25)=518,400+230,400+57,600+102,480=908,880. Since 908,880 falls between 720p and 1080p, and is closer to 720p, the overall resolution after weighted averaging is approximately 720p.

[0036] Encoding resolution can also be obtained based on a lookup table according to the location and content levels of each region. For example, first, the location and content of the region are divided into different levels; then, according to the different location and content levels, the corresponding encoding resolution is preset, that is, a two-dimensional lookup table is created, where rows represent location levels, columns represent content levels, and each cell stores the corresponding encoding resolution; using the location and content levels as indexes, the corresponding encoding resolution can be quickly obtained from the lookup table.

[0037] More specifically, location levels can be categorized as follows: Location 1: Center area, Location 2: Middle area, Location 3: Edge area; content levels can be categorized as follows: Content 1: Face, Content 2: Text, Content 3: Background, Content 4: Still area. A two-dimensional lookup table can be created, where rows represent location levels, columns represent content levels, and each cell stores the corresponding encoding resolution. Using location level and content level as indexes, quickly obtain the corresponding encoding resolution: Region 1: Located at location 1 (center region), contains content 1 (face), location level is location 1, content level is content 1, and the corresponding encoding resolution in the lookup table is 1080p; Region 2: Located at location 2 (middle region), contains content 2 (text), location level is location 2, content level is content 2, and the corresponding encoding resolution in the lookup table is 720p; Region 3: Located at location 3 (edge ​​region), contains content 3 (background), location level is location 3, content level is content 3, and the corresponding encoding resolution in the lookup table is 360p; Region 4: Located at location 3 (edge ​​region), contains content 4 (static region), location level is location 3, content level is content 4, and the corresponding encoding resolution in the lookup table is 360p.

[0038] By dividing the location and content of a region into different levels and creating a two-dimensional lookup table with a preset encoding resolution, the corresponding encoding resolution can be obtained quickly and accurately based on the location and content level. The advantage of this method is that the encoding resolution can be obtained quickly in O(1) time complexity using a lookup table, improving encoding efficiency. That is, the time required for the lookup process is fixed regardless of the size of the lookup table. Furthermore, the preset lookup table can be adjusted according to actual application needs and scenarios, flexibly adapting to different video encoding requirements. Through the dual-level division of location and content, fine-grained control of video encoding quality can be achieved, ensuring that key content and important areas receive higher quality encoding.

[0039] The position weight and position level are obtained based on the image position importance model, which includes the center priority model, the golden ratio model, and / or the heat map model.

[0040] The center-first model refers to the fact that users usually notice the central area of ​​the screen first, therefore the central area is of higher importance, while the edge areas are of lower importance. For example, the central area has the highest positional weight and positional level, the surrounding area has a medium positional weight and positional level, and the edge areas have the lowest positional weight and positional level.

[0041] The golden ratio model refers to the model based on the golden ratio theory, where users' attention is generally focused on the dividing lines and intersections of the screen, and these areas are considered to be of high importance.

[0042] A heatmap model refers to generating a heatmap by analyzing a large amount of user viewing behavior data, identifying the areas that users pay the most attention to, and assigning them a higher rating.

[0043] The content weight and content level are obtained based on content importance, content complexity, and / or user attention.

[0044] Highly important content includes faces, text, actions, foreground objects, and / or dynamic areas.

[0045] Faces are one of the most common and important elements in videos, especially in applications such as video conferencing, social media, and human-computer interaction. The facial area contains rich details and expressive information, and users typically focus their attention on the facial expressions and movements of other participants; therefore, high resolution is required to ensure clear and smooth display.

[0046] Textual information appearing in a video, such as subtitles, labels, signs, or text on a display screen, is also important visual information. Clear text is crucial for understanding video content, obtaining information, and interacting with it.

[0047] The actions in the video include the movement of people (such as gestures and steps) and the movement of objects (such as vehicles moving and objects falling), conveying the important activities and changes in the video.

[0048] Objects in the foreground of a video, such as people, pets, vehicles, or other important elements, are typically distinguished from the background. Foreground objects are often the main content of the video and are crucial for user experience and understanding of the video's context. They need to be displayed in high quality to ensure clarity of detail and visual appeal.

[0049] Dynamic regions refer to areas in a video where there is significant change, such as the content being played on the screen, fast-moving objects, or other dynamic effects. Dynamic regions require higher frame rates and correspondingly lower encoding resolutions to avoid stuttering and ensure the overall smoothness and clarity of the video.

[0050] The content complexity is obtained based on image texture richness, structural complexity, and / or color variability. Images with high texture richness contain a large amount of texture and detail, such as leaves, patterns, and building surfaces.

[0051] The user attention level is obtained based on the detection of user interaction behavior at the receiving end. Eye tracking, mouse clicks, touch, and other interactive behaviors are used to detect the area of ​​the screen currently being focused on by the user. The user attention level for different areas is determined based on the area being focused on. For example, areas with high user attention have increased resolution and bitrate, while other areas have decreased resolution to save bandwidth and computing resources.

[0052] In addition, prior to step S102, the bitrate can be obtained based on real-time bandwidth and / or receiver performance. Higher real-time bandwidth and stronger receiver performance result in a higher bitrate. The first region resolution is obtained based on the correspondence between bitrate and resolution. A higher bitrate corresponds to a higher resolution or frame rate. A higher bitrate means transmitting more data, thus supporting higher resolution or higher frame rate video and providing clearer, more detailed images. Step S102 specifically involves determining the second region resolution within the first region resolution based on the location and content of each region. The first region resolution can be a set of resolutions. Step S102 further narrows down the range within the first region resolution by combining the location and content of each region to determine the second region resolution.

[0053] By narrowing down the selection resolution range twice, the optimal resolution for each region can be determined more precisely. The first narrowing is based on the overall bitrate and receiver performance, ensuring that the selected region's resolution does not exceed the network and device's capacity. The second narrowing is based on the location and content of each region, further optimizing the resolution of each region to ensure stable and smooth video transmission. In short, by comprehensively considering various factors, the appropriate resolution for each region can be determined more accurately to meet the user's viewing needs.

[0054] Furthermore, the present invention can also obtain the audio bitrate based on real-time bandwidth and / or receiver performance.

[0055] In video transmission, besides the video bitrate, the audio bitrate is also a crucial factor. Dynamically adjusting the audio bitrate based on real-time bandwidth and receiver performance can further optimize audio transmission. For example, in situations with low network bandwidth or limited receiver performance, the audio bitrate can be reduced to save bandwidth and ensure smooth audio transmission. Conversely, when network bandwidth is ample and receiver performance is good, the audio bitrate can be increased to provide a higher quality audio experience.

[0056] S103: Encode the original image according to the encoding resolution to obtain the encoded image.

[0057] As mentioned earlier, this step can be either regional encoding or overall encoding.

[0058] Specifically, the encoded image is obtained based on SVC (Scalable Video Coding) encoding.

[0059] SVC divides the video stream into multiple different layers, each corresponding to a different resolution. Each layer can be encoded and transmitted independently, with the base layer providing the lowest resolution and the enhancement layer providing a higher resolution.

[0060] S104: Send the encoded image to the receiving end.

[0061] In addition, the receiving end can select the resolution and / or bitrate of the encoded image based on its performance and / or user interaction behavior to obtain the playback image; the receiving end then displays the playback image. The receiving end can dynamically select the required video quality level based on its own performance and user needs to ensure optimal playback performance under different device and network conditions.

[0062] For example, when the device load is low, a higher resolution and bitrate can be selected; when the device load is high, a lower resolution and bitrate should be selected to ensure smooth playback.

[0063] Exemplary System

[0064] Accordingly, embodiments of the present invention also provide a scalable audio and video encoding system. Figure 2 This is a block diagram of the scalable audio and video encoding system provided in an embodiment of the present invention, such as... Figure 2 As shown, the system 100 provided in this embodiment includes: a transmitter 101 and a receiver 102; wherein the transmitter 101 includes:

[0065] The segmentation unit 103 is used to divide the captured raw image into multiple regions;

[0066] Determining unit 104 is used to obtain the encoding resolution based on the location and content of each region;

[0067] Encoding unit 105 is used to encode the original image according to the encoding resolution to obtain the encoded image;

[0068] The sending unit 106 is used to send the encoded image to the receiving end 102.

[0069] The determining unit 104 includes:

[0070] Weighting module 107 is used to calculate the encoding resolution based on the positional weight and content weight of each region; and / or

[0071] The lookup module 108 is used to obtain the encoding resolution based on the location level and content level of each region according to the lookup table.

[0072] The position weight and position level are obtained based on the image position importance model, which includes the center priority model, the golden ratio model, and / or the heat map model.

[0073] The content weight and content level are obtained based on content importance, content complexity, and / or user attention.

[0074] Highly important content includes faces, text, actions, foreground objects, and / or dynamic areas;

[0075] The content complexity is obtained based on image texture richness, structural complexity, and / or color variation.

[0076] The user attention level is obtained based on the user interaction behavior detection of the receiving end 102, and the user interaction behavior includes eye tracking, mouse click, touch, swipe and zoom in / out.

[0077] The transmitting end 101 further includes:

[0078] A unit for determining whether the encoding method is regional encoding or overall encoding based on the encoding capability of the transmitter 101, the performance of the receiver, the real-time bandwidth and / or the user interaction behavior of the receiver 102. If the encoding method is regional encoding, the encoding resolution is the regional resolution of each region. If the encoding method is overall encoding, the encoding resolution is the overall resolution of the entire original image.

[0079] The transmitting end 101 further includes:

[0080] A unit used to obtain the code rate based on real-time bandwidth and / or receiver performance;

[0081] A unit for obtaining the resolution of the first region based on the correspondence between the bit rate and the resolution;

[0082] The determining unit 104 is further configured to determine the second region resolution in the first region resolution based on the location and content of each region.

[0083] The transmitting end 101 further includes:

[0084] A unit used to obtain the audio bitrate based on real-time bandwidth and / or receiver performance.

[0085] The receiving end 102 includes:

[0086] Selection unit 109 is used to select the resolution and / or bitrate of the encoded image based on the performance of the receiving end and / or user interaction behavior, so as to obtain the playback image;

[0087] Display unit 110 is used to display the playback screen.

[0088] The encoded image is obtained based on SVC encoding.

[0089] It should be noted that although the operations of the scalable audio and video coding method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0090] Furthermore, although several devices, units, or modules of the scalable audio and video coding system have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0091] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0092] This invention provides: 1. A scalable audio and video encoding method, characterized in that the method includes: The captured raw images are divided into multiple regions; The encoding resolution is determined based on the location and content of each region; The original image is encoded according to the encoding resolution to obtain the encoded image; The encoded image is sent to the receiving end. 2. The scalable audio and video encoding method according to item 1, characterized in that obtaining the encoding resolution based on the location and content of each region specifically involves: The encoding resolution is calculated based on the location weight and content weight of each region; or The encoding resolution is obtained based on the location level and content level of each region using a lookup table. 3. The scalable audio and video coding method according to item 2 is characterized in that the position weight and the position level are obtained based on the image position importance model, and the image importance model includes the center priority model, the golden section model and / or the heat map model. 4. The scalable audio and video encoding method according to item 2, characterized in that the content weight and the content level are obtained based on content importance, content complexity and / or user attention. 5. The scalable audio and video encoding method according to item 4, characterized in that the highly important content includes faces, text, actions, foreground objects and / or dynamic regions; The content complexity is obtained based on image texture richness, structural complexity, and / or color variation. The user attention level is obtained based on the detection of user interaction behavior at the receiving end, including eye tracking, mouse click, touch, swipe, and zoom in / out. 6. The scalable audio / video coding method according to any one of items 1-5, characterized in that the method further includes: The encoding method is determined as either regional encoding or overall encoding based on the encoding capability of the sending end, the performance of the receiving end, the real-time bandwidth, and / or the user interaction behavior of the receiving end. If the encoding method is regional encoding, the encoding resolution is the regional resolution of each region. If the encoding method is overall encoding, the encoding resolution is the overall resolution of the entire original image. 7. The scalable audio and video encoding method according to any one of items 1-5, characterized in that, before obtaining the encoding resolution based on the location and content of each region, the method further includes: The bit rate is obtained based on real-time bandwidth and / or receiver performance. The resolution of the first region is obtained based on the correspondence between the bit rate and the resolution. The specific steps for obtaining the encoding resolution based on the location and content of each region are as follows: The resolution of the second region is determined based on the location and content of each region within the resolution of the first region. 8. The scalable audio / video coding method according to any one of items 1-5, characterized in that the method further includes: The audio bitrate is obtained based on real-time bandwidth and / or receiver performance. 9. The scalable audio / video coding method according to any one of items 1-5, characterized in that the method further includes: The receiving end selects the resolution and / or bitrate of the encoded image based on its performance and / or user interaction behavior to obtain the playback image; The receiving end displays the playback screen. 10. The scalable audio and video encoding method according to any one of items 1-5, characterized in that the encoded image is obtained based on SVC encoding. 11. A scalable audio and video encoding system, characterized in that the system comprises a transmitting end and a receiving end; wherein the transmitting end comprises: A segmentation unit, which is used to divide the captured raw image into multiple regions; A determining unit, which is used to obtain the encoding resolution based on the location and content of each region; An encoding unit is used to encode the original image according to the encoding resolution to obtain an encoded image; A sending unit is used to send the encoded image to the receiving end. 12. The scalable audio / video coding system according to item 11, characterized in that the determining unit comprises: The weighting module is used to calculate the encoding resolution based on the positional and content weights of each region; and / or The lookup module is used to obtain the encoding resolution based on the location level and content level of each region according to the lookup table. 13. The scalable audio and video coding system according to item 12, characterized in that the position weight and the position level are obtained based on a picture position importance model, wherein the picture importance model includes a center-first model, a golden section model, and / or a heatmap model. 14. The scalable audio and video coding system according to item 12, wherein the content weight and the content level are obtained based on content importance, content complexity and / or user attention. 15. The scalable audio and video coding system according to item 14, wherein the highly important content includes faces, text, actions, foreground objects and / or dynamic regions; The content complexity is obtained based on image texture richness, structural complexity, and / or color variation. The user attention level is obtained based on the detection of user interaction behavior at the receiving end, including eye tracking, mouse click, touch, swipe, and zoom in / out. 16. The scalable audio / video coding system according to any one of items 11-15, characterized in that the transmitting end further includes: A unit for determining whether the encoding method is regional encoding or overall encoding based on the encoding capability of the transmitting end, the performance of the receiving end, the real-time bandwidth and / or the user interaction behavior of the receiving end. If the encoding method is regional encoding, the encoding resolution is the regional resolution of each region. If the encoding method is overall encoding, the encoding resolution is the overall resolution of the entire original image. 17. The scalable audio / video coding system according to any one of items 11-15, characterized in that the transmitting end further includes: A unit used to obtain the code rate based on real-time bandwidth and / or receiver performance; A unit for obtaining the resolution of the first region based on the correspondence between the bit rate and the resolution; The determining unit is also used to determine the second region resolution in the first region resolution based on the location and content of each region. 18. The scalable audio / video coding system according to any one of items 11-15, characterized in that the transmitting end further comprises: A unit used to obtain the audio bitrate based on real-time bandwidth and / or receiver performance. 19. The scalable audio / video coding system according to any one of claims 11-15, characterized in that the receiving end comprises: The selection unit is used to select the resolution and / or bitrate of the encoded image based on the performance of the receiving end and / or user interaction behavior, so as to obtain the playback image; The display unit is used to display the playback screen. 20. The scalable audio and video encoding system according to any one of items 11-15, characterized in that the encoded image is obtained based on SVC encoding.

Claims

1. A scalable audio and video encoding method, characterized in that, The method includes: The captured raw images are divided into multiple regions; The encoding method is determined as either regional encoding or overall encoding based on the encoding capability of the sending end, the performance of the receiving end, the real-time bandwidth, and / or the user interaction behavior of the receiving end. If the encoding method is regional encoding, the encoding resolution is the regional resolution of each region. If the encoding method is overall encoding, the encoding resolution is the overall resolution of the entire original image. The encoding resolution is obtained based on the location and content of each region. The location weight and location level are obtained based on the image location importance model, which includes the center priority model, the golden ratio model and / or the heat map model. Content weight and content level are obtained based on content importance, content complexity and / or user attention. The user attention is obtained based on the detection of user interaction behavior at the receiving end. The user interaction behavior includes eye tracking, mouse click, touch, swipe and zoom in / out. The bit rate is obtained based on real-time bandwidth and / or receiver performance; the first region resolution is obtained based on the correspondence between the bit rate and resolution. The specific steps for obtaining the encoding resolution based on the location and content of each region are as follows: determining the second region resolution based on the location and content of each region within the first region resolution; The original image is encoded according to the encoding resolution to obtain the encoded image; The receiving end selects the resolution and / or bitrate of the encoded image based on its performance and / or user interaction behavior to obtain the playback image; the receiving end then displays the playback image. The encoded image is sent to the receiving end.

2. The scalable audio and video encoding method according to claim 1, characterized in that, The encoding resolution is obtained based on the location and content of each region as follows: The encoding resolution is calculated based on the location weight and content weight of each region; or The encoding resolution is obtained based on the location level and content level of each region using a lookup table.

3. The scalable audio and video encoding method according to claim 1, characterized in that, Highly important content includes faces, text, actions, foreground objects, and / or dynamic areas; The content complexity is obtained based on image texture richness, structural complexity, and / or color variation.

4. The scalable audio / video coding method according to any one of claims 1-3, characterized in that, The method further includes: The audio bitrate is obtained based on real-time bandwidth and / or receiver performance.

5. The scalable audio / video coding method according to any one of claims 1-3, characterized in that, The encoded image is obtained based on SVC encoding.

6. A scalable audio and video encoding system, characterized in that, The system includes a transmitter and a receiver; wherein the transmitter includes: A segmentation unit, which is used to divide the captured raw image into multiple regions; The determining unit is used to determine whether the encoding method is regional encoding or overall encoding based on the encoding capability of the transmitting end, the performance of the receiving end, the real-time bandwidth and / or the user interaction behavior of the receiving end. If the encoding method is regional encoding, the encoding resolution is the regional resolution of each region. If the encoding method is overall encoding, the encoding resolution is the overall resolution of the entire original image. The unit also obtains the encoding resolution based on the position and content of each region. The position weight and position level are obtained based on the image position importance model, which includes the center priority model, the golden ratio model and / or the heat map model. Content weight and content level are obtained based on content importance, content complexity and / or user attention. The user attention is obtained based on the detection of user interaction behavior at the receiving end. The user interaction behavior includes eye tracking, mouse click, touch, swipe and zoom in / out. The transmitting end further includes a unit for obtaining the code rate based on real-time bandwidth and / or receiving end performance, and a unit for obtaining the resolution of the first region based on the correspondence between the code rate and the resolution. The determining unit is also used to determine the second region resolution in the first region resolution based on the location and content of each region. An encoding unit is used to encode the original image according to the encoding resolution to obtain an encoded image; The receiving end includes a selection unit, which is used to select the resolution and / or bitrate of the encoded image based on the performance of the receiving end and / or user interaction behavior, so as to obtain a playback image; and a display unit, which is used to display the playback image. A sending unit is used to send the encoded image to the receiving end.

7. The scalable audio and video encoding system according to claim 6, characterized in that, The determining unit includes: The weighting module is used to calculate the encoding resolution based on the positional and content weights of each region; and / or The lookup module is used to obtain the encoding resolution based on the location level and content level of each region according to the lookup table.

8. The scalable audio and video encoding system according to claim 6, characterized in that, Highly important content includes faces, text, actions, foreground objects, and / or dynamic areas; The content complexity is obtained based on image texture richness, structural complexity, and / or color variation.

9. The scalable audio / video encoding system according to any one of claims 6-8, characterized in that, The transmitting end also includes: A unit used to obtain the audio bitrate based on real-time bandwidth and / or receiver performance.

10. The scalable audio / video coding system according to any one of claims 6-8, characterized in that, The encoded image is obtained based on SVC encoding.

Images