Image coding method and device for video call, equipment, storage medium and product

Abstract

The embodiment of the invention provides an image coding method and device for a video call, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the detection bandwidth for carrying out the real-time video call with the second terminal, the video playing frame rate of the second terminal and the video packet loss rate of the first terminal are determined; a target code table is obtained by increasing the minimum code rate and the maximum code rate of a preset gear in a preset code table, the coding resolution is determined according to the detection bandwidth and the target code table, video coding is carried out on a to-be-coded image of a video call based on the coding resolution to obtain coded video information, and under the scene that the network condition is general or the network fluctuation is large, the coding efficiency is improved. According to the invention, the minimum code rate and the maximum code rate of the preset gear in the preset code table are increased, so that the image coding can trigger the down-cutting of the image coding resolution more easily, the video super-division processing can be triggered more easily at the second terminal, the call quality is improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to image encoding methods, devices, equipment, storage media and products for video calls. Background Art

[0002] Real-Time Communication (RTC) is a technology that conducts real-time audio and video communications over the Internet. With the rise and development of the Internet era, real-time communication has responded to people's urgent need for real-time communication.

[0003] A notable feature of real-time call scenarios is low latency. Users have higher requirements for the real-time performance of audio and video, and have lower tolerance for delays and freezes. Therefore, in addition to providing a clear and beautiful video playback experience, real-time calls also need to ensure smooth calls. The quality of the video encoded during a user's call is closely related to the network conditions. When the user's network conditions are good, the user's encoding resolution is high and the smoothness is good. When the user's network conditions are poor, the super-resolution algorithm can be used to provide users with better image quality support.

[0004] However, in some scenarios where the network conditions are average or the network fluctuates greatly, the user's network condition is not bad enough to use a small resolution that can trigger video super-resolution processing for video encoding. The image encoding resolution is maintained at a relatively high level, and it is difficult for the video playback end to trigger video super-resolution processing. However, the call has begun to be affected by the network and has begun to become stuck, resulting in a decrease in call quality. Summary of the invention

[0005] The embodiments of the present application provide image encoding methods, devices, equipment, storage media and products for video calls to solve the technical problem in the related art that in scenarios where the network conditions are general or the network fluctuations are relatively large, the image encoding resolution is maintained at a relatively high level, and it is difficult for the video playback end to trigger video super-resolution processing, resulting in a decrease in call quality. The embodiments of the present application can make it easier to trigger the down-cutting of the image encoding resolution and the video super-resolution processing in scenarios where the network conditions are general or the network fluctuations are relatively large, thereby improving call quality and enhancing user experience.

[0006] In a first aspect, an embodiment of the present application provides an image encoding method for a video call applied to a first terminal, including:

[0007] Determining a detection bandwidth for conducting a real-time video call with a second terminal;

[0008] Determine a video playback frame rate of the second terminal and a video packet loss rate of the first terminal;

[0009] When the video playback frame rate and the video packet loss rate meet the code table adjustment condition, increasing the minimum bit rate and the maximum bit rate of the preset gear in the preset code table to obtain the target code table;

[0010] The coding resolution is determined according to the detected bandwidth and the target code table, and video coding is performed on the to-be-coded image of the video call based on the coding resolution to obtain coded video information.

[0011] In a second aspect, an embodiment of the present application provides an image encoding device for a video call applied to a first terminal, including a bandwidth detection module, an information acquisition module, a code table adjustment module, and a video encoding module, wherein:

[0012] The bandwidth detection module is configured to determine a detection bandwidth for conducting a real-time video call with a second terminal;

[0013] The information acquisition module is configured to determine a video playback frame rate of the second terminal and a video packet loss rate of the first terminal;

[0014] The code table adjustment module is configured to increase the minimum bit rate and the maximum bit rate of the preset gear in the preset code table to obtain the target code table when the video playback frame rate and the video packet loss rate meet the code table adjustment condition;

[0015] The video encoding module is configured to determine the encoding resolution according to the detection bandwidth and the target code table, and perform video encoding on the to-be-encoded image of the video call based on the encoding resolution to obtain encoded video information.

[0016] In a third aspect, an embodiment of the present application provides an image encoding device for a video call, including: a memory and one or more processors;

[0017] The memory is used to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image encoding method for video calls as described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the image encoding method for video calls as described in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the image encoding method for video calls as described in the first aspect.

[0021] The embodiment of the present application determines the detection bandwidth for conducting a real-time video call with the second terminal, the video playback frame rate of the second terminal, and the video packet loss rate of the first terminal. When the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased to obtain the target code table, the encoding resolution is determined according to the detection bandwidth and the target code table, and the video encoding is performed on the image to be encoded of the video call based on the encoding resolution to obtain the encoded video information. In scenarios where the network condition is general or the network fluctuation is relatively large, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased, so that image encoding is easier to trigger a downward cut of the image encoding resolution and more difficult to trigger an upward cut of the image encoding resolution, and it is easier to trigger video super-resolution processing at the second terminal, thereby improving call quality and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of an image encoding method for video calls provided in an embodiment of the present application;

[0023] Figure 2 is a flowchart of another image encoding method for video calls provided in an embodiment of the present application;

[0024] Figure 3 It is a structural diagram of an image encoding device for video calls provided in an embodiment of the present application;

[0025] Figure 4 It is a structural schematic diagram of an image encoding device for video calls provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all of the content is shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0027] The image encoding method for video calls provided in the present application can be applied to image encoding during real-time video calls. It aims to increase the minimum bit rate and maximum bit rate of the preset gears in the preset code table in scenarios where the network conditions are general or the network fluctuations are relatively large, so that image encoding can more easily trigger a downward cut of the image encoding resolution and more difficult to trigger an upward cut of the image encoding resolution, and it is easier to trigger video super-resolution processing on the second terminal, thereby improving call quality and enhancing user experience.

[0028] In existing real-time video calls, a bandwidth detection algorithm is usually used to determine the bandwidth (i.e., bit rate) that can be used for image encoding in video calls, and then the corresponding resolution width and height are selected for image encoding according to the code table. When the detected bandwidth is in the minimum bit rate and maximum bit rate range of the resolution of a certain gear in the code table, it means that the current resolution can be selected for video encoding. Due to the limited computing power of mobile devices commonly used for real-time video calls, the super-resolution algorithm based on neural networks needs to limit the image resolution size of the input model if it wants to be used on mobile devices. On the other hand, some downsampling operators have comprehensive advantages in image quality and computing power when scaling to lower resolutions. When the user's network condition is good, the image encoding resolution is high and the video playback is smooth. When the user's network condition is poor, the super-resolution algorithm and downsampling algorithm can be used to provide users with image quality support, effectively alleviating the weak experience. On the one hand, due to the limited computing power of mobile devices commonly used in real-time call scenarios, many actual image pre- and post-processing operators cannot process images of all resolutions equally. On the other hand, a fixed set of code tables lacks flexibility, and cannot be adjusted in time to alleviate the freeze in some scenarios with general or large network fluctuations. In some scenarios where the network conditions are average or the network fluctuates greatly, the user's network condition is not bad enough to use a small resolution that can trigger super-resolution encoding, but their call has begun to be affected by the network and has begun to become stuck. The image encoding resolution remains at a relatively high level, and it is difficult for the video player to trigger video super-resolution processing, resulting in a decrease in call quality.

[0029] Based on this, an image encoding method for video calls in an embodiment of the present application is provided. By adjusting the code table for joint pre- and post-processing on both ends of the video first launch, the freeze situation is accurately detected, and the code table is dynamically adjusted in real time so that the encoding resolution tends to be cut down, and encoding is performed using a resolution that can trigger the pre- and post-processing operators. This is to solve the technical problem that in existing image encoding schemes, when the network conditions are general or the network fluctuations are relatively large, the image encoding resolution is maintained at a relatively high level, and it is difficult for the video playback end to trigger video super-resolution processing, resulting in a decrease in call quality.

[0030] Figure 1 A flowchart of an image encoding method for a video call provided in an embodiment of the present application is given. The image encoding method for a video call can be applied to a first terminal. The image encoding method for a video call provided in an embodiment of the present application can be executed by an image encoding device for a video call. The image encoding device for a video call can be implemented by hardware and / or software and integrated in an image encoding device for a video call.

[0031] The following description takes the video calling image encoding method performed by the video calling image encoding device as an example. Figure 1 , the image encoding method for video calls includes:

[0032] S110: Determine a detection bandwidth for conducting a real-time video call with a second terminal.

[0033] The first terminal and the second terminal provided in this solution are two terminal devices for video calls, wherein the first terminal is a terminal device for collecting (for example, collecting images through a camera or collecting desktop image information), encoding images to generate encoded video information, and sending the encoded video information to the second terminal (for example, forwarding the encoded video information to the second terminal through a preset server), and the second terminal device is a terminal device for receiving the encoded video information, decoding the encoded video information and playing it. Among them, a terminal device can be used as the first terminal and the second terminal at the same time. For example, a terminal device receives the encoded video information sent by other terminal devices and decodes and plays it while collecting and encoding images to generate encoded video information for sending.

[0034] Exemplarily, during a real-time video call between a first terminal and a second terminal, the bandwidth of the real-time video call with the second terminal can be detected in real time based on a preset bandwidth estimation algorithm (eg, BWE algorithm, Bandwidth Estimation), and the detection bandwidth can be determined according to the detection result.

[0035] S120: Determine the video playback frame rate of the second terminal and the video packet loss rate of the first terminal.

[0036] Exemplarily, the video playback frame rate of the video played by the second terminal is obtained in real time (the video playback frame rate can be collected and provided by the second terminal), and the video packet loss rate of the first terminal sending data (encoded video information) to the second terminal is counted in real time. For example, when the second terminal decodes and plays the received encoded video information, the frame rate information of the player playing the encoded video information is collected in real time, and the current frame rate information is used as the video playback frame rate, and uploaded to the preset server (such as a media server), and the first terminal can obtain the video playback frame rate corresponding to the second terminal from the preset server.

[0037] S130: When the video playback frame rate and the video packet loss rate meet the code table adjustment condition, increase the minimum bit rate and the maximum bit rate of the preset gear in the preset code table to obtain the target code table.

[0038] Exemplarily, before encoding the image to be encoded in the video call, determine whether the video playback frame rate and the video packet loss rate meet the code table adjustment conditions. The code table records multiple preset gears, and different gears are configured with corresponding resolutions, minimum bit rates, and maximum bit rates. The minimum bit rate and maximum bit rate in a gear can form a bit rate range. When it is necessary to encode the image, the encoding process can be performed based on the bit rate range corresponding to the latest determined detection bandwidth and the resolution in the same gear as the bit rate range.

[0039] In one embodiment, when the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate and maximum bit rate of the preset gear in the preset code table (i.e., the original code table) can be increased to obtain a target code table, so that the gear of the adjusted preset gear is easier to cut down and more difficult to cut up, wherein the preset gear can be one or more gears in the preset code table. At this time, the minimum bit rate and maximum bit rate of each preset gear in the target code table are improved relative to the minimum bit rate and maximum bit rate of the same gear in the preset code table. Compared with the original preset code table, the target code table can be used for encoding processing, so that the image encoding resolution as a whole shows a trend of being more difficult to cut up and easier to cut down, effectively reducing the resolution of image encoding based on the target code table, making it easier to trigger the pre- and post-processing algorithms with resolution restrictions, reducing the encoding and decoding time consumption, improving the smoothness of video playback, and can be combined with the pre- and post-processing operators with resolution restrictions to process the image frames, which can effectively maintain or even improve the image quality of real-time video calls.

[0040] In one embodiment, when the video playback frame rate and the video packet loss rate do not meet the code table adjustment condition, the current preset code table is maintained, and the to-be-encoded image of the video call is encoded according to the current preset code table to obtain encoded video information.

[0041] In a possible embodiment, when the newly determined video playback frame rate and video packet loss rate do not meet the code table adjustment conditions, and / or after the gear is switched based on the target code table, image encoding based on the preset code table can be restored.

[0042] S140: Determine a coding resolution according to the detected bandwidth and the target code table, and perform video coding on the to-be-coded image of the video call based on the coding resolution to obtain coded video information.

[0043] Exemplarily, after increasing the minimum bit rate and maximum bit rate of the preset gear in the preset code table to obtain the target code table, the corresponding bit rate range is determined in the target code table according to the above-determined detection bandwidth (that is, the detection bandwidth is between the corresponding minimum bit rate and maximum bit rate), and the resolution corresponding to the determined bit rate range is determined as the encoding resolution.

[0044] It needs to be explained that, since the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased, by increasing the minimum bit rate, the encoding resolution can be more easily triggered to down-cut during the downward migration process, and by increasing the maximum bit rate, the encoding resolution can be more difficult to trigger up-cut during the upward migration process. In a scenario where the network condition is general or the network fluctuation is relatively large, the image encoding based on the target code table will be more likely to switch to a low-resolution gear. When encoding the image to be encoded based on the low-resolution gear, the first terminal will have more computing power remaining at this time, and a downsampling processing algorithm and / or image pre-processing method with higher computing power requirements but better encoding effect can be adopted, which can achieve higher encoding quality at low encoding resolution. By increasing the minimum bit rate and the maximum bit rate of the preset gear in the preset code table, the encoding resolution tends to become smaller, which can reduce the computing power consumption of the first and terminal encoding and decoding end decoding, increase the encoding frame rate and the playback frame rate, thereby improving the fluency of the call, and can also switch the critical resolution that cannot trigger the pre- and post-processing operators to the resolution that can trigger the pre- and post-processing operators, thereby improving the picture quality of the video call.

[0045] After determining the encoding resolution, the image to be encoded for the video call can be encoded based on the encoding resolution to obtain the encoded video information. After the encoded video information is obtained by video encoding, the encoded video information can be sent to the second terminal (for example, a video stream is formed based on the continuously generated encoded video information and sent to the second terminal). After receiving the encoded video information, the second terminal can decode and play the encoded video information. Since the minimum bit rate and maximum bit rate of the preset gear in the target code table are increased, in the scenario where the network condition is general or the network fluctuation is relatively large, the image encoding will be more likely to switch to a low-resolution gear. When the second terminal decodes the encoded video information obtained based on the low-resolution gear encoding, the second terminal is more likely to trigger the video super-resolution processing of the encoded video information, thereby improving the smoothness of the video call without losing the image quality or even slightly improving the image quality, thereby improving the user's video call experience.

[0046] In the above, by determining the detection bandwidth for conducting a real-time video call with the second terminal, the video playback frame rate of the second terminal and the video packet loss rate of the first terminal, when the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased to obtain the target code table, the encoding resolution is determined according to the detection bandwidth and the target code table, and the video encoding is performed on the image to be encoded of the video call based on the encoding resolution to obtain the encoded video information. In scenarios where the network condition is general or the network fluctuation is relatively large, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased, so that image encoding is easier to trigger a downward cut of the image encoding resolution, and more difficult to trigger an upward cut of the image encoding resolution, and it is easier to trigger video super-resolution processing at the second terminal, thereby improving call quality and user experience.

[0047] Based on the above embodiments, Figure 2 A flowchart of another method for encoding an image for a video call provided in an embodiment of the present application is given. The method for encoding an image for a video call is a specific implementation of the method for encoding an image for a video call. Figure 2 , the image encoding method for video calls includes:

[0048] S210: Determine a detection bandwidth for conducting a real-time video call with a second terminal.

[0049] S220: Determine the video playback frame rate of the second terminal and the video packet loss rate of the first terminal.

[0050] S230: When the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate of the preset gear in the preset code table is increased according to the preset minimum bit rate adjustment parameter, and the maximum bit rate of the preset gear in the preset code table is increased according to the preset maximum bit rate adjustment parameter to obtain the target code table.

[0051] In a possible embodiment, whether the video playback frame rate and video packet loss rate provided by the present solution meet the code table adjustment condition can be determined based on a comparison result of the video playback frame rate and the video packet loss rate with corresponding preset thresholds.

[0052] Exemplarily, the video playback frame rate is compared with a preset frame rate threshold, and the video packet loss rate is compared with a preset packet loss threshold, and it is determined whether the video playback frame rate and the video packet loss rate meet the code table adjustment condition based on the comparison result. For example, when the video playback frame rate is less than the preset frame rate threshold, and the video packet loss rate is greater than the preset packet loss threshold, it can be considered that the video playback frame rate and the video packet loss rate meet the code table adjustment condition; it can also be considered that when the video playback frame rate is less than the preset frame rate threshold, or the video packet loss rate is greater than the preset packet loss threshold, the video playback frame rate and the video packet loss rate meet the code table adjustment condition. This solution accurately determines whether the code table adjustment condition is met based on the comparison results of the video playback frame rate and the preset frame rate threshold, and the video packet loss rate and the preset packet loss threshold, and accurately determines the timing of adjusting the code table, thereby ensuring call quality and improving user experience.

[0053] In one embodiment, after determining that the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table can be increased based on the preset adjustment parameters to obtain the target code table, where the adjustment parameters can be a proportional coefficient greater than 1, or a preset bit rate increase amplitude.

[0054] Exemplarily, the target code table can be obtained by increasing the minimum bit rate of a preset gear in the preset code table according to the preset minimum bit rate adjustment parameter, and increasing the maximum bit rate of a preset gear in the preset code table according to the preset maximum bit rate adjustment parameter. The preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter can be a proportional coefficient greater than 1, or a preset bit rate increase amplitude; the preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter can be the same or different; the preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter corresponding to different gears can be the same or different.

[0055] In one embodiment, when the preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter are proportional coefficients greater than 1, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table can be multiplied by the corresponding preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter respectively to obtain the minimum bit rate and the maximum bit rate after the preset gear is increased.

[0056] In one embodiment, when the preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter are the bit rate increase amplitude, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table can be added to the corresponding preset minimum bit rate adjustment parameter and the preset maximum bit rate adjustment parameter respectively to obtain the minimum bit rate and the maximum bit rate after the preset gear is increased.

[0057] This solution increases the minimum bit rate of a preset gear in the preset code table according to a preset minimum bit rate adjustment parameter, and increases the maximum bit rate of a preset gear in the preset code table according to a preset maximum bit rate adjustment parameter. The preset code table can be adjusted quickly and accurately to obtain a target code table, thereby ensuring call quality and improving user experience.

[0058] In one embodiment, when applying the bit rate adjustment parameter to adjust the preset code table, the gear corresponding to the critical resolution of the front and rear processing operators can be used as the preset gear, so that cutting down between the critical resolutions is easier and cutting up is more difficult. The code table adjustment of the dual-end joint front and rear processing can accurately detect the freeze of the video call, and the preset code table is adjusted in real time and dynamically so that the encoding resolution tends to be cut down. Encoding is performed using the resolution that can trigger the front and rear processing operators. This can not only alleviate the freeze of the video call to a certain extent, but also maintain or even improve the image quality by more flexibly triggering the front and rear processing operators.

[0059] The following table provides a preset code table:

[0060] Gear Resolution Minimum bit rate kbps Maximum bit rate kbps 1 R1 BR1min BR1max 2 R2 BR2min BR2max 3 R3 BR3min BR3max 4 R4 BR4min BR4max 5 R5 BR5min BR5max

[0061] For example, the resolution corresponding to gear 1 is R1, and its minimum bit rate and maximum bit rate are BR1min and BR1max respectively. Among them, the resolutions corresponding to gears 1-5 increase in sequence. Assume that the preset gears are R2 and R3, and the preset minimum bit rate adjustment parameter Fmin and the preset maximum bit rate adjustment parameter Fmax are proportionality coefficients greater than 1. Then, the target code table obtained by increasing the minimum bit rate and maximum bit rate of the preset gears in the preset code table based on the adjustment parameters is shown in the following table:

[0062] Gear Resolution Minimum bit rate kbps Maximum bit rate kbps 1 R1 BR1min BR1max 2 R2 BR2min*Fmin BR2max*Fmax 3 R3 BR3min*Fmin BR3max*Fmax 4 R4 BR4min BR4max 5 R5 BR5min BR5max

[0063] Among them, the minimum bit rate and maximum bit rate of the preset gear R2 are BR2min*Fmin and BR2max*Fmax respectively, and the minimum bit rate and maximum bit rate of the preset gear R3 are BR3min*Fmin and BR3max*Fmax respectively. The minimum bit rates and maximum bit rates of other gears remain unchanged.

[0064] It can be seen that when the minimum bit rate adjustment coefficient Fmin ∈ (1, +∞), the minimum bit rate of the preset gear becomes larger. Taking gear 2 as an example, previously, when the detection bandwidth BR < BR2min, the gear could be downshifted to gear 1 and encoded using a smaller resolution R1. However, after the minimum bit rate of the code table becomes larger, when and only when the detection bandwidth BR < BR2min*Fmin, the gear can be downshifted to gear 1, and the encoding bit rate within the range of the detection bandwidth BR ∈ (BR2min, BR2min*Fmin), which should not have been downshifted under the original logic, will start to downshift and then be encoded using the resolution R1. When the maximum bit rate adjustment coefficient Fmax ∈ (1, +∞), the maximum bit rate of the preset gear becomes larger. Taking gear 2 as an example, previously, when the detection bandwidth BR > BR2max, the gear could be upshifted to gear 3 and encoded using a larger resolution R3. However, after the maximum bit rate of the code table becomes larger, when and only when the detection bandwidth BR > BR2max*Fmax, the gear can be upshifted to gear 3, and the encoding bit rate within the range of BR ∈ (BR2max, BR2max*Fmax), which should have been upshifted under the original logic, will continue to be encoded using the resolution R2. It can be seen that in this solution, by increasing the minimum bit rate of the preset code table, it is easier to trigger downshifting during the downward migration of the encoding resolution, and by increasing the maximum bit rate of the preset code table, it is more difficult to trigger upshifting during the upward migration of the encoding resolution. On the one hand, it can reduce the computing power consumption of the sender's encoding and the decoder's decoding, improve the encoding frame rate and playback frame rate, thereby improving the call fluency. On the other hand, it can switch the critical resolution that cannot trigger the front and back processing operators to the resolution that can trigger the front and back processing operators, improve the image quality of the video call, and can improve the fluency of the video call without sacrificing image quality or even slightly improving the image quality, thereby improving the user experience.

[0065] S240: Determine a coding resolution according to the detected bandwidth and the target code table, and perform video coding on the to-be-coded image of the video call based on the coding resolution to obtain coded video information.

[0066] In a possible embodiment, the image encoding method for a video call provided by the present solution performs video encoding on an image to be encoded of a video call based on the encoding resolution to obtain encoded video information, which may be:

[0067] S2401: When the encoding resolution meets the processing computing power conditions, the first downsampling processing and image pre-processing are performed on the image to be encoded of the video call based on the first downsampling processing algorithm and the image pre-processing algorithm, and video encoding is performed on the image to be encoded after the first downsampling processing and image pre-processing to obtain encoded video information.

[0068] S2402: When the encoding resolution does not meet the processing computing power condition, a second downsampling process is performed on the image to be encoded of the video call based on a second downsampling processing algorithm, and video encoding is performed on the image to be encoded after the second downsampling process to obtain encoded video information.

[0069] Exemplarily, after determining the coding resolution in the target code table according to the detection bandwidth, it can be determined whether the coding resolution meets the preset processing power condition, wherein the coding resolution meets the preset processing power condition, which can be understood as the first terminal having excess computing power to perform more data processing when encoding an image based on the coding resolution. Optionally, the processing power condition can be a fixed configuration or dynamically configured based on different terminal models or processor models.

[0070] In one embodiment, whether the encoding resolution meets the processing power condition may be determined based on the comparison result between the encoding resolution and the first preset resolution. For example, when the encoding resolution is less than or equal to the preset resolution, the encoding resolution may be considered to meet the processing power condition. For example, the first preset resolution includes a first preset width and a first preset height, and the resolution width of the encoding resolution is compared with the first preset width, and the resolution height of the encoding resolution is compared with the first preset height. When the resolution width of the encoding resolution is less than or equal to the first preset width, and / or the resolution height of the encoding resolution is less than or equal to the first preset height, the encoding resolution is considered to meet the processing power condition.

[0071] For example, assuming that the resolution width and resolution height of the coding resolution are Ws and Hs respectively, and the first preset width and the first preset height of the first preset resolution are w1 and h1 respectively, when Ws≤w1 and Hs≤h1, it is considered that the current coding resolution is small enough, and the first terminal can use a downsampling processing algorithm with greater computing power requirements but better effects and an image pre-processing algorithm for image processing. At this time, it can be determined that the coding resolution meets the processing computing power conditions. If Ws is greater than w1 or Hs>h1, it is considered that the current coding resolution is large, and the load of the first terminal cannot support large-resolution image processing. A downsampling processing algorithm with faster speed but poorer effect can be used for processing, and image pre-processing is abandoned. At this time, it can be determined that the coding resolution does not meet the processing computing power conditions.

[0072] This solution accurately determines whether the encoding resolution meets the processing computing power conditions based on the comparison results of the resolution width of the encoding resolution and the first preset width, as well as the resolution height of the encoding resolution and the first preset height, adopts a more appropriate image processing method, effectively balances image processing efficiency and image processing quality, and improves the user's video call experience.

[0073] In one embodiment, when the encoding resolution meets the processing computing power conditions, the image to be encoded of the video call can be subjected to first downsampling processing and image pre-processing based on a first downsampling processing algorithm (which may be a downsampling processing algorithm with large computing power requirements but better effects, such as bicubic (bicubic interpolation), lanczos (Lanczos interpolation) and other downsampling processing algorithms) and image pre-processing (such as noise reduction, contrast enhancement, color conversion, etc.) algorithm, and video encoding is performed on the image to be encoded after the first downsampling processing and image pre-processing to obtain encoded video information.

[0074] In one embodiment, when the encoding resolution does not meet the processing computing power condition, a second downsampling processing can be performed on the image to be encoded of the video call based on a second downsampling processing algorithm (which can be a downsampling processing algorithm with lower computing power requirements but higher processing speed, such as a bilinear (bilinear interpolation) downsampling algorithm), and video encoding is performed on the image to be encoded after the second downsampling processing to obtain encoded video information. Image pre-processing is not required for the image to be encoded, thereby ensuring image coding efficiency.

[0075] This solution performs a first downsampling process and image pre-processing on the image to be encoded of the video call when the encoding resolution meets the processing computing power conditions, and performs video encoding on the image to be encoded after the first downsampling process and image pre-processing to obtain encoded video information. When the encoding resolution does not meet the processing computing power conditions, this solution performs a second downsampling process on the image to be encoded of the video call, and performs video encoding on the image to be encoded after the second downsampling process to obtain encoded video information, thereby effectively balancing image processing efficiency and image processing quality and improving user video call experience.

[0076] In a possible embodiment, the image encoding method for video calls provided by the present solution may further send the encoded video information to a second terminal after encoding the image to be encoded of the video call based on the encoding resolution to obtain the encoded video information, so that the second terminal can decode the encoded video information to obtain a decoded video frame, and when the decoded video frame meets a preset post-processing condition, perform image post-processing on the decoded video frame based on a preset image post-processing algorithm, and play the decoded video frame after image post-processing.

[0077] Exemplarily, after the image to be encoded of the video call is encoded to obtain the encoded video information, the encoded video information can be sent to the second terminal of the real-time video call. After receiving the encoded video information, the second terminal decodes the encoded video information to obtain a decoded video frame, and determines whether the decoded video frame meets the preset post-processing conditions. Among them, the decoded video frame meets the preset post-processing conditions, which can be understood as the second terminal has excess computing power to perform more data post-processing on the decoded video frame. Optionally, the preset post-processing conditions can be a fixed configuration, or can be dynamically configured based on different terminal models or processor models.

[0078] In one embodiment, whether the resolution of the decoded video frame satisfies the preset post-processing condition may be determined based on the comparison result between the resolution of the decoded video frame and the second preset resolution. For example, when the resolution of the decoded video frame is less than or equal to the second preset resolution, the decoded video frame may be considered to satisfy the preset post-processing condition. For example, the second preset resolution includes a second preset width and a second preset height, and the resolution width of the decoded video frame and the second preset width, and the resolution height of the decoded video frame and the second preset height are compared respectively. When the resolution width of the decoded video frame is less than or equal to the second preset width, and / or the resolution height of the decoded video frame is less than or equal to the second preset height, the decoded video frame is considered to satisfy the preset post-processing condition.

[0079] For example, assuming that the resolution width and resolution height of the decoded video frame are Wr and Hr respectively, and the second preset width and second preset height of the second preset resolution are w2 and h2 respectively, when Wr≤w2 and Hr≤h2, it is considered that the resolution of the current decoded video frame is small enough, and the second terminal can use a post-processing algorithm with greater computing power requirements but better effects for image post-processing. At this time, it can be determined that the decoded video frame meets the preset post-processing conditions. If Wr is greater than w2 or Hr>h2, it is considered that the resolution of the current decoded video frame is large, and the load of the second terminal cannot support the post-processing of the image with a large resolution, and the image post-processing can be abandoned. At this time, it can be determined that the decoded video frame does not meet the preset post-processing conditions.

[0080] This solution accurately determines whether the resolution of the decoded video frame meets the preset post-processing conditions based on the comparison results of the resolution width of the decoded video frame and the second preset width, as well as the resolution height of the decoded video frame and the second preset height, adopts a more appropriate image processing method, effectively balances image processing efficiency and image processing quality, and improves the user's video call experience.

[0081] In one embodiment, when the decoded video frame does not meet the preset post-processing conditions, the decoded video frame may not be processed, and the decoded video frame may be directly rendered and played, so as to ensure timely playback of the decoded video frame. In one embodiment, when the decoded video frame meets the preset post-processing conditions, the decoded video frame is subjected to image post-processing based on a preset image post-processing algorithm (e.g., super-resolution processing based on a preset image super-resolution algorithm, super-resolution processing based on a neural network, etc.), and the decoded video frame after image post-processing is rendered and played. This solution improves the image quality of video calls and improves the user's video call experience by performing image post-processing on the decoded video frame based on a preset image post-processing algorithm when the decoded video frame meets the preset post-processing conditions.

[0082] In the above, by determining the detection bandwidth for real-time video calls with the second terminal, the video playback frame rate of the second terminal, and the video packet loss rate of the first terminal, when the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased to obtain the target code table, the encoding resolution is determined according to the detection bandwidth and the target code table, and the video encoding of the image to be encoded of the video call is performed based on the encoding resolution to obtain the encoded video information, and in the scenario where the network condition is general or the network fluctuation is relatively large, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased, so that the image encoding is easier to trigger the image encoding resolution down-cut, and it is more difficult to trigger the image encoding resolution up-cut, and it is easier to trigger the video super-resolution processing at the second terminal, improve the call quality, and enhance the user experience. At the same time, by increasing the minimum bit rate of the preset gear in the preset code table according to the preset minimum bit rate adjustment parameter, and increasing the maximum bit rate of the preset gear in the preset code table according to the preset maximum bit rate adjustment parameter, the preset code table can be quickly and accurately adjusted to obtain the target code table, ensure the call quality, and enhance the user experience.

[0083] Figure 3 is a structural diagram of an image encoding device for video calls provided by an embodiment of the present application. Figure 3 The image encoding device for video calls can be applied to the first terminal. The image encoding device for video calls includes a bandwidth detection module 31, an information acquisition module 32, a code table adjustment module 33 and a video encoding module 34.

[0084] Wherein, the bandwidth detection module 31 is configured to determine the detection bandwidth for conducting a real-time video call with the second terminal;

[0085] An information acquisition module 32, configured to determine a video playback frame rate of the second terminal and a video packet loss rate of the first terminal;

[0086] The code table adjustment module 33 is configured to increase the minimum bit rate and the maximum bit rate of the preset gear in the preset code table to obtain the target code table when the video playback frame rate and the video packet loss rate meet the code table adjustment conditions;

[0087] The video encoding module 34 is configured to determine the encoding resolution according to the detected bandwidth and the target code table, and perform video encoding on the to-be-encoded image of the video call based on the encoding resolution to obtain encoded video information.

[0088] In the above, by determining the detection bandwidth for conducting a real-time video call with the second terminal, the video playback frame rate of the second terminal and the video packet loss rate of the first terminal, when the video playback frame rate and the video packet loss rate meet the code table adjustment conditions, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased to obtain the target code table, the encoding resolution is determined according to the detection bandwidth and the target code table, and the video encoding is performed on the image to be encoded of the video call based on the encoding resolution to obtain the encoded video information. In scenarios where the network condition is general or the network fluctuation is relatively large, the minimum bit rate and the maximum bit rate of the preset gear in the preset code table are increased, so that image encoding is easier to trigger a downward cut of the image encoding resolution, and more difficult to trigger an upward cut of the image encoding resolution, and it is easier to trigger video super-resolution processing at the second terminal, thereby improving call quality and user experience.

[0089] In a possible embodiment, the video playback frame rate and the video packet loss rate meet the code table adjustment condition, which may be that the video playback frame rate is less than a preset frame rate threshold, and the video packet loss rate is greater than a preset packet loss threshold.

[0090] In a possible embodiment, the code table adjustment module 33 increases the minimum code rate and the maximum code rate of the preset gear in the preset code table, and is configured as follows:

[0091] Increase the minimum bit rate of the preset gear in the preset code table according to the preset minimum bit rate adjustment parameter;

[0092] According to the preset maximum bit rate adjustment parameter, the maximum bit rate of the preset gear in the preset code table is increased.

[0093] In a possible embodiment, the video encoding module 34 performs video encoding on the to-be-encoded image of the video call based on the encoding resolution to obtain the encoded video information, and is configured as follows:

[0094] When the encoding resolution meets the processing computing power condition, performing first downsampling processing and image pre-processing on the image to be encoded of the video call based on the first downsampling processing algorithm and the image pre-processing algorithm, and performing video encoding on the image to be encoded after the first downsampling processing and the image pre-processing to obtain encoded video information;

[0095] When the encoding resolution does not meet the processing computing power condition, a second downsampling process is performed on the image to be encoded of the video call based on a second downsampling processing algorithm, and video encoding is performed on the image to be encoded after the second downsampling process to obtain encoded video information.

[0096] In a possible embodiment, the encoding resolution satisfies the processing power condition, which may be that the resolution width of the encoding resolution is less than or equal to the first preset width, and the resolution height of the encoding resolution is less than or equal to the first preset height.

[0097] In a possible embodiment, the image encoding device for video calls also includes a data sending module, which is configured to send encoded video information to a second terminal, so that the second terminal can decode the encoded video information to obtain a decoded video frame, and when the decoded video frame meets a preset post-processing condition, perform image post-processing on the decoded video frame based on a preset image post-processing algorithm, and play the decoded video frame after image post-processing.

[0098] In a possible embodiment, the decoded video frame satisfies a preset post-processing condition, which may be that the resolution width of the decoded video frame is less than or equal to a second preset width, and the resolution height of the decoded video frame is less than or equal to a second preset height.

[0099] It is worth noting that in the embodiment of the image encoding device for the above-mentioned video call, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0100] The embodiment of the present application also provides an image encoding device for video calls, and the image encoding device for video calls can be integrated with the image encoding apparatus for video calls provided in the embodiment of the present application. Figure 4 is a structural diagram of an image encoding device for video calls provided in an embodiment of the present application. Figure 4 The video call image encoding device includes: an input device 43, an output device 44, a memory 42 and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the video call image encoding method provided by the above embodiment. The above-mentioned video call image encoding device, equipment and computer can be used to execute the video call image encoding method provided by any of the above embodiments, and have corresponding functions and beneficial effects.

[0101] The embodiment of the present application also provides a non-volatile storage medium storing computer executable instructions, which are used to execute the image encoding method for video calls provided in the above embodiment when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the image encoding method for video calls provided above, can also execute the related operations in the image encoding method for video calls provided in any embodiment of the present application. The image encoding apparatus, device and storage medium for video calls provided in the above embodiments can execute the image encoding method for video calls provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the image encoding method for video calls provided in any embodiment of the present application.

[0102] Based on the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the image encoding method for video calls provided in each embodiment of the present application.

Claims

1. A method for encoding an image for a video call, characterized in that: Applied to the first terminal, comprising: Determining a detection bandwidth for conducting a real-time video call with a second terminal; Determine a video playback frame rate of the second terminal and a video packet loss rate of the first terminal; When the video playback frame rate and the video packet loss rate meet the code table adjustment condition, increasing the minimum bit rate and the maximum bit rate of the preset gear in the preset code table to obtain the target code table; The coding resolution is determined according to the detected bandwidth and the target code table, and video coding is performed on the to-be-coded image of the video call based on the coding resolution to obtain coded video information.

2. The image encoding method for video calls according to claim 1, characterized in that: The video playback frame rate and the video packet loss rate satisfy the code table adjustment conditions, including: The video playback frame rate is less than a preset frame rate threshold, and the video packet loss rate is greater than a preset packet loss threshold.

3. The image encoding method for video calls according to claim 1, characterized in that: The step of increasing the minimum bit rate and the maximum bit rate of the preset gear in the preset code table includes: Increase the minimum bit rate of the preset gear in the preset code table according to the preset minimum bit rate adjustment parameter; According to the preset maximum bit rate adjustment parameter, the maximum bit rate of the preset gear in the preset code table is increased.

4. The image encoding method for video calls according to claim 1, characterized in that: The step of performing video encoding on the to-be-encoded image of the video call based on the encoding resolution to obtain the encoded video information includes: When the encoding resolution meets the processing computing power condition, performing first downsampling processing and image pre-processing on the image to be encoded of the video call based on the first downsampling processing algorithm and the image pre-processing algorithm, and performing video encoding on the image to be encoded after the first downsampling processing and the image pre-processing to obtain encoded video information; When the encoding resolution does not meet the processing computing power condition, a second downsampling process is performed on the image to be encoded of the video call based on a second downsampling processing algorithm, and video encoding is performed on the image to be encoded after the second downsampling process to obtain encoded video information.

5. The image encoding method for video calls according to claim 4, characterized in that: The encoding resolution meets the processing power conditions, including: The resolution width of the encoding resolution is less than or equal to a first preset width, and the resolution height of the encoding resolution is less than or equal to a first preset height.

6. The image encoding method for video calls according to claim 1, characterized in that: After performing video encoding on the to-be-encoded image of the video call based on the encoding resolution to obtain the encoded video information, the method further includes: The encoded video information is sent to the second terminal, so that the second terminal decodes the encoded video information to obtain a decoded video frame, and when the decoded video frame meets a preset post-processing condition, performs image post-processing on the decoded video frame based on a preset image post-processing algorithm, and plays the decoded video frame after image post-processing.

7. The image encoding method for video calls according to claim 6, characterized in that: The decoded video frame meets a preset post-processing condition, including: The resolution width of the decoded video frame is less than or equal to a second preset width, and the resolution height of the decoded video frame is less than or equal to a second preset height.

8. An image encoding device for video calls, applied to a first terminal, characterized in that: It includes a bandwidth detection module, an information acquisition module, a code table adjustment module and a video encoding module, wherein: The bandwidth detection module is configured to determine a detection bandwidth for conducting a real-time video call with a second terminal; The information acquisition module is configured to determine a video playback frame rate of the second terminal and a video packet loss rate of the first terminal; The code table adjustment module is configured to increase the minimum bit rate and the maximum bit rate of the preset gear in the preset code table to obtain the target code table when the video playback frame rate and the video packet loss rate meet the code table adjustment condition; The video encoding module is configured to determine the encoding resolution according to the detection bandwidth and the target code table, and perform video encoding on the to-be-encoded image of the video call based on the encoding resolution to obtain encoded video information.

9. An image encoding device for video calls, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image encoding method for video calls as described in any one of claims 1 to 7.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to execute the image encoding method for video calls as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image encoding method for video calls according to any one of claims 1 to 7 is implemented.

Images