Method and device for optimizing image quality
By detecting the network environment and adjusting the quantization parameters (QP) according to the image data scene and resolution, the problem of difficult image quality to optimize when the network environment changes is solved, and automatic adjustment and optimization of image quality in different network environments is achieved. Experience.
Patent Information
- Application Number
- CN202310970638.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-02
- Publication Date
- 2025-02-14
AI Technical Summary
In the case of unstable network environment, it is difficult to optimize the image quality in real time, resulting in users' experience of lag and poor image quality in a weak network environment, and it is impossible to improve the image resolution in time to improve the image quality when the network environment improves.
By detecting the current network environment, obtaining network environment parameters, and determining the recommended quantization parameters (QP) based on the correspondence between image data scene, resolution, code rate and network environment parameters, to adapt to the current network conditions and adjust image encoding and transmission .
It realizes automatic adjustment of image resolution and encoding quality in different network environments, ensuring that image quality remains optimal when changes in network environments and improving user experience.
Smart Images
Figure CN119450044A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular, to a method and device for optimizing image quality. Background Art
[0002] The transmitted image data will be affected by the network environment, resulting in poor image quality. Take home monitoring and security products such as PTZ surveillance cameras, battery cameras, and smart doorbells as an example.
[0003] The real-time image preview interface in a mobile client, such as an APP, will provide automatic resolution and manual resolution for users to choose from. The automatic resolution will be switched appropriately according to the current network conditions, while the manual resolution will be set according to the resolution required by the user.
[0004] When the network environment is poor, that is, in a weak network environment, regardless of manual or automatic resolution settings, preview lag often occurs due to the poor network environment. Furthermore, when the network environment improves, if the automatic resolution mode is selected, the resolution will not be increased because the network bandwidth cannot reach the bandwidth standard corresponding to the resolution, thus affecting the user's picture quality experience.
[0005] The general approach to the above phenomenon is that when interruptions or freezes occur, if the user selects automatic resolution mode, it will continue to switch to a lower resolution until it reaches the lowest definition resolution to ensure smooth live broadcast; when the network environment improves, it will continue to switch to a higher resolution until it reaches the highest resolution that meets the current playback bandwidth; if the user selects manual resolution, the set resolution will remain unchanged regardless of the network environment. In this way, the experience of interruptions and freezes can be ignored when the network environment is poor, and good image quality cannot be obtained when the network environment is good.
[0006] The above solution cannot guarantee that the best picture quality that satisfies the current network environment can be experienced when the network environment is poor, nor can it guarantee that the best picture quality can be experienced when the network environment improves. Summary of the invention
[0007] The present application proposes a method for optimizing image quality to improve image quality in various network environments.
[0008] The first aspect of the present application provides a method for optimizing image quality, the method comprising: at a sending end of image data,
[0009] Detect the current network environment and obtain network environment parameters used to characterize the quality of the current network environment.
[0010] Based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended quantization (QP) parameter, according to the image data scene to be transmitted, determine the recommended QP parameter corresponding to the current resolution of the image data to be transmitted under the current network environment parameters,
[0011] The image data to be transmitted is encoded according to the current resolution and the determined recommended QP parameter.
[0012] In an embodiment of the present application, the correspondence between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter includes:
[0013] The corresponding relationship between the bit rate, ideal bandwidth and recommended QP parameters for each image data scene and resolution.
[0014] in,
[0015] The image data scene includes at least one of the following: a captured daytime dynamic image scene, a daytime static image scene, a black-and-white night dynamic image scene, a black-and-white night static image scene, a color night dynamic image scene, a color night static image scene,
[0016] The ideal bandwidth is the sum of the bit rate and the set redundancy. The redundancy is determined according to the network congestion level. The more serious the network congestion, the greater the redundancy.
[0017] In an embodiment of the present application, after detecting the current network environment and obtaining network environment parameters used to characterize the quality of the current network environment, the method further includes:
[0018] Based on the control method of the sending end for network congestion, determine the current fitted real bandwidth;
[0019] The step of determining, according to the scene of the image data to be transmitted, the recommended QP parameter corresponding to the currently set resolution of the image data to be transmitted under the current network environment parameters includes:
[0020] In the manual resolution mode, an ideal bandwidth closest to the current fitted real bandwidth is determined, and a recommended QP parameter and bit rate corresponding to the ideal bandwidth are determined according to the determined ideal bandwidth.
[0021] The encoding of the image data to be transmitted according to the current resolution and the determined recommended QP parameter includes:
[0022] In the manual resolution mode, the set resolution is kept unchanged, the image data to be transmitted is encoded according to the set resolution and the determined recommended QP parameter, and the encoded image data to be transmitted is transmitted according to the determined bit rate.
[0023] In an embodiment of the present application, the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, the ideal bandwidth and the recommended QP parameter further includes:
[0024] The corresponding relationship between the bit rate, ideal bandwidth, recommended QP parameters, and recommended resolution for each image data scene and resolution;
[0025] The step of determining, according to the scene of the image data to be transmitted, the recommended QP parameter corresponding to the currently set resolution of the image data to be transmitted under the current network environment parameters includes:
[0026] In the automatic resolution mode, determining an ideal bandwidth closest to the current fitted actual bandwidth, and determining a recommended QP parameter, a recommended resolution, and a bit rate corresponding to the ideal bandwidth according to the determined ideal bandwidth;
[0027] The encoding of the image data to be transmitted according to the current resolution and the determined recommended QP parameter includes:
[0028] In the automatic resolution mode, the image data to be transmitted is encoded according to the determined recommended QP parameter and the recommended resolution, and the encoded image data to be transmitted is transmitted according to the determined bit rate.
[0029] In an embodiment of the present application, detecting the current network environment and obtaining network environment parameters used to characterize the quality of the current network environment further includes:
[0030] According to the obtained network environment parameters, determine whether there is network congestion.
[0031] If yes, determine the network congestion level and current fitted real bandwidth,
[0032] Otherwise, determine whether the current fitted real network bandwidth increases to the set bandwidth threshold; if so, execute the step of determining the recommended QP parameter corresponding to the current set resolution of the image data to be transmitted under the current network environment parameters based on each image data scene, the bit rate corresponding to each resolution, the correspondence between each network environment parameter and the recommended QP parameter according to the image data scene to be transmitted; otherwise, keep the current resolution and QP parameter unchanged.
[0033] In the embodiment of the present application, the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and QP parameter, and the recommended resolution is obtained as follows:
[0034] For each image data scene,
[0035] Get the QP parameters and bit rates corresponding to different resolutions in the image data scenario.
[0036] For each bit rate corresponding to the image data scene, determine the ideal bandwidth corresponding to the bit rate under different network congestion levels, and establish a corresponding relationship between each ideal bandwidth and a recommended QP parameter and a recommended resolution.
[0037] In an embodiment of the present application, the step of obtaining QP parameters and bit rates corresponding to different resolutions in the image data scenario includes:
[0038] Inputting sample image data having the image data scene into the first neural network model to be trained, wherein the sample image data includes different sample resolutions,
[0039] According to the loss function between the sample QP parameter and the sample bit rate output by the first neural network model to be trained and the expected QP parameter and the expected bit rate, the model parameters of the first neural network model to be trained are adjusted until the expected results are achieved, thereby obtaining the trained first neural network model.
[0040] The collected image data of the image data scene with any resolution is input into the trained first neural network model to obtain the bit rate and QP parameters corresponding to the resolution.
[0041] In an embodiment of the present application, for each bit rate corresponding to the image data scene, determining the ideal bandwidth corresponding to the bit rate under different network congestion levels, and establishing a corresponding relationship between each ideal bandwidth and the recommended QP parameter and the recommended resolution include:
[0042] The sample data is input into the second neural network model to be trained to output the sample ideal bandwidth, the sample recommended resolution, and the sample recommended QP parameter, wherein the sample data includes: sample image data having the image data scene, sample bit rate data, and sample network congestion level data,
[0043] According to the loss function between the sample ideal bandwidth, sample recommended resolution, and sample recommended QP parameter output by the second neural network model to be trained and the expected ideal bandwidth, expected recommended resolution, and expected QP parameter, the model parameters of the second neural network model to be trained are adjusted until the expectations are met, thereby obtaining a trained second neural network model.
[0044] The collected image data in the image data scenario, any network congestion level data, and any bit rate data are input into the trained second neural network model to obtain the ideal bandwidth, recommended QP parameters, and recommended resolution corresponding to the bit rate under the network congestion level.
[0045] In an embodiment of the present application, the determining of the current fitted real bandwidth based on the control mode of the transmitting end for the network congestion includes:
[0046] For the cubic network congestion control method, the fitted real bandwidth is the product of the maximum segment length and the send window size divided by the round-trip delay of the data packet; or
[0047] For the bottleneck bandwidth and round-trip propagation time (BBR) network congestion control method, the fitted real bandwidth is the product of the gain of the control message sending rate and the bottleneck bandwidth.
[0048] The fitted real bandwidth within the set time period is subjected to at least one of truncating, weight filtering, and smoothing to obtain a current fitted real bandwidth.
[0049] A second aspect of the present application provides an image quality optimization device, the device comprising:
[0050] The network detection module is used to detect the current network environment and obtain network environment parameters used to characterize the quality of the current network environment.
[0051] The adaptive modulation module is used to determine the recommended QP parameter corresponding to the current set resolution of the image data to be transmitted under the current network environment parameters according to the image data scene to be transmitted, based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter,
[0052] The encoding module is used to encode the image data to be transmitted according to the current resolution and the determined recommended QP parameter.
[0053] The method for optimizing image quality provided in the embodiment of the present application determines the recommended QP parameters corresponding to the current resolution of the image data to be transmitted under the current network environment parameters by pre-establishing the correspondence between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter, so that the QP parameters under the same resolution can be adaptively adjusted according to the current network environment, which is beneficial to improving the image quality. Furthermore, the bit rate can also be adjusted according to the current network environment, thereby optimizing both the smoothness of the image when it is played and the image quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A flowchart of the image optimization method according to an embodiment of the present application is shown.
[0055] Figure 2 A flowchart of an image quality optimization method according to an embodiment of the present application is shown.
[0056] Figure 3A flowchart of the image quality optimization method according to the second embodiment of the present application.
[0057] Figure 4 A schematic diagram of an image optimization device according to an embodiment of the present application.
[0058] Figure 5 Another schematic diagram of the image optimization device according to an embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.
[0060] If the traditional solution is frozen at a certain resolution when sending a video stream, the image quality will no longer be adjusted, and the bit rate between each resolution and the granularity between the required bandwidth are large. The embodiment of the present application further subdivides according to the quantization (QP) parameters at each resolution in advance, and adaptively adjusts the QP parameters according to the current network environment parameters, so that the current network bandwidth can be fully utilized, ensuring smooth video playback while showing better picture quality. Especially in a weak network environment, when the phenomenon of freezing or interruption may occur even after cutting to the lowest resolution, the embodiment of the present application can set a variety of QP parameters in the lowest resolution, and its bit rate and required bandwidth also change with the QP granularity, counteracting resolution changes and QP parameter changes according to the weak network conditions, and ultimately improving the user experience.
[0061] See also Figure 1 As shown, Figure 1 This is a flow chart of an image optimization method according to an embodiment of the present application. The method comprises: at the sending end of the image data,
[0062] Step 101, detecting the current network environment and obtaining network environment parameters used to characterize the quality of the current network environment.
[0063] As an example, the network environment parameters used to characterize the quality of the current network environment include: at least one of: network delay, network bandwidth, packet loss rate, and packet error rate.
[0064] It should be understood that the sender of image data includes but is not limited to monitoring equipment that can send image data, terminal equipment that can upload video streams to the network, servers and clients that can transmit video streams to terminal equipment, etc.
[0065] Step 102, based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter, according to the image data scene to be transmitted, determine the recommended QP parameter corresponding to the current resolution of the image data to be transmitted under the current network environment parameters,
[0066] in,
[0067] The QP parameter is used to reflect the compression of spatial details. The smaller the QP, the richer the image details and the higher the bit rate. The larger the QP, the more image details are lost, the lower the bit rate, the stronger the image distortion and the lower the quality.
[0068] The bit rate is used to represent the number of bits transmitted per unit time during data transmission. The higher the bit rate, the more data is transmitted per second, and the clearer the image quality.
[0069] Resolution refers to the amount of information in an image, described by the number of pixels per inch of the image. For example, resolution usually includes smooth, standard, high, and ultra-clear.
[0070] The correspondence between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter can be established in advance.
[0071] As an example, for monitoring equipment, the image data scene is the collected daytime dynamic image scene, daytime static image scene, black and white night dynamic image scene, black and white night static image scene, color night dynamic image scene, color night static image scene. It should be understood that the image data scene can be set according to the application scenario of the image data, the application scenario of the image sending end, the image characteristics, the image format, etc.
[0072] As an example, the network environment parameter in the corresponding relationship is an ideal bandwidth, and the ideal bandwidth is the sum of the bit rate and the set redundancy. The redundancy is determined according to the network congestion level. The more serious the network congestion, the greater the redundancy.
[0073] Step 103: Encode the image data to be transmitted according to the current resolution and the determined recommended QP parameter.
[0074] The embodiment of the present application determines the recommended QP parameters corresponding to the current resolution of the image data to be transmitted under the current network environment parameters through the correspondence between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter, so that the resolution and QP parameters can be adaptively adjusted according to the current network environment, so that the current network bandwidth can be fully utilized, ensuring smooth video playback while showing better picture quality.
[0075] To facilitate the understanding of the present application, the following explanation is given using a home monitoring and security product as an example. It should be understood that the present application is not limited to monitoring and security products, and any other terminal, client, or server for transmitting image data may be applicable, for example, network video data.
[0076] See also Figure 2 As shown, Figure 2 This is a flow chart of an image quality optimization method according to an embodiment of the present application. The method comprises: on the monitoring device side,
[0077] Step 201, detecting the current network environment and obtaining network environment parameters used to characterize the quality of the current network environment.
[0078] As an example, in this step, network parameters such as network bandwidth, network delay, packet loss rate, and packet error rate of the image transmission link are obtained as network environment parameters.
[0079] As an example, the network parameters of the streaming link in the current and historical time periods are obtained and recorded, and the network parameters are input into a trained third neural network model for determining the network environment level to obtain the current network environment level, wherein the network environment level may include the network congestion level, and the third neural network model may adopt an existing classifier model.
[0080] As another example, the network congestion level is calculated based on the network delay and packet loss rate.
[0081] As an example, the network congestion level can be divided into three levels, 0 represents a good network condition with no congestion, 1 represents a slightly congested network condition, and 2 represents a severely congested network condition.
[0082] Step 202, obtaining the current fitting real bandwidth.
[0083] Since monitoring devices are generally developed based on the Linux system, and different monitoring devices have different network congestion algorithms configured in the Linux kernel, this embodiment matches different bandwidth calculation formulas with the network congestion algorithms configured in different Linux kernels to fit the actual bandwidth requirements. For example,
[0084] For the cubic network congestion control algorithm, the fitted real bandwidth is the product of the maximum segment length and the send window size divided by the round-trip delay of the data packet, which can be expressed mathematically as follows:
[0085] send_bps=tcpi_snd_cwnd*tcpi_snd_mss / rtt;
[0086] Among them, send_bps is the actual bandwidth, tcpi_snd_cwnd is the send window size, tcpi_snd_mss is the maximum segment length, rtt is the round-trip delay of the data packet,
[0087] For the BBR (Bottleneck Bandwidth and Round-trip propagation time) network congestion control algorithm, the fitted real bandwidth is the product of the gain of the control message sending rate and the bottleneck bandwidth, which can be expressed mathematically as follows:
[0088] send_bps=pacing_gain*bottleneck_bandwidth
[0089] Among them, pacing_gain is the gain of controlling the sending rate of messages, and bottleneck_bandwidth is the bottleneck bandwidth.
[0090] In this way, the cloud or monitoring device sends a specific data stream, obtains the corresponding network congestion control algorithm according to the Linux system call, obtains the parameters required to calculate the bandwidth from the network protocol stack through netlink, and calculates according to the matching bandwidth calculation formula to obtain the fitted real bandwidth at the current moment.
[0091] The real bandwidth fitted within the set time period is stored in a first-in-first-out queue for subsequent data processing, which includes truncating, weight filtering, and smoothing. Through data processing, the bandwidth jumps with large fluctuations in weak networks can be eliminated, the fluctuations against the trend in weak networks can be smoothed, and a more accurate real bandwidth can be fitted through specific weight parameters.
[0092] The purpose of fitting the actual bandwidth is to avoid frequent switching of resolution due to changes in bandwidth, which may cause oscillation.
[0093] Step 203, according to the fitted actual bandwidth, combined with different image data scenarios and current resolutions, based on the corresponding relationship between each image data scenario, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter, determine the recommended QP parameter, and further, determine the bit rate under the current network environment.
[0094] The following describes the process of obtaining the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter.
[0095] Step 1: Through a large amount of experimental data, the QP and bit rate corresponding to each resolution in different image data scenes can be tested. The scenes include daytime, black and white at night, and full color at night. The above three scenes are further divided into dynamic pictures and static pictures. Finally, the QP parameters and bit rates corresponding to each resolution in the scene are counted. See Table 1, which is an example of QP parameters and bit rates corresponding to different classifications in different scenes.
[0096] Table 1
[0097]
[0098] As an example, the QP and bit rate corresponding to each resolution of each image data scene in Table 1 above can be obtained as follows:
[0099] For any image data scene, a first neural network model to be trained is constructed for the image data scene, for example, a neural network model to be trained corresponding to a daytime dynamic picture is constructed.
[0100] The first neural network model to be trained is trained using sample image data corresponding to the scene, wherein the sample image data includes different sample resolutions. For example, sample image data with daytime dynamic pictures are input into the neural network model corresponding to the daytime dynamic pictures, and the sample image data includes resolutions 1, 2, ..., and resolution n.
[0101] According to the loss function between the sample QP parameter and sample bit rate output by the first neural network model to be trained and the expected QP parameter and bit rate, the model parameters of the first neural network model to be trained are adjusted until the expected results are achieved, thereby obtaining the trained first neural network model.
[0102] The image data of the image data scene with any resolution collected by the monitoring device is input into the trained first neural network model. For example, the collected daytime dynamic images and the resolution manually set by the user are input into the trained neural network model to obtain the corresponding bit rate and QP parameters at the resolution.
[0103] Step 2: Add a redundancy to each bitrate obtained in step 1 for each image data scenario to obtain the ideal bandwidth, where the redundancy is determined by the network congestion level. The more severe the network congestion, the greater the redundancy. Establish a corresponding relationship between the ideal bandwidth and recommended parameters corresponding to each network congestion level at each bitrate of each image data scenario. The recommended parameters include at least QP parameters and may also include recommended resolution. Table 2 shows the corresponding relationship between the ideal bandwidth, recommended resolution, and recommended OP parameters corresponding to each bitrate at different congestion levels in a daytime dynamic scene.
[0104] Table 2
[0105]
[0106] As an example, the correspondence between the ideal bandwidth corresponding to each bit rate and the recommended resolution and recommended QP parameter under different congestion levels in each image data scenario can be obtained as follows:
[0107] For any image data scene, a second neural network model to be trained is constructed for the image data scene. For example, a neural network model to be trained corresponding to a daytime dynamic picture is constructed.
[0108] The second neural network model to be trained is trained using the sample data, that is, the sample data is input into the neural network model to be trained, and the sample ideal bandwidth, the sample recommended resolution, and the sample recommended QP parameter are output, wherein the sample data includes: sample image data having the image data scene, sample bit rate data, and sample network congestion level data,
[0109] According to the loss function between the sample ideal bandwidth, sample recommended resolution, sample QP parameter output by the second neural network model to be trained and the expected ideal bandwidth, expected recommended resolution, and expected QP parameter, the model parameters of the second neural network model to be trained are adjusted until the expectations are met, thereby obtaining the trained second neural network model.
[0110] The image data of the scene captured by the monitoring equipment, as well as any network congestion level and any bit rate are input into the trained second neural network model. For example, the captured daytime dynamic images, bit rate and network congestion level are input into the trained neural network model to obtain the recommended resolution and recommended QP parameters corresponding to the bit rate and the network congestion level.
[0111] In automatic resolution mode, the image data scene corresponding to the image data to be transmitted is determined according to the image data to be transmitted, and Table 2 is queried according to the determined image data scene and the fitted actual bandwidth to find the ideal bandwidth in Table 2 that is closest to the fitted actual bandwidth of the determined image data scene, and the corresponding recommended resolution and recommended QP value are obtained.
[0112] When the user selects the manual resolution mode, it is also matched according to the fitted actual bandwidth to find the recommended QP parameter corresponding to the ideal bandwidth closest to the fitted actual bandwidth, keeping the resolution unchanged. In this way, while ensuring that the resolution remains unchanged, the QP parameter is adjusted according to the network congestion level and the ideal bandwidth to achieve the best image quality under the current network bandwidth.
[0113] Step 204, in the automatic resolution mode, the current image to be transmitted is encoded according to the determined recommended QP parameters and recommended resolution, and the encoded image data is transmitted at the determined bit rate; in the manual resolution mode, the resolution is kept unchanged, the current image to be transmitted is encoded according to the determined recommended QP parameters and manual resolution, and the encoded image data is transmitted at the determined bit rate.
[0114] Since different QP parameters at different resolutions bring different image quality effects, in automatic resolution mode, users in a weak network environment will get the best image quality at the current resolution while providing smooth preview. When the network environment recovers from a weak network, users can also experience the best image quality by switching to a higher resolution. In manual resolution mode, users can experience the best image quality that meets the bandwidth requirements at a fixed current resolution.
[0115] This embodiment expands each resolution into QP parameters of various granularities, and automatically adjusts the QP parameters according to the network environment conditions, so as to obtain the best image quality under the premise of smooth and stable video playback.
[0116] See also Figure 3 As shown, Figure 3 A flow chart of a method for optimizing image quality according to a second embodiment of the present application. The method comprises: on the monitoring device side,
[0117] Step 301, detecting the current network environment and obtaining network environment parameters used to characterize the quality of the current network environment, this step may be the same as step 201;
[0118] Step 302: determine whether there is network congestion based on network environment parameters.
[0119] If yes, then determine the current network congestion level and the current fitted real bandwidth, and then execute step 303,
[0120] Otherwise, it is determined whether the current fitting real bandwidth increases to the set bandwidth threshold. If so, it means that the current fitting real bandwidth has been significantly improved, and step 303 is executed. Otherwise, the current resolution and QP parameters are kept unchanged.
[0121] Step 303, based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter, according to fitting the actual bandwidth, and in combination with different image data scenes and the current resolution, determine the recommended QP parameter, and further, determine the bit rate under the current network environment.
[0122] This step is the same as step 203.
[0123] Step 304: determine whether the recommended resolution, recommended QP parameter, and bit rate determined in step 303 are the same as the current resolution, current QP parameter, and current bit rate.
[0124] If yes, then no encoding or bit rate adjustment is performed; otherwise, encoding and bit rate adjustment are performed according to the determined recommended resolution, recommended QP parameter, and bit rate.
[0125] See also Figure 4 As shown, Figure 4This is a schematic diagram of an image optimization device according to an embodiment of the present application, the device comprising:
[0126] The network detection module is used to detect the current network environment and obtain network environment parameters used to characterize the quality of the current network environment.
[0127] The adaptive adjustment module is used to determine the recommended QP parameter corresponding to the current set resolution of the image data to be transmitted under the current network environment parameters according to the image data scene to be transmitted, based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter,
[0128] The encoding module is used to encode the image data to be transmitted according to the current resolution and the determined recommended QP parameter.
[0129] The device also includes:
[0130] The network bandwidth fitting module is used to determine the current fitting real bandwidth based on the control method of the sending end for network congestion,
[0131] The sending module is used to send the encoded image data to be transmitted at the code rate determined by the adaptive adjustment module.
[0132] See also Figure 5 As shown, Figure 5 This is another schematic diagram of an image optimization device according to an embodiment of the present application. The device includes: a memory and a processor. The memory stores a computer program. The processor is configured to execute the computer program to implement the steps of the image optimization method.
[0133] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0134] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0135] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the image optimization method described in the embodiment of the present application are implemented.
[0136] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0137] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0138] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for optimizing image quality, characterized in that: The method comprises: at a sending end of image data, Detect the current network environment and obtain network environment parameters used to characterize the quality of the current network environment. Based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended quantization QP parameter, according to the image data scene to be transmitted, determine the recommended QP parameter corresponding to the current resolution of the image data to be transmitted under the current network environment parameters, The image data to be transmitted is encoded according to the current resolution and the determined recommended QP parameter.
2. The optimization method according to claim 1, characterized in that: The corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter includes: The corresponding relationship between the bit rate, ideal bandwidth and recommended QP parameters for each image data scene and resolution. in, The image data scene includes at least one of the following: a captured daytime dynamic image scene, a daytime static image scene, a black-and-white night dynamic image scene, a black-and-white night static image scene, a color night dynamic image scene, a color night static image scene, The ideal bandwidth is the sum of the bit rate and the set redundancy. The redundancy is determined according to the network congestion level. The more serious the network congestion, the greater the redundancy.
3. The optimization method according to claim 2, characterized in that: After detecting the current network environment and obtaining network environment parameters used to characterize the quality of the current network environment, the method further includes: Based on the control method of the sending end for network congestion, determine the current fitted real bandwidth; The step of determining, according to the scene of the image data to be transmitted, the recommended QP parameter corresponding to the currently set resolution of the image data to be transmitted under the current network environment parameters includes: In the manual resolution mode, an ideal bandwidth closest to the current fitted real bandwidth is determined, and a recommended QP parameter and bit rate corresponding to the ideal bandwidth are determined according to the determined ideal bandwidth. The encoding of the image data to be transmitted according to the current resolution and the determined recommended QP parameter includes: In the manual resolution mode, the set resolution is kept unchanged, the image data to be transmitted is encoded according to the set resolution and the determined recommended QP parameter, and the encoded image data to be transmitted is transmitted according to the determined bit rate.
4. The optimization method according to claim 2, characterized in that: The corresponding relationship between the bit rate corresponding to each image data scene and each resolution, the ideal bandwidth and the recommended QP parameter further includes: The corresponding relationship between the bit rate, ideal bandwidth, recommended QP parameters, and recommended resolution for each image data scene and resolution; The step of determining, according to the scene of the image data to be transmitted, the recommended QP parameter corresponding to the currently set resolution of the image data to be transmitted under the current network environment parameters includes: In the automatic resolution mode, determining an ideal bandwidth closest to the current fitted actual bandwidth, and determining a recommended QP parameter, a recommended resolution, and a bit rate corresponding to the ideal bandwidth according to the determined ideal bandwidth; The encoding of the image data to be transmitted according to the current resolution and the determined recommended QP parameter includes: In the automatic resolution mode, the image data to be transmitted is encoded according to the determined recommended QP parameter and the recommended resolution, and the encoded image data to be transmitted is transmitted according to the determined bit rate.
5. The optimization method according to claim 2, characterized in that: The detecting the current network environment and obtaining the network environment parameters used to characterize the quality of the current network environment further includes: According to the obtained network environment parameters, determine whether there is network congestion. If yes, determine the network congestion level and current fitted real bandwidth, Otherwise, determine whether the current fitted real network bandwidth increases to the set bandwidth threshold; if so, execute the step of determining the recommended QP parameter corresponding to the current set resolution of the image data to be transmitted under the current network environment parameters based on each image data scene, the bit rate corresponding to each resolution, the correspondence between each network environment parameter and the recommended QP parameter according to the image data scene to be transmitted; otherwise, keep the current resolution and QP parameter unchanged.
6. The optimization method according to claim 4, characterized in that: The corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and QP parameter, and the recommended resolution is obtained as follows: For each image data scene, Get the QP parameters and bit rates corresponding to different resolutions in the image data scenario. For each bit rate corresponding to the image data scene, determine the ideal bandwidth corresponding to the bit rate under different network congestion levels, and establish a corresponding relationship between each ideal bandwidth and a recommended QP parameter and a recommended resolution.
7. The optimization method according to claim 6, characterized in that: The obtaining of QP parameters and bit rates corresponding to different resolutions in the image data scenario includes: Inputting sample image data having the image data scene into the first neural network model to be trained, wherein the sample image data includes different sample resolutions, According to the loss function between the sample QP parameter and the sample bit rate output by the first neural network model to be trained and the expected QP parameter and the expected bit rate, the model parameters of the first neural network model to be trained are adjusted until the expected results are achieved, thereby obtaining the trained first neural network model. The collected image data of the image data scene with any resolution is input into the trained first neural network model to obtain the bit rate and QP parameters corresponding to the resolution.
8. The optimization method according to claim 6, characterized in that: The method of determining, for each bit rate corresponding to the image data scene, an ideal bandwidth corresponding to the bit rate under different network congestion levels, and establishing a corresponding relationship between each ideal bandwidth and a recommended QP parameter and a recommended resolution includes: The sample data is input into the second neural network model to be trained to output the sample ideal bandwidth, the sample recommended resolution, and the sample recommended QP parameter, wherein the sample data includes: sample image data having the image data scene, sample bit rate data, and sample network congestion level data, According to the loss function between the sample ideal bandwidth, sample recommended resolution, and sample recommended QP parameter output by the second neural network model to be trained and the expected ideal bandwidth, expected recommended resolution, and expected QP parameter, the model parameters of the second neural network model to be trained are adjusted until the expectations are met, thereby obtaining a trained second neural network model. The collected image data in the image data scenario, any network congestion level data, and any bit rate data are input into the trained second neural network model to obtain the ideal bandwidth, recommended QP parameters, and recommended resolution corresponding to the bit rate under the network congestion level.
9. The optimization method according to claim 3, characterized in that: The determining of the current fitted real bandwidth based on the control method of the sending end for network congestion includes: For the cubic network congestion control method using a cubic curve fit, the fitted real bandwidth is the product of the maximum segment length and the send window size divided by the round-trip delay of the data packet; or For the bottleneck bandwidth and round-trip propagation time BBR network congestion control method, the fitted real bandwidth is the product of the gain of the control message sending rate and the bottleneck bandwidth. The fitted real bandwidth within the set time period is subjected to at least one of truncating, weight filtering, and smoothing to obtain a current fitted real bandwidth.
10. An image quality optimization device, characterized in that: The device includes: The network detection module is used to detect the current network environment and obtain network environment parameters used to characterize the quality of the current network environment. The adaptive modulation module is used to determine the recommended QP parameter corresponding to the current set resolution of the image data to be transmitted under the current network environment parameters according to the image data scene to be transmitted, based on the corresponding relationship between each image data scene, the bit rate corresponding to each resolution, each network environment parameter and the recommended QP parameter, The encoding module is used to encode the image data to be transmitted according to the current resolution and the determined recommended QP parameter.