Video data processing method and device, equipment and storage medium
By real-time monitoring and adjustment of quantization parameters and network bandwidth during the video encoding process, and dynamically adjusting the image processing and encoding reference relationship, the problem of video data encoding quality and stability in 4G ball machines and other devices is solved, and more efficient video transmission is achieved and user experience.
Patent Information
- Application Number
- CN202411525826.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-10-30
AI Technical Summary
In the prior art, in 4G ball machines and other equipment, due to unstable network signals and bandwidth, the code rate is forced to be reduced when encoding video data, resulting in significant loss of image effects, and the effect and quality of video data encoding cannot be guaranteed.
By monitoring the quantized parameter (QP) values and network bandwidth in the encoding process in real time, dynamically adjust the image signal processing ISP image parameters and encoding reference relationships, including adjusting the frame rate, resolution and encoding reference relationships, ensuring encoding quality and transmission stability.
It improves the stability and encoding quality of video transmission, reduces bandwidth usage and storage costs, and improves user experience.
Smart Images

Figure CN119052479A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video encoding technology, and in particular to a method, device, equipment and storage medium for processing video data. Background Art
[0002] With the progress of society and the improvement of people's safety awareness, the demand for security monitoring is growing. 4G ball cameras are widely used in security monitoring in the fields of security, transportation, education, and medical care due to their advantages such as wireless transmission, high-definition image quality, and intelligent analysis. When the background area changes greatly during the rotation / zooming of the ball camera, and the 4G signal and wireless WiFi signal are weak and unstable, the encoding of video data is particularly important.
[0003] In the prior art, the network bandwidth is usually estimated through a network protocol quality service library, and then the bit rate is adjusted to ensure the encoding effect.
[0004] However, the above method forcibly reduces the encoding bit rate according to the network conditions, resulting in obvious loss of the encoded image effect, and cannot guarantee the effect and quality of video data encoding. Summary of the invention
[0005] The present application provides a video data processing method, device, equipment and storage medium, which are used to solve the problem that the encoding bit rate is forcibly reduced according to the network conditions, resulting in obvious loss of the encoded image effect and the inability to guarantee the effect and quality of the video data encoding.
[0006] In a first aspect, the present application provides a method for processing video data, which is applied to a front-end device, and the method includes:
[0007] If the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, the image signal processing ISP image parameters are adjusted;
[0008] Processing the original video data acquired at the current moment based on the adjusted ISP image parameters to obtain YUV data corresponding to the original video;
[0009] In the process of encoding the YUV data based on a preset target bit rate, if the QP values of the encoded continuous n frames of data are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, the encoding reference relationship is adjusted to the first encoding reference relationship until the encoding is completed to obtain the code stream data, and the first encoding reference relationship includes: the first frame is encoded as an intra-frame coded I frame, an I frame is encoded after each first preset interval, and after each second preset interval, it is encoded as a forward predicted Pm frame with reference to the last I frame encoded previously, and between the first frame and the P1 frame, the Pm-1 frame and the Pm frame, it is encoded as a p frame with reference to the previous frame currently encoded and the last I frame encoded previously before the current encoding, where m represents the mth second preset interval and is an integer greater than 2, the Pm-1 represents the m-1th P frame, and the Pm represents the mth P frame.
[0010] In combination with the first aspect, in some embodiments, the method further includes:
[0011] When the motion state indicates that the front-end device is moving, the encoding reference relationship is adjusted to a second encoding reference relationship until the encoding is completed to obtain the code stream data, wherein the second encoding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each third preset interval, and a Pn frame is encoded with reference to the last I frame encoded previously after each fourth preset interval; between the first frame and the P1 frame, multiple bidirectionally predicted B frames are encoded with reference to the first frame and the P1 frame; between the Pn-1 frame and the Pn frame, multiple B frames are encoded with reference to the Pn-1 frame and the Pn frame; n represents the nth fourth preset interval, which is an integer greater than 2; the Pn-1 represents the n-1th P frame; and the Pn represents the nth P frame.
[0012] In combination with the first aspect, in some embodiments, the method further includes:
[0013] In the encoding process using the first encoding reference relationship, if the QP values of the encoded consecutive n frames of data are all greater than the QP threshold and it is detected that the current network bandwidth is less than the preset first bandwidth threshold, the first encoding reference relationship is adjusted to a third encoding reference relationship for encoding, and the third encoding reference relationship includes: only the first frame is encoded as an I frame, and after each fifth preset interval, it is encoded as a Pa frame with reference to the first frame, and between the first frame and the P1 frame, the Pa-1 frame and the Pa frame, the previous frame currently encoded and the first frame are encoded as p frames, where a represents the ath fifth preset interval and is an integer greater than 2, the Pa-1 represents the a-1th P frame, and the Pa represents the ath P frame.
[0014] In combination with the first aspect, in some embodiments, the method further includes:
[0015] During the encoding process using the third encoding reference relationship, the encoding bit rate is adjusted based on the motion ratio of the original video data, and the adjusted bit rate is used for encoding the next frame, wherein the motion ratio is used to indicate the rate of change of the original video data for the macroblock pixel value.
[0016] In combination with the first aspect, in some embodiments, before adjusting the encoding bit rate based on the motion ratio of the original video data, the method further includes:
[0017] Downsampling the resolution of the original video data to generate a Bitmap table;
[0018] For each macroblock corresponding to the downsampled original video data, if the change value of the pixel value of the macroblock is greater than a preset pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 1;
[0019] If the change value of the pixel value of the macroblock is less than the pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 0;
[0020] The proportion of 1 in the Bitmap table is determined as the motion proportion.
[0021] In combination with the first aspect, in some embodiments, if the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, adjusting the image signal processing ISP image parameters includes:
[0022] Based on the QP threshold, determining ISP target image parameters from the ISP image parameters, the ISP target image parameters including noise reduction parameters and sharpening parameters;
[0023] increasing the level of the noise reduction parameter;
[0024] Reduce the level of the sharpening parameter.
[0025] In combination with the first aspect, in some embodiments, before adjusting the image signal processing ISP image parameters, the method further includes:
[0026] When the network signal weakens and the network bandwidth cannot meet the data volume of the bitstream transmission, the target bitrate is adjusted based on the network bandwidth at the current moment.
[0027] In combination with the first aspect, in some embodiments, the method further includes:
[0028] If the QP values of the consecutive n frames of data encoded after adjusting the encoding reference relationship are all greater than the QP threshold, the encoding frame rate and resolution are reduced.
[0029] In combination with the first aspect, in some embodiments, the method further includes:
[0030] When the motion state indicates that the front-end device is stationary and it is detected that the current network bandwidth is greater than the first bandwidth threshold and less than the preset second bandwidth threshold, the encoding reference relationship is adjusted to a fourth encoding reference relationship for encoding, and the fourth encoding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each sixth preset interval, and between two I frames, the previous frame and the last I frame that has been encoded are referenced as reference frames and encoded as a p frame, and the first bandwidth threshold is less than the second bandwidth threshold.
[0031] In combination with the first aspect, in some embodiments, the method further includes:
[0032] During encoding using the second encoding reference relationship, if the QP values of the encoded n consecutive frames of data are all greater than the QP threshold and it is detected that the current network bandwidth cannot meet the data volume of the code stream transmission, the motion state of the front-end device is adjusted to still.
[0033] In a second aspect, the present application provides a video data processing device, comprising:
[0034] A first adjustment module, configured to adjust an image signal processing ISP image parameter if the QP values of the consecutive n frames of data encoded before the current moment are all greater than a preset QP threshold;
[0035] A processing module, used to process the original video data acquired at the current moment based on the adjusted ISP image parameters to obtain YUV data corresponding to the original video;
[0036] The second adjustment module is used for, in the process of encoding the YUV data based on the preset target bit rate, if the QP values of the encoded continuous n frames of data are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, adjusting the encoding reference relationship to the first encoding reference relationship until the encoding is completed to obtain the code stream data, the first encoding reference relationship comprising: the first frame is encoded as an intra-frame coded I frame, an I frame is encoded after each first preset interval, and after each second preset interval, it is encoded as a forward predicted Pm frame with reference to the last I frame encoded previously, and between the first frame and the P1 frame, the Pm-1 frame and the Pm frame, it is encoded as a p frame with reference to the previous frame currently encoded and the last I frame encoded previously before the current encoding, m represents the mth second preset interval, which is an integer greater than 2, the Pm-1 represents the m-1th P frame, and the Pm represents the mth P frame.
[0037] In a third aspect, the present application provides a front-end device, comprising: a processor, and a memory, a communication interface, and an image sensor communicatively connected to the processor;
[0038] The memory stores computer-executable instructions;
[0039] The processor executes the computer-executable instructions stored in the memory to implement the method for processing video data according to any one of the first aspects.
[0040] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the video data processing method described in any of the aforementioned aspects.
[0041] In a fifth aspect, the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method for processing video data described in any of the preceding aspects is implemented.
[0042] The video data processing method, device, equipment and storage medium provided by the present application, if the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, then the image signal processing ISP image parameters are adjusted, and the original video data obtained at the current moment is processed based on the adjusted ISP image parameters to obtain the YUV data corresponding to the original video. In the process of encoding the YUV data based on the preset target bit rate, if the QP values of the consecutive n frames of data obtained by encoding are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, the encoding reference relationship is adjusted to the first encoding reference relationship until the encoding is completed to obtain the bitstream data. The above method effectively responds to the problems of network signal weakening and insufficient bandwidth by dynamically adjusting the ISP image parameters and the encoding reference relationship, improves the stability of video transmission, encoding quality and user experience, and reduces bandwidth occupancy and storage costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0044] Figure 1 A schematic diagram of the application system structure of the video data processing method provided in the embodiment of the present application;
[0045] Figure 2 A flowchart of a first embodiment of a method for processing video data provided in an embodiment of the present application;
[0046] Figure 3 is a schematic diagram of a first coding reference relationship;
[0047] Figure 4 is a schematic diagram of a third coding reference relationship;
[0048] Figure 5 A schematic diagram of a third coding reference relationship for real-time bit rate adjustment;
[0049] Figure 6 A flowchart of a second embodiment of a method for processing video data provided in an embodiment of the present application;
[0050] Figure 7 is a schematic diagram of a second coding reference relationship;
[0051] Figure 8 A flowchart of a third embodiment of a method for processing video data provided in an embodiment of the present application;
[0052] Fig. 9 Schematic diagram of the effect of noise reduction parameters on bit rate;
[0053] Fig.10 Schematic diagram of the effect of sharpening parameters on bit rate;
[0054] Fig.11 This is a schematic diagram of texture complexity comparison;
[0055] Fig.12 This is a comparison diagram of noise reduction level 0 and noise reduction level 100;
[0056] Fig.13 This is a comparison diagram of sharpening level 0 and sharpening level 100;
[0057] Fig.14 A flowchart of a fourth embodiment of a method for processing video data provided in an embodiment of the present application;
[0058] Fig.15 is a schematic diagram of a fourth coding reference relationship;
[0059] Fig.16 A schematic diagram of a fourth coding reference relationship for adjustment;
[0060] Fig.17 A schematic diagram of the structure of an embodiment of a video data processing device provided in an embodiment of the present application;
[0061] Fig.18 A schematic diagram of the structure of the front-end device provided in an embodiment of the present application.
[0062] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0063] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0064] With the progress of society and the improvement of people's safety awareness, the demand for security monitoring is growing. 4G ball cameras are widely used in security monitoring in the fields of security, transportation, education, and medical care due to their advantages such as wireless transmission, high-definition image quality, and intelligent analysis. When the background area changes greatly during the rotation / zooming process of the ball camera, and the 4G signal and wireless WiFi signal are weak and unstable, the encoding of video data is particularly important. In the prior art, the network bandwidth is usually estimated through the network protocol quality service library, and then the bit rate is adjusted to ensure the encoding effect. However, the encoding bit rate is forced to be reduced according to the network situation, resulting in a significant loss of the image effect after encoding, and the frame rate is forced to be reduced when the bit rate cannot be suppressed, resulting in an increase in the frame interval and a decrease in the smoothness of the moving target. If the image preprocessing and the encoding bit rate are linked to achieve a balance between image quality and encoding efficiency, better image quality can be maintained at a lower bit rate, but excessive preprocessing adjustment will cause the image before encoding to become blurred, and a lot of information will be lost before encoding, resulting in a decrease in clarity, and a single means of regulation.
[0065] In response to the above problems, the present application provides a method, device, equipment and storage medium for processing video data, which improves the stability and encoding quality of video transmission in the case of poor network signals. Specifically, in the prior art, the network bandwidth is estimated through the network protocol quality service library, and then the bit rate is adjusted to ensure the encoding effect. However, the encoding bit rate is forced to be reduced according to the network conditions, resulting in a significant loss of the image effect after encoding, and the frame rate is forced to be reduced when the bit rate cannot be suppressed, resulting in an increase in the frame interval and a decrease in the smoothness of the moving target. If the image preprocessing and the encoding bit rate are linked to achieve a balance between image quality and encoding efficiency, better image quality can be maintained at a lower bit rate, but excessive preprocessing adjustment will cause the image before encoding to become blurred, and a lot of information will be lost before encoding, resulting in a decrease in clarity and a single means of regulation. Taking these issues into consideration, the inventor studied whether it is possible to adjust the bit rate in real time based on the network bandwidth, and monitor the quantization parameter (QP) value of each frame of data after encoding in real time. When the QP value of multiple consecutive frames of data reaches the QP threshold, the ISP parameters are adjusted to reduce the encoding pressure. In order to ensure the video quality, when the QP value of multiple consecutive frames of data encoded after adjusting the image signal processor (ISP) parameters reaches the QP threshold, the encoding reference relationship is adjusted based on the motion state of the front-end device, thereby reducing the encoding pressure while ensuring the encoding quality. Based on this, the technical solution of this application is proposed.
[0066] Figure 1 A schematic diagram of the application system structure of the video data processing method provided in the embodiment of the present application is as follows: Figure 1 As shown, the system includes front-end devices such as a weak network, a front-end camera, etc., wherein the weak network includes 4G, WiFi, and can also be a satellite signal, etc., including an acquisition module, a processing module, an encoder, and an output module. Among them, the acquisition module is used to acquire video data from the image sensor, the processing module is used to perform image processing on the data acquired by the acquisition module to obtain YUV data, the encoder is used to encode the YUV data, and the output module is used to encode the code stream data encoded by the data encoder, and then transmit the code stream data through the 4G network. It can be transmitted to back-end devices, such as smartphones, laptops, desktop computers held by users, and can also be servers, clouds, etc.
[0067] In the above application system, in order to ensure the encoding effect and improve the video quality, the network bandwidth can be monitored in real time, and then the encoder bit rate can be adjusted in real time based on the network bandwidth, and the frame-level QP value of the data stream can be monitored. When the QP value of multiple consecutive frames of data reaches the preset threshold, it means that the current encoding pressure is large, then the processing parameters of the processing module are adjusted, and the encoding reference relationship is adjusted, or the encoding resolution and frame rate are reduced until the network bandwidth is met.
[0068] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0069] Figure 2 The flowchart of the first embodiment of the method for processing video data provided in the present application is as follows: Figure 2 As shown, the method can be applied to a front-end device, and the method specifically includes:
[0070] S201: If the QP values of the consecutive n frames of data encoded before the current moment are all greater than a preset QP threshold, adjust the image signal processing ISP image parameters.
[0071] In this step, in order to monitor the encoding pressure during the encoding process in real time, the QP value of the continuous n frames of data encoded before the current moment can be obtained in real time.
[0072] Specifically, the QP value is a key parameter that affects video quality and file size, and thus can be used to determine encoding pressure.
[0073] For example, for most encoders, detailed logging is enabled. This can usually be achieved by setting specific parameters in the encoding command. For example, in FFmpeg, you can use -loglevel debug or -loglevel verbose to enable more detailed log output. These logs will contain QP value information for each frame. After obtaining the log or output file, you can analyze the log or output file, either manually or by writing a script to automatically extract the relevant QP value. The above is only an example of obtaining the QP value, and the embodiment of the present application does not specifically limit the specific implementation method for obtaining the QP value.
[0074] It should be noted that the obtained QP value is used to determine the coding pressure, and the above n is an integer greater than 1, but the obtained QP value is not convincing if it is a frame, so in actual application scenarios, n can be set to an integer greater than 3. It can be set according to the actual application scenario, and the embodiment of the present application does not make specific limitations.
[0075] After obtaining the QP value of n consecutive frames of data, the QP value of each frame is compared with the preset QP threshold. If the QP values of n consecutive frames of data are all greater than the preset QP threshold, it means that the current encoding pressure is too high. The ISP image parameters can be adjusted to reduce the image clarity, thereby reducing the texture complexity of the data used for encoding, thereby reducing the encoding pressure.
[0076] Specifically, based on the QP threshold, the ISP target image parameters are determined in the ISP image parameters, and the ISP target image parameters include noise reduction parameters and sharpening parameters, thereby increasing the level of the noise reduction parameters and reducing the level of the sharpening parameters.
[0077] S202: Processing the original video data acquired at the current moment based on the adjusted ISP image parameters to obtain YUV data corresponding to the original video.
[0078] In this step, after adjusting the ISP image parameters, the original video data is processed based on the adjusted ISP image parameters to obtain YUV data, wherein the YUV data is used for encoding.
[0079] Specifically, ISP is used to convert the raw video data captured by the sensor (usually in RAW format) into data that is more suitable for subsequent processing and display (such as YUV format). YUV is a color encoding method, where Y represents brightness (Luminance or Luma), and U and V represent chrominance (Chrominance or Chroma) components, which is usually used for video compression and transmission. The specific processing of ISP includes:
[0080] Denoising: Reducing random noise in an image.
[0081] Defect Correction: Repairs dead pixels or defects in image sensors.
[0082] White Balance: Adjusts the image colors to reflect the colors of the actual light sources in the scene.
[0083] Auto Exposure: Adjusts the brightness of the image to ensure that the image is not overexposed or underexposed.
[0084] Color Correction: Adjust the color saturation, contrast, and hue of an image.
[0085] Sharpening: Enhances the edges and details of an image to make it look clearer.
[0086] Demosaicing: Converting RAW data (usually a Bayer pattern) into a full-color image.
[0087] After processing by the ISP, the data usually exists in RGB format (although different color spaces may be used for processing within the ISP). Then, the RGB data needs to be converted to YUV format for further video processing, compression, or display. Specifically, the conversion from RGB to YUV is usually performed using standard mathematical formulas that convert the RGB components into Y (brightness) and U, V (chrominance) components. These conversions can be linear or nonlinear, depending on the required color accuracy and computational efficiency. The embodiments of the present application are not specifically limited.
[0088] It should be noted that there are many variants of the YUV format (such as YUV420, YUV422, YUV444, etc.). These variants differ in the sampling and storage methods of the chrominance components. It is necessary to select the appropriate format according to the specific application scenario.
[0089] S203: In the process of encoding YUV data based on a preset target bit rate, if the QP values of the encoded continuous n frames of data are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, the encoding reference relationship is adjusted to the first encoding reference relationship until the encoding is completed to obtain the bitstream data.
[0090] In this step, after adjusting the ISP parameters, the encoding continues. If the QP value of the continuous n frames of data is still greater than the QP threshold during the encoding process, the encoding reference relationship can be adjusted based on the motion state of the front-end device, and finally the encoding is completed and the bitstream data is output. The motion state of the front-end device is used to indicate that the front-end device is stationary.
[0091] Specifically, when the motion state indicates that the front-end device is stationary, such as a weak tennis ball machine, most areas of the ball machine do not change when it is stationary (not rotating). In order to add more reference information, it can be determined to use the first coding reference relationship. The first coding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each first preset interval, and after each second preset interval, it is encoded as a Pm frame with reference to the last I frame encoded before. Between the first frame and the P1 frame, the Pm-1 frame and the Pm frame, the previous frame of the current encoding and the last I frame encoded before the current encoding are encoded as p frames, m represents the mth second preset interval, which is an integer greater than 2, Pm-1 represents the m-1th P frame, and Pm represents the mth P frame.
[0092] For example, Figure 3 is a schematic diagram of the first coding reference relationship, such as Figure 3 As shown, the P frame encoded in the first coding reference relationship refers not only to the previous frame but also to the I or P frame, and a P frame and an I frame are added at a certain interval to reduce the residual of the subsequent P frame. Among them, the encoding method of the P frame and the P frame is the same, except that the reference frame is different. This encoding method can optimize the transmission efficiency while maintaining the video quality. It should be noted that the reference Figure 3 When encoding according to the above reference relationship, it is also necessary to preset the interval encoding I frame. Taking the common frame rate of 25fps as an example, the I frame interval is 50 (equal to 2 seconds to generate an I frame), there are 49 frames between two I frames, each I frame is only used as an intra-frame reference, and does not refer to other frames. In this frame structure, the I frame interval is 200, the P frame interval is 50, the P frame only refers to the I frame, and the other frames are ordinary P frames, which refer to the previous frame and the I frame. In this reference structure, the user clicks on any frame, and only needs to find the P frame before the frame to decode it sequentially. At most 49 frames can be decoded to display the current frame.
[0093] In a possible implementation, if the QP values of the encoded n consecutive frames of data are still greater than the QP threshold after the encoding reference relationship is adjusted, the encoding frame rate and resolution can be reduced until the network bandwidth is met.
[0094] Optionally, when the network signal is weakened and the network bandwidth cannot meet the data volume of the bitstream transmission, the target bitrate is adjusted based on the network bandwidth at the current moment.
[0095] In this step, the network bandwidth is monitored in real time. When it is detected that the signal is weakening and the network bandwidth of the communication link cannot meet the data volume required for bitstream transmission, the bitrate value configured for the encoder is reduced. After reducing the bitrate value, the loss after encoding will theoretically increase and the image effect will deteriorate. Therefore, while reducing the bitrate, the frame-level QP value of the encoded data should be monitored in real time to ensure the video quality.
[0096] Specifically, the network signal is monitored. When the original signal from the network interface is received, it is parsed and measured to obtain the signal strength indicators, such as signal strength indication (RSSI), signal-to-noise ratio (SNR), etc. The system can preset a series of signal strength thresholds, which are determined based on network standards, device performance, and application scenario requirements. The real-time measured signal strength is compared with the preset threshold. If it is lower than a certain threshold, the signal is considered to be weak.
[0097] As for the way to obtain network bandwidth, the current network bandwidth can be evaluated by measuring the amount of data successfully transmitted per unit time. This usually involves timing and counting the transmitted data packets. Common measurement methods include throughput testing, which is to send a large amount of data within a certain period of time and measure the amount of data received.
[0098] The required amount of data to be transmitted can also be determined based on the application requirements. The actual network bandwidth required must also be calculated taking into account data redundancy, coding efficiency, and network protocol overhead.
[0099] The actual network bandwidth measured is then compared with the required data volume for code stream transmission. If the actual bandwidth is lower than the required bandwidth, it is considered that the network bandwidth of the communication link cannot meet the data volume requirement for code stream transmission.
[0100] When a decrease in network bandwidth is detected, the encoding bit rate is lowered to reduce the amount of data transmitted and avoid network congestion and buffering delays.
[0101] On the contrary, when the network bandwidth increases, the encoding bit rate can be appropriately increased to improve video quality.
[0102] It should be noted that the encoding bit rate has a preset bit rate upper limit value, and during the dynamic adjustment process, the bit rate upper limit value cannot be exceeded.
[0103] Optionally, during the encoding process using the first encoding reference relationship, the QP value of the consecutive n frames of data obtained after encoding is detected in real time. If it is still greater than the QP threshold and it is detected that the current network bandwidth is less than the preset first bandwidth threshold, the first encoding reference relationship is adjusted to a third encoding reference relationship.
[0104] Specifically, the third coding reference relationship includes: encoding as an I frame only in the first frame, encoding as a Pa frame with reference to the first frame after each fifth preset interval, and encoding as a p frame with reference to the previous frame currently encoded and the first frame between the first frame and the P1 frame, the Pa-1 frame and the Pa frame, where a represents the ath fifth preset interval, which is an integer greater than 2, Pa-1 represents the a-1th P frame, and Pa represents the ath P frame.
[0105] For example, Figure 4is a schematic diagram of the third coding reference relationship, such as Figure 4 As shown above Figure 3 The frame structure can solve the problem of reducing residual errors and quickly viewing recordings, but an I frame will still be generated every 200 frames. For extremely low-bandwidth scenarios, it is still easy to cause instantaneous bandwidth congestion, resulting in image freezes. When the QP and network bandwidth thresholds are met, I frames can no longer be generated regularly.
[0106] It should be noted that an I frame will be generated immediately in the following situations: the original frame structure cannot be decoded normally due to packet loss or other reasons, and an I frame will be generated immediately; the scene detection results for N consecutive seconds show that the current scene should change, and the original I frame is no longer of reference, and an I frame will be generated immediately.
[0107] In one possible design, during encoding with a third coding reference relationship, the frame structure can also dynamically adjust the bit rate value at the current moment according to the motion ratio of the current scene, where the motion ratio is used to indicate the rate of change of the original video data for the macroblock pixel value.
[0108] For example, Figure 5 A schematic diagram of a third coding reference relationship for real-time bit rate adjustment, such as Figure 5 As shown in the figure, when the motion ratio exceeds the set threshold, a high bit rate value is set immediately, and the next frame is required to take effect immediately, without waiting for the next I frame or large P frame to take effect. The final size structure diagram of each frame is as follows Figure 5 As shown, the frames in the dotted box correspond to the frames where the motion ratio exceeds the threshold and a high bit rate is set. During this period, the bit rate is high and the encoding effect is good. The frames outside the dotted box correspond to the sizes when the motion ratio is low.
[0109] The video data processing method provided in this embodiment, if the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, then the image signal processing ISP image parameters are adjusted, and the original video data obtained at the current moment is processed based on the adjusted ISP image parameters to obtain the YUV data corresponding to the original video. In the process of encoding the YUV data based on the preset target bit rate, if the QP values of the consecutive n frames of data obtained by encoding are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, the encoding reference relationship is adjusted to the first encoding reference relationship until the encoding is completed to obtain the bitstream data. The above method effectively responds to the problems of network signal weakening and insufficient bandwidth by dynamically adjusting the ISP image parameters and the encoding reference relationship, improves the stability of video transmission, encoding quality and user experience, and reduces bandwidth occupancy and storage costs.
[0110] Figure 6The flowchart of the second embodiment of the method for processing video data provided in the present application is as follows: Figure 6 As shown, the method includes:
[0111] S601: If the QP values of the consecutive n frames of data encoded before the current moment are all greater than a preset QP threshold, adjust the image signal processing ISP image parameters.
[0112] S602: Processing the original video data acquired at the current moment based on the adjusted ISP image parameters to obtain YUV data corresponding to the original video.
[0113] Step S601 and step 602 are implemented in the same manner as step S201 and step S202 in the aforementioned embodiment, and are not described in detail herein.
[0114] S603: In the process of encoding YUV data based on a preset target bit rate, if the QP values of the encoded continuous n frames of data are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is moving, the encoding reference relationship is adjusted to the second encoding reference relationship until the encoding is completed to obtain the bitstream data.
[0115] In this step, when the motion state indicates that the front-end device is moving, such as a weak tennis ball machine, most of the background areas are changing when the ball machine is moving (rotating), and the conventional reference relationship residual is large, resulting in high coding pressure and poor image quality. In addition, the amount of encoded code stream data may be large during the movement, causing network packet loss and other problems. The second coding reference relationship is used for encoding.
[0116] Among them, the second coding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each third preset interval, and after each fourth preset interval, it is encoded as a Pn frame with reference to the last I frame encoded previously; between the first frame and the P1 frame, it is encoded as multiple bidirectionally predicted B frames with reference to the first frame and the P1 frame; between the Pn-1 frame and the Pn frame, it is encoded as multiple B frames with reference to the Pn-1 frame and the Pn frame, n represents the nth fourth preset interval, which is an integer greater than 2, Pn-1 represents the n-1th P frame, and Pn represents the nth P frame.
[0117] For example, Figure 7 is a schematic diagram of the second coding reference relationship, such as Figure 7 As shown in the figure, B frames refer to the previous and next frames, which makes it easier to find matching blocks during motion and reduce residuals, making B frames particularly small, reducing encoding pressure and improving video encoding effects overall. At the same time, all P frames only refer to I frames, so that even if a P frame is lost, the following P frames can still be used. Because it is a 4G network, there may be a problem of poor signal, and this reference relationship can play a role in preventing packet loss.
[0118] The background changes dramatically when the ball camera rotates, and the amount of data that needs to be encoded increases. When the network bandwidth is low, the slow transmission and severe packet loss will cause the screen to freeze and display blur, which leads to a very poor user experience. In addition, the slip ring of the ball camera has a service life problem (related to the number of rotations). When the network bandwidth is low, the rotation of the ball camera not only wastes the life of the slip ring, but also has a poor preview experience. When the QP and network bandwidth thresholds are met, the ball camera can be restricted from rotating.
[0119] It should be noted that the preset interval in this embodiment is determined according to the encoding frame rate and the actual application scenario.
[0120] The video data processing method provided in this embodiment, if the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, then the image signal processing ISP image parameters are adjusted, and the original video data obtained at the current moment is processed based on the adjusted ISP image parameters to obtain the YUV data corresponding to the original video, in the process of encoding the YUV data based on the preset target bit rate, in the process of encoding the YUV data based on the preset target bit rate, if the QP values of the consecutive n frames of data obtained by encoding are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is moving, the encoding reference relationship is adjusted to the second encoding reference relationship until the encoding is completed to obtain the bitstream data. The above method optimizes the video encoding process, so that it can adapt to different motion scenes, increase the encoding flexibility, and achieve the effect of improving encoding efficiency, reducing data volume and maintaining video quality.
[0121] Figure 8 The flowchart of the third embodiment of the method for processing video data provided in the embodiment of the present application is as follows: Figure 8 As shown, based on the above embodiment, step S201 specifically includes:
[0122] S801: Based on the QP threshold, determine the ISP target image parameters in the ISP image parameters.
[0123] In this step, in order to ensure the coding pressure, the influence of each ISP image parameter on the average bit rate of the output bit stream can be analyzed based on the fixed QP, so as to determine the initial ISP target image parameters. Through the determination, it is concluded that the ISP target image parameters include noise reduction parameters and sharpening parameters.
[0124] For example, Fig. 9 This is a schematic diagram of the effect of noise reduction parameters on bit rate. Fig.10 This is a schematic diagram showing the effect of sharpening parameters on bit rate. Fig. 9 As shown in Figure 2, by evaluating the noise reduction parameters in a fixed QP manner, it can be concluded that the greater the noise reduction level, the smaller the bit rate. Fig.10As shown, by evaluating the sharpening parameters in a fixed QP manner, it can be concluded that the greater the sharpening level, the greater the bit rate.
[0125] S802: Raise the level of the noise reduction parameter.
[0126] S803: Reduce the level of sharpening parameters.
[0127] In order to reduce the texture complexity of the YUV data and reduce the encoding pressure, the level of the noise reduction parameter can be increased and / or the level of the sharpening parameter can be reduced.
[0128] Exemplarily, the texture complexity of YUV data can be measured by standard deviation. Exemplarily, in units of macroblocks, the texture complexity of a macroblock corresponds to the standard deviation of the brightness component of a 16*16 pixel area. The larger the value, the more complex the texture, and the smaller the value, the simpler the texture.
[0129] Fig.11 It is a schematic diagram of texture complexity comparison. Fig.11 As shown, for a white wall (box 1), the value of each pixel in a macroblock is very close, and the difference between the value corresponding to each pixel and the mean is very small, so the calculated standard deviation will also be very small, so the texture of the white wall is simple.
[0130] Leaves / grass (box 2), the value of each pixel in a macroblock varies greatly, some are bright, some are dark, some are white, and some are green. The difference between the value corresponding to each pixel and the mean is very large, so the calculated standard deviation will also be very large, so the texture of leaves / grass is complex.
[0131] Fig.12 This is a comparison diagram of noise reduction level 0 and noise reduction level 100. Fig.13 This is a comparison diagram of sharpening level 0 and sharpening level 100, combined with Fig.12 and Fig.13 As shown in the figure, by adjusting the noise reduction intensity or sharpening intensity, the image will look blurred. Correspondingly, the difference between each pixel becomes smaller after being processed by the noise and sharpening algorithms, and the corresponding texture complexity will also decrease.
[0132] Optionally, when there is a lot of random noise (such as brightness noise, dark noise, etc.) in the image, increasing the noise reduction level can effectively reduce these noises and make the image cleaner. In application scenarios that require high-quality video output, such as high-definition movies and professional surveillance, the purity of the image is required to be high. At this time, appropriately increasing the noise reduction level can improve the overall quality of the video. When the performance of the encoder used is limited, increasing the noise reduction level to reduce image complexity can reduce the burden on the encoder and improve encoding efficiency.
[0133] If the image has been over-sharpened, resulting in jagged edges, false textures, and other problems, the sharpening level needs to be reduced. In some application scenarios that do not require high image detail retention, such as live webcasts and video conferencing, appropriately reducing the sharpening level can reduce high-frequency information in the image, thereby reducing encoding pressure. In some cases, excessively high sharpening parameters may introduce additional encoding noise. By reducing the sharpening level, this noise can be reduced and the quality of the encoded image can be improved.
[0134] When the image quality is not high and there is obvious noise, it is necessary to increase the noise reduction level to reduce the noise and appropriately reduce the sharpening level to avoid introducing additional noise or aliasing. When the encoder performance is limited but a certain image quality needs to be guaranteed, it is necessary to adjust the noise reduction and sharpening parameters at the same time to find the best balance. That is, it is necessary to reduce image noise while avoiding quality problems caused by over-sharpening. In some specific application scenarios (such as night mode for security monitoring), in order to ensure that the image clarity and noise level are within an acceptable range, it may be necessary to adjust the noise reduction and sharpening parameters at the same time.
[0135] The embodiments of the present application do not specifically limit the implementation of specific adjustment parameters for reducing texture complexity.
[0136] The specific implementation method of reducing the noise reduction level and increasing the sharpening level is not specifically limited in the embodiment of the present application.
[0137] The video data processing method provided in this embodiment determines the ISP target image parameters in the ISP image parameters based on the QP threshold, increases the level of the noise reduction parameters, and reduces the level of the sharpening parameters. The above method of adjusting the ISP image parameters based on the QP threshold improves the image quality, can also adapt to different network environments, improves the user experience, and saves bandwidth resources.
[0138] Fig.14 The flowchart of the fourth embodiment of the method for processing video data provided in the embodiment of the present application is as follows: Fig.14 As shown, based on the above embodiment, the method further includes:
[0139] S1401: Downsample the resolution of the original video data to generate a Bitmap table.
[0140] S1402: For each macroblock corresponding to the downsampled original video data, if the change value of the pixel value of the macroblock is greater than a preset pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 1.
[0141] S1403: If the change value of the pixel value of the macroblock is less than the pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 0.
[0142] S1404: Determine the proportion of 1 in the Bitmap table as the motion proportion.
[0143] For example, the width and height of the resolution captured by the sensor are downsampled by 1 / 16, and a bitmap table of (original image width / 16)*(original image height / 16) is generated at the same time, and each element in the table only occupies 1 bit. Frame difference is performed on the downsampled image to determine the change in the pixel value of the 16*16 macroblock. If the change value reaches the set threshold, it is considered that the macroblock is moving, and the corresponding position in the bitmap table is set to 1, otherwise it is set to 0. Finally, a bitmap table representing the movement and stillness of all 16*16 macroblocks is generated, and the sum of the number of 1s in the number of all elements is used to obtain the motion ratio.
[0144] The video data processing method provided in this embodiment performs downsampling processing on the resolution of the original video data to generate a Bitmap table. For each macroblock corresponding to the downsampled original video data, if the change value of the pixel value of the macroblock is greater than the preset pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 1; if the change value of the pixel value of the macroblock is less than the pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 0, and the proportion of 1 in the Bitmap table is determined as the motion proportion. By adjusting the encoding bit rate of the next frame in real time according to the motion proportion, the encoding effect is improved and the video quality is guaranteed.
[0145] Optionally, based on the foregoing embodiment, when the motion state indicates that the front-end device is stationary and it is detected that the current network bandwidth is greater than the first bandwidth threshold and less than the preset second bandwidth threshold, the coding reference relationship can be adjusted to a fourth coding reference relationship.
[0146] Specifically, the fourth coding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each sixth preset interval, and between two I frames, the previous frame and the last I frame that has been encoded are referenced as reference frames and encoded as a p frame, and the first bandwidth threshold is less than the second bandwidth threshold.
[0147] For example, Fig.15 is a schematic diagram of the fourth coding reference relationship, such as Fig.15 As shown, the original coding reference relationship is blocked by a moving target in the originally static area, and only referring to part of the previous frame area cannot find a matching block, resulting in a large coding residual of the current frame. If the previous frame can be referred to while also referring to an earlier I frame, it is easier to find a matching block, thereby reducing the residual.
[0148] Optional, Fig.16 A schematic diagram of a fourth coding reference relationship adjusted in real time, such as Fig.16 As shown above Fig.15The medium reference relationship can reduce the coding residual, but the I frame interval is still 50. Every 50 frames, a large I frame will still be generated, causing regular bit rate fluctuations. This is not friendly to products with weak networks or data-saving products. In order to reduce bit rate fluctuations, the I frame interval can be increased, and the original I frame interval of 50 can be changed to an I frame interval of 200. However, this reference method will cause the display time to be too long when viewing the video. If the user clicks on the N+199 frame when viewing the video, it will be necessary to start from the N frame and decode 199 frames to preview the frame image clicked by the user. In this way, the user needs to wait too long, and the user experience is not good. The frame reference relationship can be adjusted to Figure 3 The reference relationship shown can solve the problem of users decoding video quickly.
[0149] Fig.17 A schematic diagram of the structure of an embodiment of a video data processing device provided in an embodiment of the present application is shown in FIG. Fig.17 As shown, the video data processing device 1700 includes:
[0150] The first adjustment module 1701 is used to adjust the image signal processing ISP image parameters if the QP values of the consecutive n frames of data encoded before the current moment are all greater than a preset QP threshold.
[0151] The processing module 1702 is used to process the original video data obtained at the current moment based on the adjusted ISP image parameters to obtain YUV data corresponding to the original video.
[0152] The second adjustment module 1703 is used to adjust the encoding reference relationship to the first encoding reference relationship during the encoding of YUV data based on the preset target bit rate, if the QP values of the encoded continuous n frames of data are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, until the encoding is completed to obtain the code stream data, the first encoding reference relationship includes: the first frame is encoded as an intra-frame coded I frame, an I frame is encoded after each first preset interval, and after each second preset interval, it is encoded as a forward predicted Pm frame with reference to the last I frame encoded previously, and between the first frame and the P1 frame, the Pm-1 frame and the Pm frame, the frame is encoded as a p frame with reference to the previous frame currently encoded and the last I frame encoded previously before the current encoding, m represents the mth second preset interval, which is an integer greater than 2, Pm-1 represents the m-1th P frame, and Pm represents the mth P frame.
[0153] Optionally, the second adjustment module 1703 may also be used to:
[0154] When the motion state indicates that the front-end device is moving, the encoding reference relationship is adjusted to the second encoding reference relationship until the encoding is completed to obtain the code stream data. The second encoding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each third preset interval, and after each fourth preset interval, it is encoded as a Pn frame with reference to the last I frame encoded previously. Between the first frame and the P1 frame, multiple B frames are encoded with reference to the first frame and the P1 frame. Between the Pn-1 frame and the Pn frame, multiple B frames are encoded with reference to the Pn-1 frame and the Pn frame. n represents the nth fourth preset interval, which is an integer greater than 2, Pn-1 represents the n-1th P frame, and Pn represents the nth P frame.
[0155] Optionally, the second adjustment module 1703 may also be used to:
[0156] During the encoding process using the first encoding reference relationship, if the QP values of the encoded consecutive n frames of data are all greater than the QP threshold and it is detected that the current network bandwidth is less than the preset first bandwidth threshold, the first encoding reference relationship is adjusted to a third encoding reference relationship for encoding, and the third encoding reference relationship includes: only the first frame is encoded as an I frame, and after each fifth preset interval, it is encoded as a Pa frame with reference to the first frame, and between the first frame and the P1 frame, and the Pa-1 frame and the Pa frame, it is encoded as a p frame with reference to the previous frame currently encoded and the first frame, where a represents the ath fifth preset interval and is an integer greater than 2, Pa-1 represents the a-1th P frame, and Pa represents the ath P frame.
[0157] Optionally, the second adjustment module 1703 may also be used to:
[0158] During the encoding process using the third encoding reference relationship, the encoding bit rate is adjusted based on the motion ratio of the original video data, and the adjusted bit rate is used for encoding the next frame. The motion ratio is used to indicate the rate of change of the original video data for the macroblock pixel value.
[0159] Optionally, the processing module 1702 may also be used to:
[0160] Downsample the resolution of the original video data to generate a Bitmap table;
[0161] For each macroblock corresponding to the downsampled original video data, if the change value of the pixel value of the macroblock is greater than the preset pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 1;
[0162] If the change in the pixel value of the macroblock is less than the pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 0;
[0163] The proportion of 1 in the Bitmap table is determined as the motion proportion.
[0164] Optionally, the first adjustment module 1701 is specifically configured to:
[0165] Based on the QP threshold, an ISP target image parameter is determined in the ISP image parameter, and the ISP target image parameter includes a noise reduction parameter and a sharpening parameter;
[0166] Increase the level of noise reduction parameters;
[0167] Reduce the level of sharpening parameters.
[0168] Optionally, the first adjustment module 1701 is further configured to:
[0169] When the network signal weakens and the network bandwidth cannot meet the data volume of the bitstream transmission, the target bitrate is adjusted based on the current network bandwidth.
[0170] If the QP values of the encoded n consecutive frames of data after adjusting the encoding reference relationship are all greater than the QP threshold, the encoding frame rate and resolution are reduced.
[0171] Optionally, the second adjustment module 1703 may also be used to:
[0172] When the motion state indicates that the front-end device is stationary and it is detected that the current network bandwidth is greater than the first bandwidth threshold and less than the preset second bandwidth threshold, the coding reference relationship is adjusted to a fourth coding reference relationship for encoding, and the fourth coding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each sixth preset interval, and between two I frames, the previous frame and the last I frame that has been encoded are referenced as reference frames and encoded as p frames, and the first bandwidth threshold is less than the second bandwidth threshold.
[0173] Optionally, the second adjustment module 1703 may also be used to:
[0174] During encoding using the second encoding reference relationship, if the QP values of the encoded n consecutive frames of data are all greater than the QP threshold and it is detected that the current network bandwidth cannot meet the data volume of the code stream transmission, the motion state of the front-end device is adjusted to still.
[0175] Fig.18 A schematic diagram of the structure of the front-end device provided in the embodiment of the present application, such as Fig.18 As shown, the front-end device 1800 includes: a processor 1802, and a memory 1801, a communication interface 1803, and an image sensor 1804 that are communicatively connected to the processor 1802;
[0176] The memory 1801 stores computer executable instructions;
[0177] The processor 1802 executes the computer-executable instructions stored in the memory 1801 to implement the video data processing method in any method embodiment.
[0178] The communication interface 1803 is used for communication connection with external devices.
[0179] The image sensor 1804 is used to collect video data in real time.
[0180] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to execute the video data processing methods provided by the various embodiments described above.
[0181] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0182] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0183] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program is stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution provided by any of the above method embodiments.
[0184] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship; in the formula, the character " / " indicates that the previous and next associated objects are in a "division" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0185] It is to be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of description and are not intended to limit the scope of the embodiments of the present application. In the embodiments of the present application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0186] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0187] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for processing video data, characterized in that: Applied to a front-end device, the method comprises: If the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, the image signal processing ISP image parameters are adjusted; Processing the original video data acquired at the current moment based on the adjusted ISP image parameters to obtain YUV data corresponding to the original video; In the process of encoding the YUV data based on the preset target bit rate, if the QP values of the encoded continuous n frames of data are all greater than the QP threshold, and the motion state of the front-end device indicates that the front-end device is stationary, the encoding reference relationship is adjusted to the first encoding reference relationship until the encoding is completed to obtain the code stream data, and the first encoding reference relationship includes: the first frame is encoded as an intra-frame coded I frame, an I frame is encoded after each first preset interval, and after each second preset interval, the last I frame encoded in the front is encoded as a forward prediction P frame. m Frame, between the first frame and P1 frame, P m-1 Frame and P m Between frames, the previous frame of the current encoding and the last I frame encoded before the current encoding are encoded as p frames, m represents the mth second preset interval, which is an integer greater than 2, and the P m-1 represents the m-1th P frame, the P m Indicates the mth P frame.
2. The method according to claim 1, characterized in that The method further comprises: When the motion state indicates that the front-end device is moving, the coding reference relationship is adjusted to a second coding reference relationship until the coding is completed to obtain the code stream data, wherein the second coding reference relationship includes: the first frame is coded as an I frame, an I frame is coded after each third preset interval, and a P frame is coded after each fourth preset interval with reference to the last I frame coded previously. n frame, between the first frame and the P1 frame, the first frame and the P1 frame are referenced to encode into a plurality of bidirectional prediction B frames, between the P n-1 Frame and P n Between frames, refer to the P n-1 Frame and the P n The frame is encoded as a plurality of B frames, n represents the nth fourth preset interval, which is an integer greater than 2, and the P n-1 represents the n-1th P frame, the P n Indicates the nth P frame.
3. The method according to claim 1, characterized in that The method further comprises: In the encoding process using the first encoding reference relationship, if the QP values of the encoded consecutive n frames of data are all greater than the QP threshold and it is detected that the current network bandwidth is less than the preset first bandwidth threshold, the first encoding reference relationship is adjusted to a third encoding reference relationship for encoding, and the third encoding reference relationship includes: only encoding the first frame as an I frame, and encoding the first frame as a P frame after each fifth preset interval with reference to the first frame. a Frame, between the first frame and P1 frame, P a-1 Frame and P a Between frames, the previous frame and the first frame of the current encoding are encoded as p frames, a represents the ath fifth preset interval, which is an integer greater than 2, and the P a-1 represents the a-1th P frame, the P a Indicates the ath P frame.
4. The method according to claim 3, characterized in that The method further comprises: During the encoding process using the third encoding reference relationship, the encoding bit rate is adjusted based on the motion ratio of the original video data, and the adjusted bit rate is used for encoding the next frame, wherein the motion ratio is used to indicate the rate of change of the original video data for the macroblock pixel value.
5. The method according to claim 4, characterized in that Before adjusting the encoding bit rate based on the motion ratio of the original video data, the method further includes: Downsampling the resolution of the original video data to generate a Bitmap table; For each macroblock corresponding to the downsampled original video data, if the change value of the pixel value of the macroblock is greater than a preset pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 1; If the change value of the pixel value of the macroblock is less than the pixel value threshold, the corresponding position of the macroblock in the Bitmap table is set to 0; The proportion of 1 in the Bitmap table is determined as the motion proportion.
6. The method according to any one of claims 1 to 5, characterized in that: If the QP values of the consecutive n frames of data encoded before the current moment are all greater than the preset QP threshold, adjusting the image signal processing ISP image parameters includes: Based on the QP threshold, determining ISP target image parameters from the ISP image parameters, the ISP target image parameters including noise reduction parameters and sharpening parameters; increasing the level of the noise reduction parameter; Reduce the level of the sharpening parameter.
7. The method according to any one of claims 1 to 5, characterized in that: Before adjusting the image signal processing ISP image parameters, the method further includes: When the network signal weakens and the network bandwidth cannot meet the data volume of the bitstream transmission, the target bitrate is adjusted based on the network bandwidth at the current moment.
8. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: If the QP values of the consecutive n frames of data encoded after adjusting the encoding reference relationship are all greater than the QP threshold, the encoding frame rate and resolution are reduced.
9. The method according to claim 1, characterized in that: The method further comprises: When the motion state indicates that the front-end device is stationary and it is detected that the current network bandwidth is greater than the first bandwidth threshold and less than the preset second bandwidth threshold, the encoding reference relationship is adjusted to a fourth encoding reference relationship for encoding, and the fourth encoding reference relationship includes: the first frame is encoded as an I frame, an I frame is encoded after each sixth preset interval, and between two I frames, the previous frame and the last I frame that has been encoded are referenced as reference frames and encoded as a p frame, and the first bandwidth threshold is less than the second bandwidth threshold.
10. The method according to claim 2, characterized in that The method further comprises: During encoding using the second encoding reference relationship, if the QP values of the encoded n consecutive frames of data are all greater than the QP threshold and it is detected that the current network bandwidth cannot meet the data volume of the code stream transmission, the motion state of the front-end device is adjusted to still.
Citation Information
Patent Citations
Video coding method and device and video decoding method and device
CN105847793A
Encoding method and device, encoder, equipment and computer readable storage medium
CN113489993A
Video coding method and device and electronic equipment
CN115914629A
Image processing method, encoding device, decoding device and storage medium
CN116800965A
Image encoder and control method and program thereof
JP2018082252A
Cited By
Monitoring video efficient compression method and system for intelligent monitoring
CN120547360A
