Coding method, device, encoder, equipment and computer-readable storage medium

Abstract

The embodiment of the present application discloses a coding method, apparatus, encoder, device and computer-readable storage medium, the method comprising: determining video data to be encoded, and performing image processing on the video data to be encoded to obtain processed video data; determining a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data; encoding the processed video data according to the first QP value and the second QP value to generate coded data; encapsulating the coded data to generate a transmission video packet and sending it.

Description

Technical Field

[0001] The present application relates to the field of live broadcast application technology, and in particular to an encoding method, apparatus, encoder, device and computer-readable storage medium. Background Art

[0002] With the development of computer technology, live broadcast has become a popular interactive communication method. It allows users to watch live videos in real time, and using video encoding technology to provide clear and smooth live videos is crucial to the user's viewing experience.

[0003] In the related art, for video encoding methods, terminal devices currently usually adopt hardware encoding solutions, but the compatibility of hardware encoding is poor, and traditional video encoding methods also consume more network bandwidth, which reduces the live broadcast effect. Summary of the invention

[0004] The present application proposes a coding method, apparatus, encoder, device and computer-readable storage medium. By introducing a software coding scheme, the coding quality and compatibility can be improved. Moreover, the introduction of ROI coding can also solve the problem in related technologies that more network bandwidth is consumed during live video broadcast.

[0005] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an encoding method, the method comprising:

[0007] Determining the video data to be encoded, and performing image processing on the video data to be encoded to obtain processed video data;

[0008] Determine a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data;

[0009] encoding the processed video data according to the first QP value and the second QP value to generate encoded data;

[0010] The encoded data is encapsulated to generate transmission video packets and send them.

[0011] In a second aspect, an embodiment of the present application provides a coding device, the coding device comprising a determining unit, a processing unit and a coding unit; wherein,

[0012] A determination unit, configured to determine the video data to be encoded;

[0013] A processing unit configured to perform image processing on the video data to be encoded to obtain processed video data;

[0014] The determination unit is further configured to determine a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data;

[0015] an encoding unit configured to encode the processed video data according to the first QP value and the second QP value to generate encoded data;

[0016] The processing unit is further configured to encapsulate the encoded data, generate a transmission video packet and send it.

[0017] In a third aspect, an embodiment of the present application provides an encoder, the encoder comprising a memory and a processor; wherein:

[0018] A memory for storing computer programs that can be run on the processor;

[0019] A processor is used to execute the method of the first aspect when running a computer program.

[0020] In a fourth aspect, an embodiment of the present application provides a terminal device, the terminal device comprising a memory and a processor; wherein:

[0021] A memory for storing computer programs that can be run on the processor;

[0022] A processor is used to execute the method of the first aspect when running a computer program.

[0023] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the method of the first aspect.

[0024] The embodiments of the present application provide a coding method, apparatus, encoder, device and computer-readable storage medium, which determine video data to be encoded, and perform image processing on the video data to be encoded to obtain processed video data; determine a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data; encode the processed video data according to the first QP value and the second QP value to generate coded data; encapsulate the coded data to generate a transmission video packet and send it. In this way, by introducing a software encoding solution, the encoding quality and compatibility can be improved, and the phenomenon that the encoding quality problem caused by the integration of hardware encoding can only be solved by relying on chip suppliers can be avoided; in addition, by introducing ROI encoding, that is, reducing the QP value of the foreground area and increasing the QP value of the background area, a higher bit rate can be allocated to the foreground area and a lower bit rate can be allocated to the background area during the encoding process. Therefore, under the same bit rate or a lower bit rate, not only the video quality of the ROI area can be improved, but also the network bandwidth can be saved, thereby achieving the purpose of saving data traffic when users use data network live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of another encoding method proposed in an embodiment of the present application;

[0027] Figure 3 A flowchart of another encoding method proposed in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a flow chart of another encoding method proposed in an embodiment of the present application;

[0029] Figure 5A A schematic diagram of an interface of a terminal device APP streaming terminal provided in an embodiment of the present application;

[0030] Figure 5B A schematic diagram of a display interface of a B station live broadcast server provided in an embodiment of the present application;

[0031] Figure 6 A detailed flowchart of an encoding method provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of the structure of a coding device provided in an embodiment of the present application;

[0033] Figure 8A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the hardware structure of an encoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0037] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] In machine vision and image processing, the area to be processed is outlined in the form of a box, circle, ellipse, irregular polygon, etc. from the processed image, which can be called a region of interest (ROI). Among them, ROI encoding is a video encoding technology based on the region of interest. The quantization parameter (QP) value is reduced for the region of interest in the image, thereby allocating more bit rate to improve the picture quality, while the QP value is increased for the non-interested region in the image, thereby allocating less bit rate (the picture quality of this part of the area will be reduced as a result). In this way, without losing the overall image quality, network bandwidth occupancy and video storage space can be saved, or, without increasing network bandwidth occupancy and storage space, the overall quality of the video can be improved.

[0039] Terminal devices (such as smart phones, Android phones, etc.) usually use digital signal processing (DSP) hardware codec solutions. With the improvement of the performance of the central processing unit (CPU) on the terminal device side, the terminal device can also support software codec solutions, which have better compatibility than hardware codecs. Among them, the highly compressed digital video codec standard (H.264) is the current mainstream video encoding solution, and the open source H.264 video encoding function library (x264) is the application version of H.264. x264-ROI encoding is to add the ROI encoding solution to the x264 encoding framework, which can improve the video quality of the ROI area at a relatively low bit rate.

[0040] At present, the live video industry is booming. With the improvement of the pixel and video quality of the terminal device itself, live broadcasting on the terminal device side has become a development trend. However, in the current live video industry, the bandwidth cost investment is very huge. How to reduce the network bandwidth in live video has become an important problem that technicians in this field need to solve urgently.

[0041] The embodiment of the present application provides a coding method. After the video data to be coded is image processed, the QP values ​​corresponding to the coding blocks in the foreground area and the background area can be determined for the processed video data, and then the corresponding QP values ​​are used for coding, and then the obtained coded data is packaged and processed to generate a transmission video packet and send it. In this way, by introducing a software coding scheme, the coding quality and compatibility can be improved, and the phenomenon that the coding quality problem caused by the integration of hardware coding can only be solved by relying on chip suppliers can be avoided; in addition, by introducing ROI coding, that is, reducing the QP value of the foreground area and increasing the QP value of the background area, a higher bit rate can be allocated to the foreground area and a lower bit rate can be allocated to the background area during the coding process, so that under the same bit rate or lower bit rate, not only the video quality of the ROI area can be improved, but also the network bandwidth can be saved, and the purpose of saving data traffic when the user adopts data network live broadcast is achieved.

[0042] The technical solutions in each embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0043] In one embodiment of the present application, see Figure 1 , which shows a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application. Figure 1 As shown, the method may include:

[0044] S101: Determine video data to be encoded.

[0045] It should be noted that the encoding method of the embodiment of the present application is applied to an encoding device, or a terminal device integrated with the encoding device. Wherein, the terminal device can be various types of devices with video encoding capabilities during implementation. For example, the terminal device can be an encoder, or it can be any device with video encoding and decoding capabilities or only video encoding capabilities such as a smart phone, a tablet computer, a handheld computer, a television, a projector, a personal computer, a personal digital assistant (PDA), a wearable device, etc., without any limitation here.

[0046] It should also be noted that the functions implemented by the encoding method in the embodiment of the present application can be implemented by calling a computer program by the processor in the terminal device, and of course the computer program can be stored in the memory; it can be seen that the terminal device includes at least a memory and a processor.

[0047] In some embodiments, the video data to be encoded may be obtained by collecting data using a camera configured by the terminal device.

[0048] Here, the data collection process can be to call the application programming interface (Application Programming Interface, API) of the Xth generation camera software (CameraX) of the Android system at the JAVA layer. Among them, CameraX is a Jetpack support library designed to simplify the development of camera applications. It can provide a consistent and easy-to-use API interface that is suitable for most terminal devices with Android systems. Specifically, Camera1 needs to manage the Camera camera instance by itself, and needs to deal with issues such as the inherited class of the view (SurfaceView), preview size, and screen size; while Camera2 does not solve the problem of more complex use. It provides more calling interfaces, better customizability, but more difficult to use. In the embodiment of the present application, CameraX is used. CameraX is actually based on Camera2, but the calling API interface is encapsulated to solve the problem of complexity of use.

[0049] Furthermore, if the live broadcast scene is darker, in order to improve the brightness of the live broadcast scene, fill light processing can also be performed on the environment. Therefore, in some embodiments, the determination of the video data to be encoded may include:

[0050] Detect the ambient light intensity in the current environment;

[0051] When the ambient light intensity is less than the preset intensity threshold, fill light to the current environment;

[0052] According to the environment after fill-lighting, data is collected through the camera to obtain the video data to be encoded.

[0053] Furthermore, the method may also include: when the ambient light intensity is greater than or equal to a preset intensity threshold, there is no need to supplement the current environment with light, and data can be directly collected through the camera to obtain the video data to be encoded.

[0054] In the embodiment of the present application, in addition to the camera, the terminal device may also be equipped with a light sensing device and a fill light device. The light sensing device may be an ambient light sensor, a device light sensor, etc., for detecting the ambient light intensity in the current environment, or for sensing the ambient light brightness; the fill light device may be a fill light, a flashlight, a device screen, etc., for performing fill light processing on the current environment in a darker live broadcast scene.

[0055] In a specific example, the fill lighting for the current environment may include: if the camera is a front camera of the terminal device, fill lighting is performed through the screen of the terminal device; or, if the camera is a rear camera of the terminal device, fill lighting is performed through the flashlight of the terminal device.

[0056] It should be noted that, taking the terminal device as a mobile phone as an example, the embodiment of the present application can call the mobile phone sensor to sense the ambient light intensity in the current environment. If the ambient light intensity is greater than or equal to the preset intensity threshold, then the video image data can be obtained by collecting through the CameraX API interface. Otherwise, if the ambient light intensity is less than the preset intensity threshold, that is, in a darker live broadcast scene, if the front camera is used at this time, the mobile phone screen backlight can be used for fill light; if the rear camera is switched, it can be determined whether the flashlight needs to be automatically turned on for fill light according to the ambient light intensity detected by the mobile phone sensor; and then the video image data, that is, the video data to be encoded, is collected through the CameraX API interface.

[0057] S102: Perform image processing on the video data to be encoded to obtain processed video data.

[0058] In the embodiment of the present application, the image format of the video data to be encoded is the YUV format. After obtaining the YUV video data, image processing can also be performed on it. In some embodiments, Figure 2 FIG. 2 shows a flow chart of another encoding method proposed in an embodiment of the present application. Figure 2 As shown, after determining the video data to be encoded, that is, after step S101, the method may further include:

[0059] S201: Determine a reference frame of video data to be encoded.

[0060] S202: Perform grayscale histogram statistics on the reference frame to determine the grayscale histogram of the reference frame.

[0061] It should be noted that the video data to be encoded may include at least one frame. Since it is time-consuming to perform grayscale histogram statistics for each frame, in order to save the amount of calculation, some frames may be selected for grayscale histogram statistics.

[0062] In a specific example, determining the reference frame of the video data to be encoded may include: performing frame skipping processing on the video data to be encoded at a preset interval to determine the reference frame. In other words, the grayscale histogram may be counted in a frame skipping manner. Exemplarily, the preset interval may be set to 10 frames, 20 frames, etc., but is not specifically limited thereto.

[0063] In addition, the grayscale histogram is a function of grayscale distribution, which is a statistic of grayscale distribution in an image. The grayscale histogram is a function of grayscale, which indicates the number of pixels with a certain grayscale in an image, and reflects the frequency of a certain grayscale in an image.

[0064] In an embodiment of the present application, in a darker live broadcast scene, night fill lighting is sometimes only effective at close distances; while for relatively far distances, the grayscale histogram can be statistically calculated by skipping frames (such as at intervals of 10 frames), and then the grayscale histogram can be used to determine whether the video data to be encoded needs image enhancement processing.

[0065] S203: Determine whether to perform image enhancement processing on the video data to be encoded according to the grayscale histogram of the reference frame.

[0066] In an embodiment of the present application, a preset ratio threshold may be set. If the pixels exceeding the preset ratio threshold are located in a low brightness area of ​​a reference frame, the video data to be encoded needs to be subjected to image enhancement processing. Specifically, in some embodiments, determining whether to perform image enhancement processing on the video data to be encoded based on a grayscale histogram of a reference frame may include:

[0067] Calculate the proportion of target pixels included in the reference frame in the reference frame according to the grayscale histogram of the reference frame;

[0068] If the proportion exceeds a preset proportion threshold, it is determined that the video data to be encoded is to be subjected to image enhancement processing;

[0069] If the proportion does not exceed the preset proportion threshold, it is determined that the video data to be encoded does not need to be subjected to image enhancement processing.

[0070] It should be noted that the preset percentage threshold is a measurement indicator for determining whether the video data to be encoded needs to be subjected to image enhancement processing. Exemplarily, the preset percentage threshold can be set to 50%, 70%, and so on. Taking 50% as an example, if the percentage exceeds 50%, it can be determined that the video data to be encoded needs to be subjected to image enhancement processing; if the percentage does not exceed 50%, it can be determined that the video data to be encoded does not need to be subjected to image enhancement processing.

[0071] It should also be noted that the target pixel may refer to a pixel located in a low brightness area. In a possible implementation, the method may further include: comparing the brightness value of each pixel in the reference frame with a preset brightness range; if the brightness value of the current pixel is within the preset brightness range, determining that the current pixel is the target pixel.

[0072] In other words, the preset brightness interval is a measure for determining whether the brightness value of each pixel in the reference frame is low brightness. If the brightness value of the current pixel is within the preset brightness interval, then the current pixel can be determined to be a target pixel, that is, the pixel belongs to the low brightness area; otherwise (the brightness value of the current pixel is outside the preset brightness interval), the current pixel is a non-target pixel, that is, the pixel does not belong to the low brightness area. In this way, when it is counted that most pixels belong to the low brightness area (that is, the proportion exceeds the preset proportion threshold), it is necessary to perform image enhancement processing on the video data to be encoded.

[0073] S204: If it is determined that the video data to be encoded needs to be subjected to image enhancement processing, the video data to be encoded needs to be subjected to image enhancement processing to obtain processed video data.

[0074] S205: If it is determined that the video data to be encoded does not undergo image enhancement processing, the video data to be encoded is determined as processed video data.

[0075] In a specific example, performing image enhancement processing on the video data to be encoded may include: performing image enhancement processing on the video data to be encoded using a preset image enhancement algorithm.

[0076] In an embodiment of the present application, the preset image enhancement algorithm may include a tone mapping algorithm, a local tone mapping algorithm, or an image enhancement algorithm based on deep learning, and even other traditional image enhancement algorithms may be selected, which are not specifically limited here.

[0077] It should be noted that image enhancement algorithms are commonly used to adjust the brightness, contrast, saturation, hue, etc. of an image, increase its clarity, reduce noise, etc. Image enhancement often completes the above functions through a combination of multiple algorithms. For example, image denoising is equivalent to a low-pass filter, and increasing clarity is a high-pass filter. Of course, enhancing an image is mainly to obtain useful information from the image. The general image enhancement algorithm process can be: image denoising, increasing clarity (contrast), graying, or obtaining image edge features, or convolution and binarization of the image, etc., to solve the problem of insufficient overall contrast or local darkening, overexposure, and other detail loss caused by the lighting environment.

[0078] Tone Mapping algorithm refers to the algorithm for transforming the color of an image. Its purpose is to adjust the grayscale of the image so that the processed image looks more comfortable and can better express the information and features in the original image. Tone Mapping algorithms can be roughly divided into two categories: global algorithms and local algorithms. Among them, the global algorithm refers to the method that uses the same mapping function for the entire image, such as gamma correction, which is a global tone mapping algorithm; the local algorithm refers to the method that the grayscale values ​​after mapping may be different due to the different locations of pixels in the entire image, and the mapping results of the pixels will be affected by other factors.

[0079] In an embodiment of the present application, when the video data to be encoded needs to be processed for image enhancement, a Tone Mapping algorithm based on gamma correction is usually used so that the brightness of the low-brightness area can be improved, and the originally bright area will not be overexposed.

[0080] S103: Determine a first QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data.

[0081] It should be noted that after obtaining the processed video data, the processed video data can be divided into regions of interest and regions of non-interest through region of interest (ROI) detection, that is, the foreground region and background region of the processed video data are determined, and then the first coding block and the second coding block are determined.

[0082] It can be understood that the region of interest is the foreground region of the processed video data, and the region of non-interest is the background region of the processed video data. Specifically, in some embodiments, Figure 3 FIG. 2 shows a flow chart of another encoding method proposed in an embodiment of the present application. Figure 3As shown, after obtaining the processed video data, that is, after step S102, the method may further include:

[0083] S301: Perform format conversion on the processed video data to obtain format-converted video data.

[0084] S302: Perform region of interest detection on the video data after format conversion.

[0085] S303: Determine the detected region of interest as a foreground region of the processed video data, and determine the non-region of interest except the region of interest as a background region of the processed video data.

[0086] In the embodiment of the present application, the image format of the processed video data is still in YUV format. Considering that the video data required for ROI detection is in RGB format, before performing ROI detection, each frame of the processed video data needs to be format converted so that the video data in RGB format can be obtained.

[0087] Exemplarily, in an embodiment of the present application, the video data mainly includes a video sequence in the YUV420 format. For the images in the video sequence, the video data in the YUV420 format can be first converted into video data in the YUV444 format through a bilinear interpolation algorithm, and then the video data in the YUV444 format can be converted into RGB888 video data through the following formula to obtain the video data after format conversion.

[0088]

[0089] In this way, ROI detection is performed on the video data after format conversion (i.e., video data in RGB format), and the detected ROI area can be determined as the foreground area of ​​the processed video data, and the non-ROI area except the ROI area can be determined as the background area of ​​the processed video data. Then step S304 can be executed: determining a first QP value of the first coding block corresponding to the foreground area and a second QP value of the second coding block corresponding to the background area.

[0090] In the embodiment of the present application, for the detection and extraction of the ROI area, the ROI area can be a regular area (such as a rectangular area) or an irregular area (such as a triangular area, a polygonal area, etc.), and the embodiment of the present application does not make any specific limitation.

[0091] It should be noted that the video data to be encoded may be one frame or multiple frames. That is, the video data to be encoded may include at least one frame, and accordingly, the processed video data may also include at least one frame. In the embodiment of the present application, for S103, determining the first QP value of the first coding block and the second QP value of the second coding block in the processed video data may specifically include: determining the first QP value of the first coding block and the second QP value of the second coding block in each frame of the processed video data.

[0092] Here, each frame may include a foreground area and a background area, that is, each frame has a first coding block and a second coding block and corresponding first QP values ​​and second QP values; specifically, the first coding block may be a coding block corresponding to the foreground area in each frame of the processed video data; the second coding block may be a coding block corresponding to the background area in each frame of the processed video data. In addition, the first coding block and the second coding block may also be referred to as macro blocks (MB).

[0093] Furthermore, in each frame of the processed video data, determining the first QP value and the second QP value may, in some embodiments, include:

[0094] Obtain a first QP initial value for the first coding block and a second QP initial value for the second coding block;

[0095] The first QP initial value and the second QP initial value are adjusted to obtain a first QP value of the first coding block and a second QP value of the second coding block.

[0096] In a specific example, the adjusting the first QP initial value and the second QP initial value to obtain the first QP value of the first coding block and the second QP value of the second coding block may include:

[0097] Determine a first QP compensation value of the first coding block according to the pixel value of the first coding block; and determine a second QP compensation value of the second coding block according to the pixel value of the second coding block;

[0098] Performing a sum operation on the first QP initial value and the first QP compensation value to obtain a first QP value of the first coding block;

[0099] A second QP initial value and the second QP compensation value are summed to obtain a second QP value for the second coding block.

[0100] It should be noted that, in the embodiment of the present application, the first QP compensation value can be used to adjust the first QP initial value of the first coding block corresponding to the foreground area, and the second QP compensation value can be used to adjust the second QP initial value of the second coding block corresponding to the background area.

[0101] It should also be noted that, in the embodiment of the present application, in order to improve the video quality of the ROI area during the decoding process at the same bit rate or a low bit rate, it is not only necessary to distinguish and encode the foreground area and the background area according to different bit rates, but also necessary to increase the bit rate corresponding to the foreground area as much as possible. Therefore, it is necessary to adjust the first QP initial value of the first coding block corresponding to the foreground area and the second QP initial value of the second coding block corresponding to the background area respectively.

[0102] Specifically, first, different QP compensation values ​​may be determined for the foreground area and the background area, and then the first QP initial value and the second QP initial value may be adjusted using the different QP compensation values.

[0103] Furthermore, different calculation models may be used to calculate the QP compensation value for the foreground area and the background area, so that different QP compensation values ​​may be determined for different areas.

[0104] In a possible implementation, when calculating a first QP compensation value of a first coding block and a second QP compensation value of a second coding block, the first QP compensation value may be calculated based on a preset QP reference value, a first calculation model, and a pixel value of the first coding block; at the same time, the second QP compensation value may also be calculated based on the preset QP reference value, a second calculation model, and a pixel value of the second coding block.

[0105] It can be understood that the first calculation model is used to determine the QP compensation value corresponding to the foreground area. Specifically, the first calculation model is used to reduce the initial QP value of the foreground area.

[0106] Exemplarily, the first calculation model can be expressed as the following formula, and the first QP compensation value (expressed as qp_val1) corresponding to the foreground area can be calculated using the first calculation model:

[0107] qp_val1=pix_val1 / 255×Δqp (2)

[0108] Wherein, Δqp is a preset QP reference value, and pix_val1 is a pixel value of the first coding block.

[0109] It can also be understood that the second calculation model is used to determine the QP compensation value corresponding to the background area. Specifically, the second calculation model is used to increase the initial QP value of the background area.

[0110] Exemplarily, the second calculation model can be expressed as the following formula, and the second QP compensation value (expressed as qp_val2) corresponding to the background area can be calculated using the second calculation model:

[0111] qp_val2=(255-pix_val2) / 255×(-Δqp) (3)

[0112] Wherein, Δqp is a preset QP reference value, and pix_val2 is a pixel value of the second coding block.

[0113] It should be noted that the preset QP reference value can be any value pre-set by the terminal device for calculating the QP compensation value. The terminal device can set the preset QP reference value to a fixed value, or use an algorithm to analyze and classify the video scene, and then automatically set parameters for different scenes, that is, different scenes correspond to different preset QP reference values.

[0114] Exemplarily, in an embodiment of the present application, the terminal device may set the preset QP reference value Δqp to -15. If the first QP compensation value qp-val1 calculated by the above formula (2) is a negative number, then the first QP initial value is adjusted by the first QP compensation value, so that the first QP value of the first coding block can be reduced. Correspondingly, if the second QP compensation value qp-val2 calculated by the above formula (3) is a positive number, then the second QP initial value is adjusted by the second QP compensation value, so that the second QP value of the second coding block can be increased.

[0115] In short, the video encoding process is carried out at the code native service layer. YUV video data is commonly used for encoding, and the encoding scheme adopts x264 software encoding and adds ROI encoding function to adjust the QP value corresponding to the ROI area. Among them, the ROI area can be manually set as a coordinate rectangular area, or the ROI area in the scene can be detected and extracted by a computer vision algorithm. In addition, ROI encoding mainly reduces the first QP value corresponding to the macroblock in the ROI area and increases the second QP value of the macroblock in the background area, so that a higher bit rate is allocated to the ROI area during the encoding process, and a lower bit rate is allocated to the background area, so that the video quality of the ROI area of ​​the decoded video can be guaranteed during the encoding process under the same bit rate or lower bit rate.

[0116] S104: Encode the processed video data according to the first QP value and the second QP value to generate encoded data.

[0117] In the embodiment of the present application, the processed video data may include at least one frame. For S104, encoding the processed video data according to the first QP value and the second QP value to generate encoded data may specifically include: encoding each frame in the processed video data according to the first QP value and the second QP value of each frame to generate encoded data; wherein the encoded data may include frame encoded data corresponding to each frame.

[0118] It should be noted that, in each frame of the processed video data, for the foreground area, the first QP initial values ​​corresponding to different first coding blocks may be the same or different; correspondingly, for the background area, the second QP initial values ​​corresponding to different second coding blocks may be the same or different.

[0119] It should also be noted that after the first QP compensation value is calculated according to the first calculation model for reducing the initial QP value of the foreground area, and the second QP compensation value is calculated according to the second calculation model for increasing the initial QP value of the background area, the first QP initial value of the foreground area and the second QP initial value of the background area can be adjusted according to the first QP compensation value and the second QP compensation value, respectively, so as to obtain a reduced first QP value and an increased second QP value.

[0120] Furthermore, in an embodiment of the present application, the first QP value of the first coding block corresponding to the foreground area is different from the second QP value of the second coding block corresponding to the background area. For the foreground area, the first QP values ​​corresponding to different first coding blocks may be the same or different; correspondingly, for the background area, the second QP values ​​corresponding to different second coding blocks may be the same or different. In addition, it should be noted that for the division of the foreground area and the background area, the first QP value of the first coding block is less than the second QP value of the second coding block; in this way, when the bit rate is fixed, a higher bit rate can be allocated to the foreground area, so that the video quality of the ROI area of ​​the decoded video can be guaranteed during the encoding process under the same bit rate or lower bit rate.

[0121] That is to say, when encoding according to the first QP value and the second QP value, since the smaller the QP value, the higher the allocated bit rate, and the larger the QP value, the lower the allocated bit rate, therefore for the processed video data, encoding the first coding block and the second coding block according to the first QP value and the second QP value respectively can greatly improve the encoding quality of the foreground area, thereby improving the video quality of the ROI area during the decoding process.

[0122] S105: Encapsulate the encoded data, generate a transmission video packet, and send it.

[0123] It should be noted that the Real Time Messaging Protocol (RTMP) is a commonly used protocol for live streaming. The protocol is based on the Transmission Control Protocol (TCP) and is a protocol family that includes the RTMP basic protocol and multiple variants such as RTMPT / RTMPS / RTMPE. Among them, RTMP is a network protocol designed for real-time data communication, mainly used for audio, video and data communication between animation (Flash) / cross-operating system runtime (Adobe Integrated Runtime, AIR) platform and streaming media / interactive servers that support the RTMP protocol.

[0124] It should also be noted that RTMP live streaming is currently the mainstream streaming protocol. Since terminal devices are performing live streaming, it is particularly important to create a live streaming application software (Application, APP) that can save traffic. In the embodiment of the present application, the encoded video data can be packaged into a transmission video packet in the RTMP protocol format with sequence parameter set (Sequence Parameter Set, SPS), picture parameter set (Picture Parameter Set, PPS) and other data for transmission.

[0125] It is understood that in some embodiments, Figure 4 FIG. 2 shows a flow chart of another encoding method proposed in an embodiment of the present application. Figure 4 As shown, for S105, the method may further include:

[0126] S401: Obtain SPS data and PPS data corresponding to each frame in the encoded data.

[0127] In an embodiment of the present application, for SPS, a variety of different NAL Unit types are specified in the H.264 standard protocol, among which type 7 indicates that the data stored in the NAL Unit is SPS. Among the various syntax elements of H.264, the information in SPS is crucial, which stores a set of global parameters of the encoded video sequence. The so-called encoded video sequence is a sequence of structures composed of the pixel data of the original video frame by frame after encoding, and the parameters on which the encoded data of each frame depends are stored in the picture parameter set (PPS). For PPS, it is another important parameter set in the H.264 standard protocol. Normally, PPS is similar to SPS, and is stored separately in a NAL Unit in the raw code stream of H.264, except that the nal_unit_type value of PPS NAL Unit is 8; in the encapsulation format, PPS is usually stored in the file header of the video file together with SPS.

[0128] See Table 1, which shows an example of a package structure of SPS data and PPS data areas provided in an embodiment of the present application.

[0129] Table 1

[0130]

[0131] S402: Pack the coded data and the SPS data and PPS data corresponding to each frame to generate a transmission video packet.

[0132] It should be noted that the embodiment of the present application adopts the H.264 standard protocol for encoding, so the encoded data can also be called H264 data; in addition, the embodiment of the present application adopts the RTMP live streaming protocol, so the transmission video packet here is the RTMP video packet.

[0133] See Table 2, which shows a structural example of encapsulating H264 data into RTMP video packets provided by an embodiment of the present application.

[0134] Table 2

[0135] Keyframes 0x17 0x01 0x00 0x00 0x00 4-byte data length H264 raw data Non-keyframe 0x27 0x01 0x00 0x00 0x00 4-byte data length H264 raw data SPS vs PPS 0x17 0x00 0x00 0x00 0x00 SPS+PPS data

[0136] It should also be noted that video uses the principle of human visual persistence to give the human eye a sense of motion by playing a series of images. In video compression, each frame represents a still image. Simply put, an I frame is a key frame and belongs to intra-frame compression; it can be understood as the complete retention of this frame of the picture, and only the data of this frame is needed to complete the decoding (because it contains the complete picture). A P frame is a forward prediction frame, which specifically represents the difference between this frame and the previous key frame (or P frame). When decoding, the difference defined by this frame needs to be superimposed with the previously cached picture to generate the final picture. A B frame is a bidirectional prediction frame, that is, a B frame records the difference between this frame and the previous and next frames. In other words, to decode a B frame, not only the previous cached picture must be obtained, but also the picture after decoding, and the final picture is obtained by superimposing the previous and next pictures with the data of this frame.

[0137] In the embodiment of the present application, as shown in Table 2, the first bytes of key frames (I frames) and non-key frames (B frames and P frames) are different, the first byte of key frames is 0x17, and the first byte of non-key frames is 0x27. In addition, conventional H264 encoding only needs to store an SPS and PPS in front of the encoded data, as shown in Table 1. However, since there is no complete video sequence in the RTMP push stream process, it may be necessary to push the stream after collecting one frame. Unlike a complete video sequence, the width and height of the video frame may change due to the rotation of the camera, so SPS data and PPS data need to be added in front of each frame of video encoding data. In addition to the key frame and non-key frame information, configuration information and the length information of each frame of H264 data can also be added before each frame of encoded code stream data.

[0138] It should also be pointed out that in a video sequence, multiple images can be grouped into a group of pictures (GOP). The video data to be encoded described in the embodiment of the present application can be regarded as a GOP, which specifically refers to the distance between two I frames, that is, a group of images is formed between two I frames, and the group of images is the video data to be encoded.

[0139] In this way, for S402, after obtaining the SPS data and PPS data corresponding to each frame, the encoded data and the SPS data and PPS data corresponding to each frame are packaged to generate a transmission video packet, which can specifically include: adding the SPS data and PPS data corresponding to each frame before the frame encoding data of each frame and packaging them to generate a transmission video packet.

[0140] S403: Send the transmission video packet in a preset protocol format.

[0141] It should be noted that in the embodiment of the present application, the preset protocol format can be the RTMP_Send Packet() format. In addition, the transmission video packet is pushed to the RTMP server. In the embodiment of the present application, the RTMP server can use the B station server (by obtaining the RTMP address and personal live broadcast code in the personal live broadcast room of the B station, and combining them to obtain the personal RTMP live broadcast address).

[0142] Based on the encoding method described in the embodiment of the present application, the live broadcast effect is as follows: Figure 5A and Figure 5B As shown. Among them, Figure 5A The following is a schematic diagram of the interface of the terminal device APP streaming end. Figure 5B A schematic diagram of the display interface of the B station live broadcast server is shown.

[0143] In summary, the live streaming of the embodiment of the present application adopts a software encoding solution. The improvement of the CPU performance of the terminal device creates conditions for x264 software encoding, and software encoding and decoding can become a solution to replace hardware encoding and decoding on the terminal device side. Although the APP in the embodiment of the present application currently only adds H.264 encoding, encoding frameworks such as H.265, VP8, and VP9 can also be added, so that software encoding can also have as many encoding schemes as hardware encoding.

[0144] Furthermore, the embodiment of the present application adopts the live streaming server of Station B, but you can also build your own server for live streaming, and the image enhancement algorithm can be transplanted to the server side, thereby reducing the power consumption problem introduced by the algorithm on the terminal device side.

[0145] Furthermore, there may be a power consumption problem when obtaining the ROI area through an algorithm. The embodiment of the present application may also improve the algorithm selection and algorithm optimization to make the algorithm faster and more power-saving.

[0146] The embodiment of the present application provides a coding method, by determining the video data to be coded, and performing image processing on the video data to be coded, to obtain the processed video data; determining the first quantization parameter QP value of the first coding block and the second QP value of the second coding block in the processed video data; wherein the first coding block is the coding block corresponding to the foreground area in the processed video data, and the second coding block is the coding block corresponding to the background area in the processed video data; encoding the processed video data according to the first QP value and the second QP value to generate the coded data; encapsulating the coded data, generating the transmission video packet and sending it. In this way, by introducing the software coding scheme, the coding quality and compatibility can be improved, and the phenomenon that the coding quality problem caused by the integration of hardware coding can only be solved by relying on chip suppliers can be avoided; in addition, by introducing ROI coding, that is, reducing the QP value of the foreground area and increasing the QP value of the background area, a higher bit rate can be allocated to the foreground area and a lower bit rate can be allocated to the background area during the coding process, so that under the same bit rate or lower bit rate, not only the video quality of the ROI area can be improved, but also the network bandwidth can be saved, and the purpose of saving data traffic when the user adopts data network live broadcast is achieved.

[0147] In another embodiment of the present application, see Figure 6 , which shows a detailed flow chart of a coding method provided in an embodiment of the present application. The terminal device takes a smart phone as an example, such as Figure 6 As shown, the detailed process may include:

[0148] S601: Sense the ambient light intensity in the current environment.

[0149] S602: Whether to perform ambient fill light.

[0150] S603: Increase the backlight of the mobile phone screen and / or turn on the flashlight.

[0151] S604: The mobile phone camera collects data.

[0152] S605: Obtain YUV video data.

[0153] It should be noted that, for S602, if ambient fill light is required, that is, the judgment result is yes, then S603 needs to be executed to perform ambient fill light by increasing the mobile phone screen backlight and / or turning on the flashlight, and then obtain YUV video data based on the data collected by the mobile phone camera; if ambient fill light is not required, that is, the judgment result is no, then the YUV video data can be directly obtained based on the data collected by the mobile phone camera.

[0154] S606: Calculate grayscale histogram.

[0155] S607: Whether to perform image enhancement.

[0156] S608: Perform image enhancement based on a tone mapping algorithm.

[0157] S609: Perform x264-ROI encoding.

[0158] S610: Obtain H264 data.

[0159] It should be noted that, for S607, if image enhancement is required, that is, the judgment result is yes, then S608 needs to be executed, image enhancement is performed through the Tone Mapping algorithm, and then S609 and S610 are executed; if image enhancement is not required, that is, the judgment result is no, then S609 and S610 can be directly executed to obtain H264 data.

[0160] S611: Obtain RTMP video package.

[0161] S612: Send according to RTMP_SendPacket().

[0162] In the embodiment of the present application, the specific process of the live streaming is as follows: Figure 6 As shown. First, data is collected through the mobile phone camera, and ambient light is filled in according to the ambient light intensity perception. Then, the grayscale histogram of the video image is counted by skipping frames, and the scene that needs image enhancement is performed; then the video data is compressed and encoded to obtain H264 data. Among them, x264-ROI encoding adds ROI encoding function on the basis of x264 encoding, and the ROI area can be obtained by manual setting or computer vision algorithm. Furthermore, SPS, PPS and other data are added to the encoded H264 data to be packaged into an RTMP video packet, and the RTMP video packet is pushed to the RTMP server through RTMP_SendPacket(), wherein the embodiment of the present application adopts the B station live streaming server.

[0163] for Figure 6 The detailed process shown in the figure is as follows:

[0164] (1) Mobile phone camera data collection

[0165] The data collection process calls the CameraX API interface at the JAVA layer. Specifically, on the mobile phone side, the mobile phone sensor can be called to sense the ambient light intensity in the current environment. In a darker live broadcast scene, when using the front camera, the mobile phone screen backlight can be used for fill light. When switching to the rear camera, it can determine whether to automatically turn on the flashlight for fill light based on the ambient light intensity; finally, the YUV video data is collected through the CameraX API interface.

[0166] (2) Image enhancement of YUV video data

[0167] Fill-in lighting at night is sometimes only effective at close range. At relatively far distances, you can skip frames to count the image grayscale histogram (since statistics for each frame are time-consuming, 10 frames can be used as an interval), and then set a threshold (such as more than 50%). When the pixels are located in the low-brightness area, it is determined that the YUV video data needs to be enhanced.

[0168] Here, the image enhancement can use the Tone Mapping algorithm based on gamma correction, so that the low-brightness area can be brightened and the originally bright area will not be overexposed. In addition, the image enhancement can also choose other traditional image enhancement algorithms or video image enhancement algorithms based on deep learning, which are not specifically limited in the embodiments of this application.

[0169] (3) ROI intelligent compression encoding of YUV video data

[0170] The encoding process is carried out at the Native layer. Common YUV video data is used for encoding. This encoding scheme uses x264 software encoding and adds ROI encoding function to adjust the QP value corresponding to the ROI area macroblock. Among them, the ROI area can be manually set as a coordinate rectangular area, or the ROI area in the scene can be detected and extracted by a computer vision algorithm. It should be pointed out that ROI encoding mainly reduces the QP value corresponding to the ROI area macroblock and increases the QP value of the background area macroblock, so that a higher bit rate is allocated to the ROI area during the encoding process, and a lower bit rate is allocated to the background area, so that the ROI area of ​​the decoded video can be guaranteed to be clear during the encoding process under the same bit rate or lower bit rate.

[0171] (4) Encapsulate H264 data into RTMP video packets

[0172] According to the above Tables 1 and 2, the first bytes of key frames (I frames) and non-key frames (B frames, P frames) are different. The first byte of key frames is 0x17, and the first byte of non-key frames is 0x27. In addition, conventional H264 encoding only needs to store an SPS and PPS in front of the encoded data, but because there is no complete video sequence during the RTMP push stream, it may be necessary to push the stream after collecting one frame. Unlike a complete video sequence, the width and height of the video frame may change due to the rotation of the camera, so SPS and PPS and other data need to be added in front of each frame of video encoding data. In addition to the key frame and non-key frame information, configuration information and the length information of each frame of H264 data need to be added before each frame of encoded code stream data.

[0173] (5) Push the packaged data to the RTMP server via RTMP_SendPacket()

[0174] It should be noted that the obtained RTMP video packet is pushed to the RTMP server through RTMP_SendPacket(). Among them, the embodiment of the present application can use the B station server (by obtaining the RTMP address and personal live broadcast code in the personal live broadcast room of B station, and combining them to obtain the personal RTMP live broadcast address).

[0175] It should also be noted that, for RTMP video packets, taking into account performance such as network bandwidth, the embodiment of the present application can also segment the RTMP video packets and then send the segmented small packets to the RTMP server one by one.

[0176] In summary, smartphones usually use DSP hardware codec solutions. With the improvement of CPU performance, mobile phones can also support software codecs, which have better compatibility than hardware codecs. H.264 standard protocol is the current mainstream video coding solution, x264 is the application version of H.264, and x264-ROI coding is to add ROI coding scheme to the x264 coding framework, which can improve the video quality of ROI area at a relatively low bit rate. At present, the live video industry is hot. With the improvement of mobile phone pixels and video quality, mobile phone live broadcast has become a trend. RTMP is a common protocol for live streaming. The embodiment of the present application can perform video image enhancement encoding on the video data collected by the mobile phone according to the scene and then transmit it through the protocol. In addition, in the process of transmitting media data, the sending end (i.e., the mobile phone end) can first encapsulate the media data into messages, then divide the messages into message blocks, and finally send the divided message blocks through the TCP protocol.

[0177] In the embodiment of the present application, for the live broadcast scene, in the dark night, the CPU can call the flashlight or increase the backlight of the mobile phone screen to perform fill light. The grayscale histogram of the scene image after fill light is counted every other frame. When most of the pixels fall in the low brightness range, the YUV video data needs to perform the image enhancement step of the Tone Mapping algorithm.

[0178] Furthermore, in the embodiment of the present application, mobile phone video software encoding is not widely used at present, and the introduction of software encoding will have better compatibility than hardware, and the encoding quality is reliable. In addition, developers can also modify the software encoding framework, while the encoding quality problems caused by the integration of hardware encoding make users rely on chip suppliers to solve them, so that the efficiency of software encoding development can be relatively improved.

[0179] Furthermore, in the embodiment of the present application, by adding the ROI encoding function to the encoding framework, the developer can set the ROI area manually or by algorithm (ROI encoding can also be performed on irregular areas on the mobile phone side). Thus, the QP value of the macroblock corresponding to the foreground area and the background area is adjusted to achieve a higher bit rate allocation to the ROI area.

[0180] Thus, the embodiment of the present application provides a coding method, and the specific implementation of the aforementioned embodiment is elaborated in detail through the above embodiment. It can be seen that based on the technical solution of mobile phone environment perception x264-ROI coding and RTMP live streaming provided by the embodiment of the present application, on the one hand, the live streaming of the embodiment of the present application adopts software coding; the improvement of the CPU performance of the mobile phone creates conditions for x264 software coding, and software coding and decoding can become a solution to replace hardware coding and decoding on the mobile phone. On the other hand, since many RTMP live servers do not have a video enhancement processing process, the brightness of the live scene can be improved by ambient fill light in the dark night, which can also improve the live effect. On the other hand, the embodiment of the present application introduces ROI coding to save network bandwidth, thereby saving data traffic when users use data network live broadcast; in addition, some mobile phone chips can also support ROI coding, but the ROI area can only be a regular rectangular area, and the corresponding irregular area is not currently supported, and the software coding provided by the embodiment of the present application can formulate ROI adjustment rules by itself. On the other hand, RTMP live streaming is a mainstream streaming protocol, and live streaming on mobile phones has become a trend. The mobile live streaming solution provided in the embodiment of the present application not only saves data traffic, but also can achieve the purpose of improving the live streaming effect.

[0181] In another embodiment of the present application, see Figure 7 , which shows a schematic diagram of the composition structure of an encoding device 70 provided in an embodiment of the present application. Figure 7 As shown, the encoding device 70 may include: a determining unit 701, a processing unit 702 and an encoding unit 703; wherein,

[0182] The determination unit 701 is configured to determine the video data to be encoded;

[0183] The processing unit 702 is configured to perform image processing on the video data to be encoded to obtain processed video data;

[0184] The determination unit 701 is further configured to determine a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data;

[0185] an encoding unit 703, configured to encode the processed video data according to the first QP value and the second QP value to generate encoded data;

[0186] The processing unit 702 is further configured to encapsulate the encoded data, generate a transmission video packet, and send the packet.

[0187] In some embodiments, see Figure 7 , the encoding device 70 may further include a collection unit 704; wherein,

[0188] The processing unit 702 is further configured to detect the ambient light intensity in the current environment; and to fill in the current environment with light when the ambient light intensity is less than a preset intensity threshold;

[0189] The acquisition unit 704 is configured to acquire data through a camera according to the environment after fill-lighting to obtain the video data to be encoded.

[0190] In some embodiments, the processing unit 702 is further configured to provide fill light through the screen of the terminal device if the camera is a front camera of the terminal device; or, if the camera is a rear camera of the terminal device, provide fill light through the flashlight of the terminal device.

[0191] In some embodiments, the determination unit 701 is further configured to determine a reference frame of the video data to be encoded;

[0192] The processing unit 702 is further configured to perform grayscale histogram statistics on the reference frame to determine the grayscale histogram of the reference frame; and determine whether the to-be-encoded video data is to be subjected to image enhancement processing according to the grayscale histogram of the reference frame;

[0193] The determination unit 701 is further configured to, if it is determined that the video data to be encoded is to be subjected to image enhancement processing, perform image enhancement processing on the video data to be encoded to obtain the processed video data; or, if it is determined that the video data to be encoded is not to be subjected to image enhancement processing, determine the video data to be encoded as the processed video data.

[0194] In some embodiments, the processing unit 702 is further configured to perform frame skipping processing on the to-be-encoded video data at a preset interval to determine the reference frame.

[0195] In some embodiments, the processing unit 702 is further configured to calculate the proportion of the target pixels included in the reference frame in the reference frame according to the grayscale histogram of the reference frame;

[0196] The determination unit 701 is further configured to determine that the video data to be encoded is to be subjected to image enhancement processing if the proportion exceeds a preset proportion threshold; or, if the proportion does not exceed the preset proportion threshold, determine that the video data to be encoded is not to be subjected to image enhancement processing.

[0197] In some embodiments, the processing unit 702 is further configured to compare the brightness value of each pixel in the reference frame with a preset brightness range;

[0198] The determination unit 701 is further configured to determine that the current pixel is the target pixel if the brightness value of the current pixel is within the preset brightness range.

[0199] In some embodiments, the processing unit 702 is further configured to perform image enhancement processing on the video data to be encoded using a preset image enhancement algorithm; wherein the preset image enhancement algorithm includes a Tone Mapping algorithm, a LocalTone Mapping algorithm, or an image enhancement algorithm based on deep learning.

[0200] In some embodiments, the processing unit 702 is further configured to perform format conversion on the processed video data to obtain format-converted video data; and perform region of interest detection on the format-converted video data;

[0201] The determination unit 701 is further configured to determine the detected region of interest as a foreground region of the processed video data, and determine a non-region of interest except the region of interest as a background region of the processed video data.

[0202] In some embodiments, the processing unit 702 is further configured to obtain a first QP initial value of the first coding block and a second QP initial value of the second coding block; and adjust the first QP initial value and the second QP initial value to obtain a first QP value of the first coding block and a second QP value of the second coding block.

[0203] In some embodiments, the determining unit 701 is further configured to determine a first QP compensation value of the first coding block according to the pixel value of the first coding block; and determine a second QP compensation value of the second coding block according to the pixel value of the second coding block;

[0204] The processing unit 702 is further configured to sum the first QP initial value and the first QP compensation value to obtain a first QP value of the first coding block; and sum the second QP initial value and the second QP compensation value to obtain a second QP value of the second coding block.

[0205] In some embodiments, the processing unit 702 is also configured to obtain sequence parameter set SPS data and image parameter set PPS data corresponding to each frame in the encoded data; package the encoded data and the SPS data and the PPS data corresponding to each frame to generate the transmission video packet; and send the transmission video packet in a preset protocol format.

[0206] In some embodiments, the preset protocol format is RTMP_SendPacket() format.

[0207] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0208] It should also be noted that in the embodiments of the present application Figure 7 The division of modules in the coding device shown is schematic, which is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0209] In the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product, which is stored in a storage medium and includes several instructions for enabling the terminal device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store computer programs or executable instructions, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk.

[0210] Therefore, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0211] In yet another embodiment of the present application, based on the composition of the aforementioned encoding device 70 and the computer-readable storage medium, Figure 8A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application. Figure 8 As shown, the terminal device 80 may include a processor 801, and the processor 801 may call and run a computer program from a memory to implement the steps of any one of the methods in the aforementioned embodiments.

[0212] Alternatively, if Figure 8 As shown, the terminal device 80 may further include a memory 802. The memory 802 is used to store a computer program that can be run on the processor 801, so that the processor 801 can call and run the computer program from the memory 802 to implement the steps of any one of the methods in the foregoing embodiments.

[0213] In the embodiment of the present application, the memory 802 may be a separate device independent of the processor 801 , or may be integrated into the processor 801 .

[0214] Alternatively, if Figure 8 As shown, the terminal device 80 may further include a communication interface 803 , and a data bus 804 for connecting the processor 801 , the memory 802 , and the communication interface 803 .

[0215] Optionally, the terminal device 80 may be a device such as a smart phone, a tablet computer, a PDA, a laptop computer, a desktop computer, or the like described in the foregoing embodiments, or a device integrated with the encoding device 70 described in any of the foregoing embodiments. Here, the terminal device 80 may implement the corresponding processes described in the various methods of the embodiments of the present application, which will not be described in detail for the sake of brevity.

[0216] In yet another embodiment of the present application, based on the composition of the aforementioned encoding device 70 and the computer-readable storage medium, Fig. 9 The hardware structure diagram of an encoder provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the encoder 90 may include a processor 901, and the processor 901 may call and run a computer program from a memory to implement the steps of any one of the methods in the aforementioned embodiments.

[0217] Alternatively, if Fig. 9 As shown, the encoder 90 may further include a memory 902. The memory 902 is used to store a computer program that can be run on the processor 901, so that the processor 901 can call and run the computer program from the memory 902 to implement the steps of any one of the methods in the foregoing embodiments.

[0218] In the embodiment of the present application, the memory 902 may be a separate device independent of the processor 901 , or may be integrated into the processor 901 .

[0219] Alternatively, if Fig. 9 As shown, the encoder 90 may further include a communication interface 903 , and a data bus 904 for connecting the processor 901 , the memory 902 , and the communication interface 903 .

[0220] Optionally, the encoder 90 may be a device integrated with the encoding device 70 described in any of the above embodiments. Here, the encoder 90 may also implement the corresponding processes described in the various methods of the embodiments of the present application, which will not be described in detail for the sake of brevity.

[0221] It should be noted that the processor of the aforementioned embodiment may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0222] It should also be noted that the memory in the aforementioned embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that the memory of the systems and methods described in the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0223] It is understandable that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units or their combinations for performing functions described in the present application.For software implementation, the technology described in the present application can be realized by the module (such as process, function, etc.) performing the functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in a processor or outside a processor.

[0224] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0225] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0226] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0227] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0228] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0229] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0230] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0231] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A coding method, characterized in that: The method comprises: Determining video data to be encoded, and performing image processing on the video data to be encoded to obtain processed video data; Determine a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data; Encoding the processed video data according to the first QP value and the second QP value to generate encoded data; Encapsulating the encoded data to generate a transmission video packet and sending the packet; The performing image processing on the video data to be encoded to obtain processed video data includes: determining a reference frame of the video data to be encoded, wherein the reference frame is a partial frame in the video data to be encoded; performing grayscale histogram statistics on the reference frame to determine a grayscale histogram of the reference frame; calculating, according to the grayscale histogram of the reference frame, a proportion of target pixels included in the reference frame in the reference frame, wherein the target pixels include pixels in the reference frame whose brightness values ​​are within a preset brightness range; and determining, according to the proportion, whether to perform image enhancement processing on the video data to be encoded to obtain the processed video data; The encapsulation processing of the encoded data to generate and send a transmission video packet includes: obtaining sequence parameter set SPS data and image parameter set PPS data corresponding to each frame in the encoded data; packaging the encoded data and the SPS data and the PPS data corresponding to each frame to generate the transmission video packet; and sending the transmission video packet in a preset protocol format.

2. The method according to claim 1, characterized in that The step of determining the video data to be encoded comprises: Detect the ambient light intensity in the current environment; When the ambient light intensity is less than a preset intensity threshold, supplementing the current environment with light; According to the environment after fill-in light, data is collected through a camera to obtain the video data to be encoded.

3. The method according to claim 2, characterized in that The step of supplementing the current environment with light includes: If the camera is a front camera of the terminal device, fill light is provided through the screen of the terminal device; or, If the camera is a rear camera of a terminal device, fill light is provided by a flashlight of the terminal device.

4. The method according to claim 1, characterized in that: The method further comprises: If it is determined that the video data to be encoded is to be subjected to image enhancement processing, image enhancement processing is performed on the video data to be encoded to obtain the processed video data; If it is determined that the to-be-encoded video data does not undergo image enhancement processing, the to-be-encoded video data is determined as the processed video data.

5. The method according to claim 1, characterized in that The determining a reference frame of the video data to be encoded comprises: The frame skipping process is performed on the video data to be encoded at a preset interval to determine the reference frame.

6. The method according to claim 1, characterized in that The determining, according to the proportion, whether to perform image enhancement processing on the video data to be encoded includes: If the proportion exceeds a preset proportion threshold, determining that the to-be-encoded video data is to be subjected to image enhancement processing; If the proportion does not exceed the preset proportion threshold, it is determined that the to-be-encoded video data does not need to be subjected to image enhancement processing.

7. The method according to claim 6, characterized in that Before calculating the proportion of the target pixels included in the reference frame in the reference frame, the method further includes: Comparing the brightness value of each pixel in the reference frame with a preset brightness range; If the brightness value of the current pixel is within the preset brightness range, the current pixel is determined to be the target pixel.

8. The method according to claim 4, characterized in that The performing image enhancement processing on the video data to be encoded comprises: The video data to be encoded is subjected to image enhancement processing using a preset image enhancement algorithm; wherein the preset image enhancement algorithm includes a tone mapping algorithm, a local tone mapping algorithm, or an image enhancement algorithm based on deep learning.

9. The method according to claim 1, characterized in that: Before determining a first QP value of a first coding block and a second QP value of a second coding block in the processed video data, the method further includes: Performing format conversion on the processed video data to obtain format-converted video data; Performing region of interest detection on the video data after the format conversion; The detected region of interest is determined as a foreground region of the processed video data, and a non-region of interest other than the region of interest is determined as a background region of the processed video data.

10. The method according to claim 1, characterized in that Determining a first QP value of a first coding block and a second QP value of a second coding block in the processed video data includes: Obtaining a first QP initial value of the first coding block and a second QP initial value of the second coding block; The first QP initial value and the second QP initial value are adjusted to obtain a first QP value of the first coding block and a second QP value of the second coding block.

11. The method according to claim 10, characterized in that The adjusting the first QP initial value and the second QP initial value to obtain a first QP value of the first coding block and a second QP value of the second coding block includes: Determine a first QP compensation value of the first coding block according to the pixel value of the first coding block; and determine a second QP compensation value of the second coding block according to the pixel value of the second coding block; Performing a sum operation on the first QP initial value and the first QP compensation value to obtain a first QP value of the first coding block; A sum operation is performed on the second QP initial value and the second QP compensation value to obtain a second QP value of the second coding block.

12. The method according to any one of claims 1 to 11, characterized in that: The preset protocol format is the real-time messaging protocol RTMP_SendPacket() format.

13. A coding device, characterized in that: The encoding device comprises a determining unit, a processing unit and an encoding unit; wherein, The determining unit is configured to determine the video data to be encoded; The processing unit is configured to perform image processing on the video data to be encoded to obtain processed video data; The determining unit is further configured to determine a first quantization parameter QP value of a first coding block and a second QP value of a second coding block in the processed video data; wherein the first coding block is a coding block corresponding to a foreground area in the processed video data, and the second coding block is a coding block corresponding to a background area in the processed video data; The encoding unit is configured to encode the processed video data according to the first QP value and the second QP value to generate encoded data; The processing unit is further configured to perform encapsulation processing on the encoded data, generate a transmission video packet and send it; Wherein, the determining unit is further configured to determine a reference frame of the video data to be encoded, wherein the reference frame is a partial frame in the video data to be encoded; The processing unit is further configured to perform grayscale histogram statistics on the reference frame to determine the grayscale histogram of the reference frame; calculate the proportion of target pixels included in the reference frame in the reference frame according to the grayscale histogram of the reference frame, wherein the target pixels include pixels whose brightness values ​​in the reference frame are within a preset brightness range; and determine whether the to-be-encoded video data is subjected to image enhancement processing according to the proportion to obtain the processed video data; The processing unit is also configured to obtain the SPS data and PPS data corresponding to each frame in the encoded data; package the encoded data and the SPS data and PPS data corresponding to each frame to generate the transmission video packet; and send the transmission video packet in a preset protocol format.

14. An encoder, characterized in that: The encoder includes a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 12 when running the computer program.

15. A terminal device, characterized in that: The terminal device includes a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 12 when running the computer program.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images