Video frame processing method and device, equipment, medium and product

Abstract

The invention relates to the technical field of data processing, in particular to a video frame processing method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a first effective write-in frequency of a pixel point of a current video frame, and recognizing the difference degree between the current video frame and a previous frame; whether the current video frame is the key video frame is judged according to the first effective write-in frequency, and the frame interval obtained by the next first effective write-in frequency is dynamically adjusted, so that the video frame processing efficiency is improved; when the bus state is confirmed to be busy and the current frame is not the key frame, the processing of the current video frame is stopped, so that the problem that the key frame may be lost due to insufficient resources because the non-key frame occupies bandwidth resources when the bus resources are insufficient is effectively avoided; and for a non-busy state or a key video frame, image compression processing is performed and the image is written into an on-chip cache, so that the bandwidth can be saved, and timely transmission of the key frame can be ensured, thereby remarkably improving the display quality of a remote end.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a video frame processing method, device, equipment, medium and product. Background Art

[0002] Generally, the video data is processed by the video processing function on the management control chip. The processing flow of the video function in the server management control chip is as follows: the original video data on the host side is transmitted to the VGA (Video Graphics Array) module inside the server management control chip through PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard). The VGA generates original video data in RGB format, and then undergoes color space conversion to generate video data in YUV format. The YUV data is converted into BLOCKs and then compressed by the video core (H.264 format, JPEG format, etc.) to obtain compressed video data. The compressed data is written to the external DDR (Double Data Rate) through the output control module.

[0003] In the related art, many other functional modules and software programs in the server management control chip need to occupy the memory bus and operate the memory DDR. Therefore, the output control module will not always obtain control of the bus when outputting video data, while the host-side video data is continuously generated. Since the video data cannot be output in time, a large number of frame losses occur in the output control module, especially the loss of key frames, which greatly reduces the display quality of the remote end. At the same time, the large number of non-key frames transmitted is a huge waste of the chip's memory bandwidth and bus bandwidth, resulting in low video combing efficiency and poor video quality. Summary of the invention

[0004] The purpose of the present invention is to provide a video frame processing method, device, equipment, medium and product, which can effectively avoid the problem that non-key frames occupy bandwidth resources when bus resources are tight, thereby causing key frames to be lost due to insufficient resources; and can ensure the timely and rapid transmission of key frames, thereby significantly improving the display quality of the remote end.

[0005] In a first aspect, a video frame processing method is provided, comprising: obtaining a first valid write count of a pixel point of a current video frame, the first valid write count indicating the total number of pixel point data having differences between the current video frame and a previous video frame; determining a first target preset number range in which the first valid write count lies according to a first preset number range; wherein if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; according to the first target preset number range, adjusting a frame interval, the frame interval being the number of intervals between two adjacent determinations of the first valid write count; obtaining a bus status; if the bus status is a busy status and the current video frame is not a key video frame, stopping processing of the current video frame; otherwise, performing image compression processing on the current video frame, and writing the compressed data into an on-chip cache.

[0006] In a preferred example, the present invention can be further configured as follows: according to the first target preset number range, the frame interval is adjusted, and the frame interval is the number of intervals between two adjacent first valid write times, including: if the first target preset number range is the preset number range of the minimum value, then the frame interval is determined to be unchanged, wherein the frame interval is the number of intervals between two adjacent first valid write times; if the first target preset number range is the preset number range of the middle value, then the average effective number of second valid write times of each of multiple consecutive video frames is obtained; according to the second target preset number range of the second preset number range where the average effective number is located, the frame interval is adjusted to obtain the first frame interval; if the first target preset number range is the preset number range of the largest value, then the frame interval is determined to be the second frame interval, and the second frame interval is smaller than the frame interval and smaller than the first frame interval.

[0007] In a preferred example, the present invention can be further configured as follows: adjusting the frame interval according to a second target preset number range of the second preset number range where the average effective number of times is located, including: determining an adjustment coefficient according to the second target preset number range of the second preset number range where the average effective number of times is located; obtaining a first frame interval according to the adjustment coefficient and the frame interval;

[0008] And / or, when the first preset number range and the second preset number range are the same, the method also includes: if the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, then obtaining multiple consecutive average effective times; if there is a target average effective number that is less than the minimum value of the first target preset number range, then increasing the frame interval.

[0009] In a preferred example, the present invention can be further configured as follows: before determining whether the current video frame is a key video frame based on the first valid write count, it also includes: obtaining the third valid write count of each of the previous multiple video frames; determining whether the current video frame is a key video frame based on the third valid write count of each of the previous multiple video frames; if it is a key video frame, determining in advance that the frame interval is the second frame interval.

[0010] In a preferred example, the present invention can be further configured as: determining whether the current video frame is a key video frame based on the third valid write times of each of the previous multiple video frames, including: determining whether the valid write times gradually increase based on the third valid write times of each of the previous multiple video frames; if it gradually increases, and the first valid write times of the current video frame is within the set range of the preset maximum times threshold, determining that the current video frame is a key video frame; otherwise, determining that the current video frame is not a key video frame.

[0011] In a preferred example, the present invention can be further configured as follows: according to the third valid write times of each of the previous multiple video frames, determining whether the third valid write times gradually increases, including: obtaining the difference in the third valid write times of two adjacent video frames; for the two adjacent number differences, determining whether the first value corresponding to the previous number difference is less than the second value corresponding to the subsequent number difference, and obtaining a determination result; the first value is obtained based on the previous number difference and the corresponding first weight, and the second value is obtained based on the subsequent number difference and the corresponding second weight; according to the determination results of all the adjacent number differences, determining whether the third valid write times gradually increases, wherein if all the determination results are yes, it is determined that the third valid write times gradually increases.

[0012] In a preferred example, the present invention can be further configured as follows: it also includes: if the current video frame is determined to be a key video frame based on the first valid write number and the third valid write number, then the second weight is increased; if the current video frame is determined not to be a key video frame based on the first valid write number, and if the current video frame is determined to be a key video frame based on the third valid write number, then the first weight is increased.

[0013] In a preferred example, the present invention can be further configured as: if the bus state is a busy state, and the current video frame is not a key video frame, then stop processing the current video frame, including: if the bus state is a busy state, and based on the first valid write number and the third valid write number, it is determined that the current video frame is not a key video frame, then stop processing the current video frame.

[0014] In a preferred example, the present invention can be further configured as follows: before writing the compressed data into the on-chip cache, it also includes: obtaining the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; determining whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; if the compressed data can be completely written into the on-chip cache, writing the compressed data into the on-chip cache.

[0015] In a preferred example, the present invention can be further configured as follows: determining whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame, including: determining a third value based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and a third weight; determining a fourth value based on the compressed length of the key frame and the second write rate of the frame data into the on-chip cache; determining whether the compressed data can be completely written into the on-chip cache based on the sizes of the third and fourth values, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

[0016] In a preferred example, the present invention can be further configured as: obtaining a first write rate of a double data rate memory, including: obtaining the number of signal validity times during which a write data valid signal of an interactive interface of the double data rate memory and a write data allowance indication signal output by a bus of the double data rate memory are simultaneously valid within a preset time period; determining the first write rate of the double data rate memory based on the number of signal validity times and the preset time period.

[0017] In a preferred example, the present invention can be further configured as follows: it also includes: if the first write rate is less than a preset rate threshold, determining that the bus state is a busy state and reducing the preset time period; if the first write rate is not less than the preset rate threshold, increasing the preset time period.

[0018] In a preferred example, the present invention can be further configured as follows: it also includes: if the on-chip cache cannot be completely written and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if the on-chip cache cannot be completely written and the current video frame is not a key video frame, the compressed data will be discarded.

[0019] In a preferred example, the present invention can be further configured as follows: it also includes: obtaining the total amount of discarded non-critical frame data discarded in the on-chip cache; if the total amount of discarded data reaches a preset total amount of discarded data threshold, lowering the third weight; after lowering the third weight, if it is detected that a preset amount of non-critical frame data has not been discarded, resetting the third weight.

[0020] In a second aspect, a video frame processing device is provided, comprising: a video capture module, used to obtain a first valid write count of a pixel point of a current video frame, wherein the first valid write count represents the total number of pixel point data with differences between the current video frame and the previous video frame; an interval adaptation module, used to determine a first target preset number range in which the first valid write count is located according to a first preset number range; wherein if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; the interval adaptation module is also used to adjust the frame interval according to the first target preset number range, wherein the frame interval is the number of intervals between two adjacent first valid write counts; a non-key frame processing module, used to obtain a bus status; if the bus status is a busy state and the current video frame is not a key video frame, the processing of the current video frame is stopped; otherwise, the video processing module is triggered; a video processing module, used to perform image compression processing on the current video frame; and a data output control module, used to write the compressed data into an on-chip cache.

[0021] In a third aspect, a server management control device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the video frame processing method described in any one of the first aspects.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by a processor to implement the video frame processing method as described in any one of the first aspects.

[0023] In a fifth aspect, a computer program product is provided, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the video frame processing method as described in any one of the first aspects is implemented.

[0024] In summary, the video frame processing method provided by the present invention includes the following beneficial technical effects: obtaining the first valid write count of the pixel point of the current video frame, so as to identify the degree of difference between the current video frame and the previous frame; judging whether the current video frame is a key video frame according to the number range of the first valid write count; and, because in the monitoring mode, since the previous and subsequent frames of video basically do not change much, each video frame is compared with the previous video frame to determine the valid write count, which increases the meaningless processing of the chip and the video frame processing efficiency is low. The present scheme also dynamically adjusts the frame interval obtained by the next first valid write count according to the preset number range of the first valid write count, thereby improving the video frame processing efficiency; when it is confirmed that the bus state is busy and the current frame is not a key frame, the processing of the current video frame is stopped, effectively avoiding the problem that non-key frames occupy bandwidth resources when bus resources are tight, thereby causing the key frames to be lost due to insufficient resources; for non-busy states or key video frames, image compression processing is performed and written into the on-chip cache, which can save bandwidth and ensure the timely transmission of key frames, thereby significantly improving the display quality of the remote end.

[0025] In addition, the present invention also provides a video frame processing device, equipment, medium and product, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a schematic diagram of implementing the video function in a server management control chip of the related technology.

[0028] Figure 2 It is a hardware processing flow chart of a video data output control module in the related technology.

[0029] Figure 3 This is a schematic diagram of the implementation of the BLOCK conversion process.

[0030] Figure 4 It is a flowchart of a video processing method provided by an embodiment of the present invention.

[0031] Figure 5 It is a schematic diagram of coefficient correspondence provided by an embodiment of the present invention.

[0032] Figure 6 The present invention provides a schematic diagram of a video hardware processing flow of a server management and control device.

[0033] Figure 7 The present invention provides a data output control module hardware flow chart.

[0034] Figure 8 It is a structural schematic diagram of a video processing device provided by an embodiment of the present invention.

[0035] Fig. 9 It is a structural diagram of a server management control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0038] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0039] In order to better understand and illustrate the solutions of the embodiments of the present invention, some technical terms involved in the embodiments of the present invention are briefly explained below.

[0040] Server management and control equipment: refers to the server field. This chip is used to monitor the status of the server (temperature, fan, main CPU operation status, etc.). It also has an important function, which is to transmit local video information to the remote end through the network for remote display and monitoring.

[0041] SoC: System on Chip, or SoC for short, is also known as system on chip. SoC is generally defined as the integration of microprocessors, analog IP cores, digital IP cores, and memory (or off-chip memory control interface) on a single chip. It is usually customized or a standard product for specific purposes.

[0042] Compressed video controller: The original image information output by the local graphics card needs to be compressed before being transmitted to the remote end through the network. The purpose is to reduce the amount of video data and reduce the network bandwidth.

[0043] See also Figure 1 and Figure 2 , Figure 1 It is a schematic diagram of realizing the video function in the server management control chip of the related technology. Figure 2 It is the hardware processing flow of the video data output control module in the related technology.

[0044] The processing process of the video processing function in the related art is:

[0045] 1: The original video data (operating system interface) on the server host is transmitted to the VGA module inside the server management control chip through PCIe. VGA generates original video data in RGB format (in the process of generating data, VGA needs to interact with the off-chip DDR).

[0046] 2: The video capture module obtains video data in frames through the DVI interface (Digital Visual Interface) of VGA.

[0047] 3: Color space conversion module (RGB2YUV module), according to the matrix conversion formula, realizes color space conversion and converts the video data in RGB format into video data in YUV format.

[0048] The matrix conversion formula is as follows:

[0049] .

[0050] .

[0051] .

[0052] 4: The block generation conversion module realizes the BLOCK conversion of YUV data. According to the YUV444 or YUV422 / YUV420 format, the size of BLOCK is , the specific BLOCK conversion process is as follows Figure 3 .

[0053] 5: The video core compression module receives YUV data in BLOCK format and then performs video compression. There are many compression formats, the most common ones are JPEG / H.264, etc. The compressed data is output to the video output control module.

[0054] 6: The video output control module receives the compressed video data, generates control logic, and writes the compressed data into DDR.

[0055] like Figure 2As shown, the compressed data cache control submodule is used to control the writing control of compressed data in the on-chip cache. The on-chip cache is generally implemented with a RAM with a large depth. The write status cache control submodule is used to store the write status of each frame of compressed data in the on-chip cache, that is, complete write and incomplete write. The read control submodule is used to control the reading of compressed data from the on-chip cache. The interface timing control submodule is used to generate the AXI (Advanced eXtensibleInterface) interface protocol to write the compressed data to the off-chip DDR. The frame status generation submodule is used to save the status information of each frame written to the DDR, such as the frame header DDR address, frame tail DDR address, frame length, etc., for the software driver to read.

[0056] 7: The driver of the network module reads the DDR to obtain the compressed data, and sends it to the remote end through the network for display.

[0057] 8: There are two operation modes for remote users, one is monitoring mode and the other is management mode. In monitoring mode, users do not care about the specific information of each frame, while in management mode, users want to obtain the video screen of the local operating system as soon as possible.

[0058] In the related art, many other functional modules and software programs in the server management control chip need to occupy the memory bus and operate the memory DDR. Therefore, the output control module will not always obtain control of the bus when outputting video data. The host-side video data is continuously generated. Since the video data cannot be output in time, a large number of frame losses occur in the output control module, especially the loss of key frames, which greatly reduces the display quality of the remote end. At the same time, a large number of non-key frames are transmitted, which is a huge waste of the chip's memory bandwidth and bus bandwidth.

[0059] In this context, it is necessary and urgent to optimize the video frame loss function of the server management control chip, reduce the frame loss rate of key frames, discard some non-key frames, improve the remote display quality and human-computer interaction experience, and at the same time reduce the memory bandwidth and bus bandwidth occupied by the video function and improve the overall performance of the chip.

[0060] Based on this, the present invention relates to a hardware design method for optimizing the key frame processing of a server management control chip, aiming to improve the drawbacks of traditional solutions, reduce the frame loss rate of key frames, improve the display quality of the remote end, reduce the memory bandwidth and bus bandwidth occupied by video functions, optimize the overall chip performance, and improve user experience.

[0061] Specifically, Figure 4 A flow chart of a video frame processing method provided in an embodiment of the present invention is provided. The method provided in the embodiment of the present invention can be executed by a server management control device. The video frame processing method includes:

[0062] S101, obtaining a first valid write count of a pixel point of a current video frame.

[0063] The first valid write count indicates the total number of pixel data that differ between the current video frame and the previous video frame, which can be obtained by detecting the write address and write data interface of the VGA, and counting the number of valid write addresses and write data in one frame, updata_cnt, as the first valid write count.

[0064] In the embodiment of the present invention, the video capture module is used to obtain the first valid write count of the pixel point of the current video frame. The video capture module has a new data comparison function, which is to detect the behavior of the interface bus between the VGA module and the DDR (VGA video memory) in units of frames. Figure 5 The A space of the off-chip DDR in is the video memory space of the VGA. The behavior of the VGA is to only update the data of the same pixel position in the current frame compared to the previous frame that has changed. For the data of the same pixel position in the previous and next frames that has not changed, the VGA does not update its data. For example, if the pixel data of the 10th row and 12th column of the 1st frame is consistent with the pixel data of the 10th row and 12th column of the 0th frame, then when the VGA updates the video memory data of the 1st frame, the pixel data is not updated, that is, the VGA does not send the write address and write data of the pixel position. Compared with the related art, the video capture module of the present invention has a new function of detecting the write address and write data interface of the VGA in units of frames, and counting the number of valid write addresses and write data in 1 frame, updata_cnt. This value reflects the changed pixel value of the subsequent frame compared to the previous frame. The frame is also a new function, that is, the data comparison function of the capture module needs to use the first and last addresses of the VGA video memory as the range to detect and calculate the updata_cnt described above. The value of updata_cnt is passed to Figure 5 The key frame identification module in the video frame can at least determine whether the current video frame is a key video frame.

[0065] S102: Determine a first target preset number range in which a first valid writing number is located according to a first preset number range.

[0066] Among them, if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; wherein, the key video frame refers to a frame with significant changes in a video sequence.

[0067] In an embodiment of the present invention, the interval adaptive module can determine whether it is a key video frame according to the first valid write count. In some embodiments, the first valid write count can be compared with a preset key frame judgment threshold, and if it is greater than the key frame judgment threshold, the current video frame is determined to be a key frame.

[0068] Specifically, the first preset number range is a pre-set range, which can be composed of multiple thresholds, such as (0, a1], [a1, a2), [a2, +∞), a1<a2; through the adaptive adjustment module, the first target preset number range is determined according to the first valid write number; if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the range it is in is the largest range such as [a2, +∞), indicating that the current video frame is a key video frame; otherwise, it is considered that the current video frame is not a key video frame.

[0069] It can be seen that in the embodiment of the present invention, the first valid write count is compared with a plurality of preset count ranges to determine the first target preset count range to which the first valid write count belongs. If the data in the range are all significantly greater than the maximum values ​​of other ranges, that is, the range with the largest value, the current video frame is determined to be a key video frame, and the key frame containing important visual changes can be more accurately identified.

[0070] S103, adjusting the frame interval according to the first target preset number range, where the frame interval is the number of intervals between two adjacent first valid write times.

[0071] Furthermore, the function of the adaptive adjustment module may also include adaptively adjusting the frame interval for comparing the previous and next frames. Since the VGA interface behavior detection and data comparison behavior are not performed for each frame, in the actual application of the server management control chip, especially when the local server is in monitoring mode, the interface of the server host operating system basically does not change, that is, the previous and next frames, or the previous and next dozen frames, dozens of frames will not change significantly, so there is no need to compare and detect each frame with the previous frame, because this will increase the extra power consumption of the chip and meaningless processing, and reduce the performance of the chip.

[0072] Therefore, the interval adaptive module is designed in the present invention, which can calculate and output the frame interval number frame_period used for data detection and comparison by the video capture module. The implementation method is to first use the default frame interval N, the N value can adopt the default value, or can be configured by the user through a register. Generally, the default value of N is 8 to 15, such as 8, 10, 11, 15; at the same time, the output result of the capture module is detected, the value of updata_cnt.

[0073] Specifically, if the first target preset number range is the preset number range of the minimum value, the frame interval is determined to be unchanged; if the first target preset number range is the preset number range of the middle value, the average effective number of the second effective write times of multiple consecutive video frames is obtained; according to the second target preset number range of the second preset number range where the average effective number is located, the frame interval is adjusted to obtain the first frame interval; if the first target preset number range is the preset number range of the maximum value, the frame interval is determined to be the second frame interval; wherein the frame interval is the number of intervals between two adjacent determinations of the first effective write times, and the second frame interval is less than the frame interval, and less than the first frame interval.

[0074] Among them, the preset number range of the minimum value represents the range of the minimum value. For example, the preset number range includes: (0, a0], [a0, a1), [a1, a2), [a2, +∞), where a0<a1<a2, then the preset number range of the minimum value is (0, a0]; the maximum preset number range is [a2, +∞); the preset number range of the intermediate value includes: [a0, a1), [a1, a2).

[0075] The adjustment strategy of the N value is that when the value of the first valid write count updata_cnt is less than or equal to the threshold a0, that is, it belongs to the preset number of times with the smallest value, N is not adjusted; when the value of the first valid write count updata_cnt is greater than the threshold a2, that is, it belongs to the preset number of times with the largest value, it means that the change between frames is very large, and N is directly set to the second frame interval, and the second frame interval can be 1, that is, the data capture module requires continuous comparison and detection of adjacent frames; when the value of the first valid write count updata_cnt is greater than the threshold a0 and less than or equal to the threshold a1, or, greater than the threshold a1 and less than or equal to the threshold a2, start to continuously detect the average value updata_cnt_average of the values ​​of multiple frames updata_cnt as the average valid number, and adjust the frame interval according to the second target preset number range of the second preset number range where the average valid number is located to obtain the first frame interval.

[0076] Among them, the second preset number range includes multiple ranges, and the second preset number range can be the same as the first preset number range, or it can be different. The embodiment of the present invention is no longer limited, as long as the purpose of the embodiment of the present invention can be achieved. Exemplarily, the second preset number range includes: (0, a10], [a10, a11), [a11, a12), [a12, +∞).

[0077] It can be understood that the first frame interval is greater than the second frame interval, and the first frame interval is less than the default frame interval N; and the greater the average effective number of times, the smaller the first frame interval. The embodiment of the present invention does not limit the number of consecutive multiple video frames, and the user can set it according to actual needs.

[0078] S104, obtaining the bus status; if the bus status is busy and the current video frame is not a key video frame, then stopping the processing of the current video frame.

[0079] Wherein, the bus state includes a busy state and a non-busy state. In an embodiment of the present invention, the feedback signal (bus_state) of the data output control module, i.e., the DDR bus state, is obtained through the non-key frame processing module. When the signal is bus_state=2'b10, it indicates that the current DDR bus state is very busy, and the data output control module cannot write data into the DDR in time. At the same time, if it is determined that the current video frame is not a key video frame, the non-key frame processing module will output a control signal to the RGB2YUV module to stop the subsequent processing of the current non-key frame, i.e., pull up the signal stop_process_cur_frame to stop the processing of the current video frame, i.e., the current non-key frame does not perform subsequent RGB2YUV, block processing, compression, transmission and other processing, thereby reducing the amount of data written into the DDR by the data output control module, and reducing the bus occupancy of the chip video function without affecting the remote end video display quality, thereby improving the overall performance of the chip.

[0080] At the same time, the embodiment of the present invention detects in real time in units of frames. When the state of bus_state is not 2'b10 (busy state) or the current frame is a key frame, it is necessary to exit the above-mentioned frame loss mode and compress and transmit the current video frame normally. That is, this function can realize that non-key frames do not perform video compression and other processing under the premise that the system bus is very busy, and also includes compression processing of non-key frames and key frames under normal system bus status, or normal compression processing of key frames under busy system bus conditions.

[0081] S105: Otherwise, perform image compression processing on the current video frame.

[0082] If the bus state is busy and the current video frame is a key video frame, image compression processing is performed on the current video frame; if the bus state is non-busy, regardless of whether the current video frame is a key video frame or not, image compression processing is performed on the current video frame through the video processing module.

[0083] The video processing module includes: an RGBYUV module, a block generation module and a video core compression module.

[0084] S106 , writing the compressed data into the on-chip cache.

[0085] In the embodiment of the present invention, the compressed data is written into the on-chip cache through the data output control module.

[0086] It can be seen that in the embodiment of the present invention, the first valid write count of the pixel point of the current video frame is obtained, and the difference between the current video frame and the previous frame can be identified; according to the number range of the first valid write count, it is determined whether the current video frame is a key video frame; and, because in the monitoring mode, since the previous and subsequent frames of video basically do not change much, each video frame is compared with the previous video frame to determine the valid write count, which increases the meaningless processing of the chip and the video frame processing efficiency is low. The scheme also dynamically adjusts the frame interval obtained by the next first valid write count through the preset number range of the first valid write count, thereby improving the video frame processing efficiency; when it is confirmed that the bus state is busy and the current frame is not a key frame, the processing of the current video frame is stopped, which effectively avoids the non-key frame occupying bandwidth resources when the bus resources are tight, thereby causing the key frame to be lost due to insufficient resources; for non-busy states or key video frames, image compression processing is performed and written into the on-chip cache, which can save bandwidth and ensure the timely transmission of key frames, thereby significantly improving the display quality of the remote end.

[0087] In one feasible method, the frame interval is adjusted according to a second target preset number range of a second preset number range in which the average effective number of times is located, including: determining an adjustment coefficient according to the second target preset number range of the second preset number range in which the average effective number of times is located; and obtaining a first frame interval according to the adjustment coefficient and the frame interval.

[0088] In an embodiment of the present invention, different second preset number ranges are provided with different adjustment coefficients, and the adjustment coefficient is less than 1. If the value of the second preset number range (i.e., the second target preset number range) where the average effective number of times is located is larger, it means that the difference between the previous and next frames is larger, and it may tend to be a key frame, then the corresponding adjustment coefficient is smaller, so as to reduce the frame interval and perform comparisons more frequently to ensure that no key frames are missed.

[0089] In a possible embodiment, the adjustment coefficient can be determined only according to the second target preset number range of the second preset number range where the average effective number of times is located, wherein the larger the value of the second preset number range is, the smaller the adjustment coefficient is. Exemplarily, each second preset number range corresponds to an adjustment coefficient, and the first frame interval can be directly based on the adjustment coefficient*frame interval.

[0090] In another possible embodiment, each first preset number range corresponds to a first coefficient, and each second preset number range corresponds to a second coefficient adjustment coefficient. The first frame interval can be used as the first coefficient * adjustment coefficient * frame interval, wherein the larger the value of the first preset number range is, the smaller the first coefficient is.

[0091] For example, see Figure 5 When the value of updata_cnt is greater than the threshold a0 and less than or equal to the threshold a1, start to continuously detect the average effective number of updata_cnt values ​​of multiple frames updata_cnt updata_average. If the value of updata_cnt_average is greater than the threshold a10 and less than or equal to the threshold a11, start to adjust N, and adjust it in the direction of small, N*coefficient 1 (coefficient 1 is the adjustment coefficient of (a0, a1])*coefficient 3, coefficient 1≤1, coefficient 3<1. If the value of updata_cnt_average is greater than the threshold a11 and less than or equal to the threshold a12, adjust N, and adjust it in the direction of small, N*coefficient 1*coefficient 4, coefficient 4<coefficient 3<1; if the value of updata_cnt_average is greater than the threshold a12, adjust N to the second frame interval.

[0092] When the value of updata_cnt is greater than the threshold a1 and less than or equal to the threshold a2, start to continuously detect the average value updata_cnt_average of the values ​​of multiple frames of updata_cnt. If the value of updata_cnt_average is greater than the threshold a11 and less than or equal to the threshold a12, continue to adjust N to a smaller direction, N*coefficient 2 (coefficient 2 is the adjustment coefficient of (a1, a2], less than the adjustment coefficient of (a0, a1])*adjustment coefficient 6; adjustment coefficient 6<1. If the value of updata_cnt_average is greater than the threshold a10 and less than or equal to the threshold a11, continue to adjust N to a smaller direction, N*coefficient 2*coefficient 5. If the value of updata_cnt_average is greater than the threshold a12, adjust N to the second frame interval, where coefficient 5 is greater than coefficient 6.

[0093] In the embodiment of the present invention, the user can set coefficient 1, coefficient 2, coefficient 3, coefficient 4, coefficient 5 and coefficient 6 according to actual needs, as long as the purpose of the present invention can be achieved.

[0094] Furthermore, the setting of the frame interval N in the embodiment of the present invention can be synchronously detected and set in two directions. When the first preset number range and the second preset number range are the same, that is, when the value of updata_cnt_average reaches a threshold higher than the first valid write number, the result of continued detection is that updata_cnt_average changes toward a smaller threshold, then N is synchronously adjusted to increase the N value.

[0095] Specifically, when the first preset number range and the second preset number range are the same, the method also includes: if the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, then obtaining multiple consecutive average effective times; if there is a target average effective number that is less than the minimum value of the first target preset number range, then increasing the frame interval.

[0096] Exemplarily, when the value of updata_cnt_average reaches a threshold value higher than the first valid write count, and the result of continued detection is that updata_cnt_average changes toward a smaller threshold value, N is synchronously adjusted to increase the value of N.

[0097] When the value of updata_cnt is greater than threshold a1 and less than or equal to threshold a2, start to continuously detect the average value updata_cnt_average of the updata_cnt values ​​of multiple frames. If the value of updata_cnt_average starts to be less than threshold a1, adjust N in a larger direction or keep N unchanged and continue detecting.

[0098] In the embodiment of the present invention, the function of two-way detection enables the present solution to realize rapid identification and processing from non-key frames to key frames, thereby ensuring that the remote end can obtain key frames with high quality, and can also realize recovery from key frames to non-key frames. On the basis of ensuring the video quality, the occupation of the system bus and memory bus by the video function is minimized, thereby improving the overall performance of the chip.

[0099] Furthermore, before determining whether the current video frame is a key video frame based on the first valid write count, the method also includes: obtaining the third valid write count of each of the previous multiple video frames; determining whether the current video frame is a key video frame based on the third valid write count of each of the previous multiple video frames; if it is a key video frame, determining in advance that the frame interval is the second frame interval.

[0100] If it is not a key video frame, no operation is required.

[0101] In the embodiment of the present invention, the key frame prediction module is used to predict the key frame, and its basic principle is to determine the number of valid write times corresponding to each of the previous multiple video frames.

[0102] Specifically, based on the change rate of the effective write times updata_cnt of the current multiple video frames, it is possible to predict in advance whether updata_cnt is about to reach the highest threshold value such as a2. If the prediction result is that the highest threshold value is about to be reached, the frame interval N value is adjusted in advance, that is, the number of frame intervals for data detection and comparison is adjusted in advance.

[0103] It can be seen that in the embodiment of the present invention, before judging based on the first valid write number, the third valid write number of the previous multiple video frames is additionally considered; the attribute parameters of the current frame are predicted by analyzing the historical data; if the prediction result shows that the current frame is a key video frame, the frame interval is adjusted to the second frame interval in advance, further reducing the risk of key frame loss.

[0104] In an achievable manner, determining whether the current video frame is a key video frame according to the third valid write times of each of the previous multiple video frames includes: determining whether the valid write times gradually increase according to the third valid write times of each of the previous multiple video frames; if it gradually increases, and the first valid write times of the current video frame is within the setting range of the preset maximum times threshold, then determining that the current video frame is a key video frame; otherwise, determining that the current video frame is not a key video frame. Wherein, if |updata_cnt- a2|<preset value is satisfied, wherein the preset value can be set by a user, then it means that the first valid write times of the current video frame is within the setting range of the preset maximum times threshold.

[0105] Specifically, according to the third valid write times of each of the previous multiple video frames, determine whether the third valid write times gradually increases, including: obtaining the difference in the third valid write times of two adjacent video frames; for the two adjacent number differences, determine whether the first value corresponding to the previous number difference is less than the second value corresponding to the subsequent number difference, and obtain a determination result; the first value is obtained based on the previous number difference and the corresponding first weight, and the second value is obtained based on the subsequent number difference and the corresponding second weight; according to the determination results of all the adjacent number differences, determine whether the third valid write times gradually increases, wherein if all the determination results are yes, it is determined that the third valid write times gradually increases.

[0106] In an embodiment of the present invention, the difference in the third effective write count updata_cnt of M frames (M is less than N) is detected. The difference is increasing, and the difference between the updata_cnt of the latest frame and a2 is less than a preset value, where M is configurable and the preset value is configurable. The specific formula (hardware judgment logic) is as follows: (updata_cnt_1 - updata_cnt_0) × the first first weight < (updata_cnt_2 - updata_cnt_1) × the first second weight; (updata_cnt_2 - updata_cnt_1) × the second first weight < (updata_cnt_3 - updata_cnt_2) × the second second weight; (updata_cnt_3 - updata_cnt_2) × the third first weight < (updata_cnt_4 - updata_cnt_3) × the third second weight; …; (updata_cnt_M - 2 - updata_cnt_M - 3) × the (M - 2)th first weight < (updata_cnt_M - 1 - updata_cnt_M - 2) × the (M - 2)th second weight; and |updata_cnt_M - 1 - Threhold_2| < Threhold_pre, where updata_cnt_M - 1 is the effective write count of the current video frame. Among them, the first weight and the second weight marked in the above formula (hardware judgment logic) can adopt default values such as 1, or can be configured by the user. At the same time, in the embodiment of the present invention, the above parameters are adaptively adjusted according to the correctness of the predicted result. When the above formulas are all satisfied, it is determined that the current video frame is a key frame, the frame interval N value is adjusted in advance, the current frame is processed as a key frame, and at the same time, the output result of the interval adaptive module is awaited.

[0107] Further, the video frame processing method further includes: if it is determined that the current video frame is a key video frame according to both the first effective write count and the third effective write count, then increase the second weight; if it is determined that the current video frame is not a key video frame according to the first effective write count, and it is determined that the current video frame is a key video frame according to the third effective write count, then increase the first weight.

[0108] If the judgment result of the interval adaptive module after a few frames is also a key frame, that is, the value of updata_cnt is greater than the threshold a2, then the judgment result of the key frame prediction mechanism is considered correct. In this case, the second weight is increased, and the increased value can be customized by the user, that is, the probability of all the above formulas being true is increased, and the probability of judging the key frame is increased, that is, the above formula can continue to judge the key frame with a high probability. If the judgment result of the interval adaptive module after a few frames is not a key frame, and the value of updata_cnt does not have the threshold a2, then the judgment result in this step is considered incorrect. In this case, the first weight is increased to reduce the probability of all the above formulas being true, that is, reduce the probability of judging the key frame. That is to say, it is guaranteed that the above formula reduces the probability of judging the key frame.

[0109] In the embodiment of the present invention, the weight is adjusted in the above manner, so that the determination result of the key frame prediction mechanism is more accurate.

[0110] Correspondingly, if the bus state is busy and the current video frame is not a key video frame, the processing of the current video frame is stopped, including: if the bus state is busy, and according to the first valid write number and the third valid write number, it is determined that the current video frame is not a key video frame, then the processing of the current video frame is stopped. Wherein, after receiving the feedback signal bus_state of the post-stage data output control module that the current DDR bus state is very busy, the data output control module cannot write data to the DDR in time, and at the same time, the results of the interval adaptive module and the key frame prediction module both indicate that the current frame is a non-key frame. At this time, the non-key frame processing module outputs a control signal to the RGB2YUV module to stop the subsequent processing of the current non-key frame.

[0111] It can be seen that in the embodiment of the present invention, when the bus state is busy and both key frames are determined not to be key video frames, the video processing is stopped, so that the attributes of the current video frame can be determined more accurately.

[0112] See also Figure 6 , Figure 6 It is a schematic diagram of the video hardware processing flow of a server management and control device provided by an embodiment of the present invention; the function of the video capture module is optimized, and an interval adaptive module, a key frame prediction module, and a non-key frame processing module are newly added to realize adaptive prediction and recognition of key frames, dynamically adjust the discrimination parameters, and improve the discrimination accuracy.

[0113] Furthermore, see Figure 7 , Figure 7It is a hardware flow diagram of a data output control module provided by an embodiment of the present invention; the embodiment of the present invention also adds a key frame analysis submodule, a bus state prediction submodule, and a frame loss statistics submodule in the data output control module, and improves the read control submodule; it not only realizes adaptive pre-judgment and recognition of key frames, dynamically adjusts the discrimination parameters, and improves the discrimination accuracy, but also improves the complete write rate of the on-chip storage space of key frames, realizes autonomous frame loss for non-key frames, reduces the occupation of memory bandwidth and bus bandwidth by video functions, and improves the overall performance of the chip.

[0114] A possible implementation method of an embodiment of the present application, before writing the compressed data into the on-chip cache, also includes: obtaining the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; determining whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; if the compressed data can be completely written into the on-chip cache, writing the compressed data into the on-chip cache.

[0115] In an embodiment of the present invention, the key frame analysis and judgment submodule in the data output control module receives the output result of the previous key frame prediction module. If the current video frame is judged to be a key frame, the key frame prediction module outputs a control signal to the key frame analysis and judgment submodule, that is, the key_frame signal is pulled high. After the key frame analysis and judgment submodule receives this signal, the data output control module enters the urgent mode (urgent_mode).

[0116] Furthermore, after the data output control module enters the emergency mode, the read control submodule receives the output results of the bus state judgment submodule and the compressed data cache control module, and pre-judges whether the key frame can be completely written into the on-chip cache; the output result of the compressed data cache control submodule is the remaining space of the on-chip cache; the output result of the bus state judgment submodule is the first write rate of the double data rate memory. If the pre-judgement result is that the key frame can be completely written into the on-chip cache, normal reading is performed.

[0117] If the on-chip cache cannot be completely written, and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if the on-chip cache cannot be completely written, and the current video frame is not a key video frame, the compressed data will be discarded. Specifically, if the result of the read control submodule's prediction is that the key frame cannot be completely written to the on-chip cache, the non-key frame data in the on-chip cache will be read in advance in units of frames and discarded. A frame information table is pre-set in the read control submodule, which records the address space range of non-key frames in the on-chip cache, that is, these non-key frames read out are not written to the external DDR.

[0118] It can be seen that in the embodiment of the present invention, the compressed data is written into the on-chip cache only when it is determined that the compressed data can be completely written into the on-chip cache. If the compressed data cannot be completely written into the on-chip cache and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames. If the compressed data cannot be completely written into the on-chip cache and the current video frame is not a key video frame, the compressed data will be discarded to reduce the frame loss rate of the key frames and improve the display quality of the remote end.

[0119] Specifically, a possible implementation method of an embodiment of the present invention determines whether the compressed data can be completely written into the on-chip cache according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compression length of the key frame, including: determining a third value according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the third weight; determining a fourth value according to the compression length of the key frame and the second write rate of the frame data into the on-chip cache; determining whether the compressed data can be completely written into the on-chip cache according to the sizes of the third value and the fourth value, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

[0120] The calculation process is as follows:

[0121] 1: The compressed data cache control submodule outputs the remaining space space_can_use of the on-chip cache, that is, the space to be written.

[0122] 2: The bus state judgment submodule outputs the first write rate wr_ddr_rate of the double data rate memory within a preset time period T, where T can adopt a default value or can be configured by a user through a register.

[0123] 3: The rate at which frame data is written to the on-chip cache is fixed (depending on the module's clock frequency). The second write rate at which frame data is written to the on-chip cache is recorded as wr_buffer_speed.

[0124] 4: That is, the compressed length of the key frame is frame_compress_lenth, and the judgment formula (hardware judgment logic) is as follows: according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data written to the on-chip cache and the third weight PARAM0, determine the third value, wherein the third value = space_can_use / (wr_buffer_speed-wr_ddr_rate×PARAM0); wherein PARAM0 is an adjustable parameter, and the default value is 1; according to the compressed length of the key frame and the second write rate of the frame data written to the on-chip cache, determine the fourth value; the fourth value = rame_compress_lenth / wr_buffer_speed; if the third value is greater than the fourth value, it is determined that the current key frame can be completely written to the on-chip cache; if the third value is not greater than the fourth value, it is determined that the current key frame cannot be completely written to the on-chip cache.

[0125] It can be seen that in the embodiment of the present invention, the third value is determined according to the cache margin, DRAM write rate, cache write rate and weight; the fourth value is determined according to the key frame compression length and the cache write rate; the two values ​​are compared, and if the third value is larger, the compressed data can be completely stored in the cache.

[0126] A possible implementation of the embodiment of the present application is to obtain a first write rate of a double data rate memory, including: obtaining the number of times a write data valid signal of an interactive interface of the double data rate memory and a write data allowance indication signal output by a bus of the double data rate memory are simultaneously valid within a preset time period; and determining the first write rate of the double data rate memory according to the number of signal valid times and the preset time period.

[0127] The bus state is detected, counted, and judged by the bus state judgment submodule, and then two data are output, one is bus_state output to the front-stage non-key frame processing module, and the other is wr_ddr_rate output to the read control module. The process of detecting the bus state is to count the valid data write signal wr_data_vld of the interface (usually AXI) that the data output control module interacts with DDR and the write data indication signal (wr_ready) output by the bus of the double data rate memory within the preset time period. The valid number of signals valid_cnt when these two signals are valid at the same time (indicating that a piece of data is effectively written into the DDR), where wr_ddr_rate = valid_cnt / preset time period T.

[0128] In one implementation, if the first write rate is less than a preset rate threshold, the bus state is determined to be a busy state and the preset time period is reduced; if the first write rate is not less than the preset rate threshold, the preset time period is increased.

[0129] When the first write rate wr_ddr_rate is less than the preset rate threshold, it means that the write rate of DDR is very slow, the system bus resources are tight, and the data output control module cannot write data to the off-chip DDR in time. At this time, bus_state=2'b10 is set, where the preset rate threshold is the bus state threshold, and the value of the preset time period is reduced, that is, the bus state is further detected in time, and the detection cycle of the bus state is shortened, in order to detect the state change in time. The specific reduced value can be set according to the user's configuration parameters.

[0130] When the first write rate wr_ddr_rate is not less than the preset rate threshold, it means that the write rate of DDR meets the preset requirement, and the data output control module can write the data to the off-chip DDR in time. At this time, bus_state=2'b01 is set, and the value of the preset time period is increased, and the detection cycle of the bus state is appropriately relaxed. The specific increased value can be set according to the user's configuration parameters.

[0131] A possible implementation method of an embodiment of the present application also includes: obtaining the total amount of discarded non-critical frame data in the on-chip cache; if the total amount of discarded data reaches a preset total amount of discarded data threshold, lowering the third weight; after lowering the third weight, if it is detected that a preset amount of non-critical frame data has not been discarded, resetting the third weight.

[0132] In an embodiment of the present invention, through the frame loss statistics submodule, the frame loss statistics submodule receives the total amount of non-critical frames dropped and continuously discarded in the on-chip cache read by the read control submodule. When drop_cnt continuously accumulates to reach the preset total amount of discarded threshold, that is, the total amount of discarded> the preset total amount of discarded threshold, it means that there are many non-critical frames that are continuously discarded, which will affect the display of the remote end. At this time, the third weight is adjusted to reduce the third weight and reduce the frame loss rate. At the same time, after drop_cnt detects that a preset number (1 frame or 2 frames) of non-critical frames are not lost, it automatically resets. Re-detect and restore the third weight to 1 at the same time.

[0133] It can be seen that in the embodiment of the present invention, when it is detected that the discarded non-key frame data in the on-chip cache reaches a certain total amount, the third weight used to determine the key frame is reduced, and the discarding of non-key frames is reduced, thereby balancing the video quality. When it is detected that the preset amount of non-key frame data has not been discarded, the third weight is reset to the initial state, thereby improving the efficiency and quality of the overall video transmission.

[0134] Based on any of the above embodiments, the embodiment of the present invention optimizes the function of the video capture module, adds an interval adaptive module, a key frame prediction module, and a non-key frame processing module, and improves the data output control module, adds a key frame analysis submodule, a bus state prediction submodule, and a frame loss statistics submodule in the data output control module, and improves the read control submodule. In the embodiment of the present invention, adaptive prediction and recognition of key frames are realized, and the discrimination parameters are dynamically adjusted to improve the discrimination accuracy, while improving the complete write rate of the on-chip storage space of key frames, and realizing autonomous frame loss for non-key frames, reducing the occupation of memory bandwidth and bus bandwidth by video functions, and improving the overall performance of the chip.

[0135] Specifically, the embodiment of the present invention realizes adaptive prediction of key frames, proposes a prediction formula (hardware judgment logic) of key frames and adaptive adjustment of parameters, which greatly improves the accuracy of prediction.

[0136] The embodiment of the present invention realizes adaptive adjustment of the detection interval, realizes rapid identification and processing from non-key frames to key frames, thereby ensuring that the remote end can obtain key frames with high quality, and can also realize recovery from key frames to non-key frames. On the basis of ensuring video quality, it minimizes the occupancy of the system bus and memory bus by the video function, thereby improving the overall performance of the chip.

[0137] The embodiment of the present invention processes non-key frames in the on-chip cache in advance according to the prediction information of the key frames and the bus status and other information, thereby improving the complete writing rate of the key frames in the on-chip cache space and improving the display quality of the remote end.

[0138] The embodiment of the present invention automatically detects the continuous frame loss rate of non-key frames, and autonomously adjusts the judgment calculation logic, thereby greatly reducing the impact of non-key frame loss processing on remote end display.

[0139] The following is an introduction to a device provided by an embodiment of the present invention. The device described below and the method described above can be referred to each other. The device of this embodiment is set in a server management control device. Figure 8 , Figure 8: is a structural block diagram of an apparatus of one embodiment of the present invention, comprising: a video capture module 210, used to obtain a first valid write count of a pixel point of a current video frame, the first valid write count indicating the total number of pixel point data with differences between the current video frame and the previous video frame; an interval adaptive module 220, used to determine a first target preset count range where the first valid write count is located according to a first preset count range; wherein, if any data in the first target preset count range is greater than the maximum data in other first preset count ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; the interval adaptive module 220, further used to adjust the frame interval according to the first target preset count range, the frame interval being the number of intervals between two adjacent determinations of the first valid write count; a non-key frame processing module 230, used to obtain a bus state; if the bus state is a busy state and the current video frame is not a key video frame, the processing of the current video frame is stopped; otherwise, the video processing module is triggered; a video processing module 240, used to perform image compression processing on the current video frame; a data output control module 250, used to write the compressed data into an on-chip cache.

[0140] In one achievable manner, the interval adaptation module 220 is also used for: if the first target preset number range is the preset number range of the minimum value, then determining that the frame interval remains unchanged, wherein the frame interval is the number of intervals between two adjacent determinations of the first effective write number; if the first target preset number range is the preset number range of the intermediate value, then obtaining the average effective number of second effective write times of each of a plurality of consecutive video frames; adjusting the frame interval according to the second target preset number range of the second preset number range in which the average effective number is located to obtain the first frame interval; if the first target preset number range is the preset number range of the maximum value, then determining that the frame interval is the second frame interval, the second frame interval is smaller than the frame interval, and smaller than the first frame interval.

[0141] In an achievable manner, the interval adaptive module 220 is further used to: determine an adjustment coefficient according to a second target preset number range of a second preset number range in which the average effective number of times is located; and obtain a first frame interval according to the adjustment coefficient and the frame interval.

[0142] In one implementable manner, the interval adaptation module 220 is also used to: if the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, then obtain multiple consecutive average effective times; if there is a target average effective number that is less than the minimum value of the first target preset number range, then increase the frame interval.

[0143] In one implementable manner, it also includes: a key frame prediction module, which is used to: obtain the third valid write times of each of the previous multiple video frames; determine whether the current video frame is a key video frame based on the third valid write times of each of the previous multiple video frames; if it is a key video frame, determine in advance that the frame interval is the second frame interval.

[0144] In one implementable manner, the key frame prediction module is specifically used to: determine whether the effective write times are gradually increasing based on the third effective write times of each of the previous multiple video frames; if it is gradually increasing, and the first effective write times of the current video frame is within the set range of the preset maximum times threshold, then determine that the current video frame is a key video frame; otherwise, determine that the current video frame is not a key video frame.

[0145] In one implementable manner, the key frame prediction module is specifically used to: obtain the difference in the number of third valid write times of two adjacent video frames; for two adjacent number differences, determine whether a first value corresponding to the previous number difference is less than a second value corresponding to the subsequent number difference, and obtain a determination result; the first value is obtained based on the previous number difference and the corresponding first weight, and the second value is obtained based on the subsequent number difference and the corresponding second weight; based on the determination results of all adjacent number differences, determine whether the third valid write number gradually increases, wherein if all the determination results are yes, it is determined that the third valid write number gradually increases.

[0146] In one implementable manner, the key frame prediction module is specifically used to: if the current video frame is determined to be a key video frame based on the first valid write count and the third valid write count, then increase the second weight; if the current video frame is determined not to be a key video frame based on the first valid write count, and if the current video frame is determined to be a key video frame based on the third valid write count, then increase the first weight.

[0147] In one achievable manner, the non-key frame processing module 230 is used to stop processing the current video frame if the bus state is busy and it is determined that the current video frame is not a key video frame based on the first valid write count and the third valid write count.

[0148] In one implementable manner, the data output control module 250 includes: a read control submodule, which is used to: obtain the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data to the on-chip cache, and the compressed length of the key frame; determine whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data to the on-chip cache, and the compressed length of the key frame; if the compressed data can be completely written into the on-chip cache, write the compressed data into the on-chip cache.

[0149] In one implementable manner, the data output control module 250 includes: a read control submodule, which is specifically used to: determine a third value based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data writing the on-chip cache, and the third weight; determine a fourth value based on the compression length of the key frame and the second write rate of the frame data writing the on-chip cache; determine whether the compressed data can be completely written into the on-chip cache based on the sizes of the third value and the fourth value, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

[0150] In one achievable manner, the data output control module 250 further includes: a bus status judgment submodule, which is used to: obtain the number of signal validity times during which the write data valid signal of the interactive interface of the double data rate memory and the write data permission indication signal output by the bus of the double data rate memory are simultaneously valid within a preset time period; and determine the first write rate of the double data rate memory according to the signal validity times and the preset time period.

[0151] In one achievable manner, the data output control module 250 further includes: a bus status judgment submodule, which is also used to: if the first write rate is less than a preset rate threshold, determine that the bus status is a busy state and reduce the preset time period; if the first write rate is not less than the preset rate threshold, increase the preset time period.

[0152] In one implementable manner, the read control submodule is also used to: if the on-chip cache cannot be completely written and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if the on-chip cache cannot be completely written and the current video frame is not a key video frame, the compressed data will be discarded.

[0153] In one implementable manner, the data output control module 250 also includes: a frame loss statistics submodule, which is used to: obtain the total amount of discarded non-critical frame data discarded in the on-chip cache; if the total amount of discarded data reaches a preset total amount of discarded data threshold, reduce the third weight; after reducing the third weight, if it is detected that a preset amount of non-critical frame data has not been discarded, reset the third weight.

[0154] Figure 8 The description of the features in the corresponding embodiments can be found in Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0155] Fig. 9 A structural diagram of a server management control device provided by an embodiment of the present invention, such as Fig. 9As shown, the server management control device includes: a memory 60 for storing a computer program; a processor 61 for implementing the steps of the video frame processing method of the above embodiment when executing the computer program.

[0156] The server management and control device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0157] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0158] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein the computer program, after being loaded and executed by the processor 61, can implement the relevant steps of the video frame processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc.

[0159] In some embodiments, the server management control device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0160] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the server management control device, and may include more or fewer components than those shown in the figure.

[0161] It is understandable that if the video frame processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the contribution to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program codes.

[0162] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video frame processing method are implemented.

[0163] An embodiment of the present invention further provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned video frame processing method when executed by a processor.

[0164] The above is a detailed introduction to a video frame processing method, device, equipment, medium and product provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0165] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0166] The above is a detailed introduction to a video frame processing method, device, equipment, medium and product provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A video frame processing method, characterized in that: include: Obtaining a first valid write count of pixels of a current video frame, where the first valid write count represents a total number of pixel data having differences between the current video frame and a previous video frame; According to the first preset number range, determining the first target preset number range in which the first valid write number is located; wherein, if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; According to the first target preset number range, adjusting the frame interval, the frame interval being the number of intervals between two adjacent first valid write times; Obtaining a bus state; if the bus state is a busy state and the current video frame is not a key video frame, stopping processing of the current video frame; Otherwise, image compression processing is performed on the current video frame, and the compressed data is written into an on-chip cache.

2. The video frame processing method according to claim 1, characterized in that: Adjusting the frame interval according to the first target preset number range includes: If the first target preset number range is the preset number range of the minimum value, then determining that the frame interval remains unchanged; If the first target preset number range is a preset number range of an intermediate value, then obtaining an average effective number of second effective write times of each of a plurality of consecutive video frames; adjusting the frame interval according to the second target preset number range of the second preset number range in which the average effective number of times is located, to obtain a first frame interval; If the first target preset number range is the preset number range with the largest value, the frame interval is determined to be the second frame interval, and the second frame interval is smaller than the frame interval and smaller than the first frame interval.

3. The video frame processing method according to claim 2, characterized in that: Adjusting the frame interval according to a second target preset number range of the second preset number range in which the average effective number of times is located includes: Determine an adjustment coefficient according to a second target preset number range of the second preset number range in which the average effective number of times lies; According to the adjustment coefficient and the frame interval, a first frame interval is obtained; And / or, when the first preset number range is the same as the second preset number range, the method further includes: If the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, then obtaining a plurality of consecutive average effective times; If there is a target average effective number of times that is less than the minimum value of the first target preset number range, the frame interval is increased.

4. The video frame processing method according to claim 1, characterized in that: Before determining whether the current video frame is a key video frame according to the first valid write count, the method further includes: Obtaining the third valid write times of each of the previous plurality of video frames; Determining whether the current video frame is a key video frame according to respective third valid write times of a plurality of previous video frames; If it is a key video frame, the frame interval is determined in advance to be the second frame interval.

5. The video frame processing method according to claim 4, characterized in that: Determining whether the current video frame is a key video frame according to respective third valid write times of a plurality of previous video frames includes: determining whether the valid writing times gradually increase according to respective third valid writing times of a plurality of previous video frames; If it gradually increases and the first valid writing times of the current video frame is within the set range of the preset maximum times threshold, the current video frame is determined to be a key video frame; otherwise, the current video frame is determined not to be a key video frame.

6. The video frame processing method according to claim 5, characterized in that: Determining whether the third valid writing times gradually increases according to respective third valid writing times of a plurality of previous video frames includes: Obtain the difference between the third valid write times of two adjacent video frames; For two adjacent number of times difference, determine whether a first value corresponding to the previous number of times difference is less than a second value corresponding to the next number of times difference, and obtain a determination result; the first value is obtained based on the previous number of times difference and the corresponding first weight, and the second value is obtained based on the next number of times difference and the corresponding second weight; According to the determination results of all the differences between two adjacent times, it is determined whether the third valid writing times gradually increases, wherein if all the determination results are yes, it is determined that the third valid writing times gradually increases.

7. The video frame processing method according to claim 6, characterized in that: Also includes: If the current video frame is determined to be a key video frame according to the first valid writing number and the third valid writing number, the second weight is increased; If it is determined that the current video frame is not a key video frame according to the first valid writing number, and if it is determined that the current video frame is a key video frame according to the third valid writing number, the first weight is increased.

8. The video frame processing method according to claim 4, characterized in that: If the bus state is a busy state and the current video frame is not a key video frame, then the processing of the current video frame is stopped, including: If the bus state is a busy state, and according to the first valid write count and the third valid write count, it is determined that the current video frame is not a key video frame, then the processing of the current video frame is stopped.

9. The video frame processing method according to any one of claims 1 to 8, characterized in that: Before writing the compressed data into the on-chip cache, it also includes: Obtaining remaining space of the on-chip cache, a first write rate of the double data rate memory, a second write rate of writing frame data into the on-chip cache, and a compressed length of the key frame; determining whether the compressed data can be completely written into the on-chip cache according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; If the on-chip cache can be completely written, the compressed data is written to the on-chip cache.

10. The video frame processing method according to claim 9, characterized in that: Determining whether the compressed data can be completely written into the on-chip cache according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of writing the frame data into the on-chip cache, and the compressed length of the key frame, including: Determine a third value according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of writing the frame data into the on-chip cache, and the third weight; Determine a fourth value according to the compressed length of the key frame and a second write rate at which the frame data is written into the on-chip cache; Whether the compressed data can be completely written into the on-chip cache is determined according to the sizes of the third value and the fourth value, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

11. The video frame processing method according to claim 9, characterized in that: Obtaining a first write rate of a double data rate memory, comprising: Acquire the number of times that a write data valid signal of the interactive interface of the double data rate memory and a write data permission indication signal output by the bus of the double data rate memory are simultaneously valid within a preset time period; A first write rate of the double data rate memory is determined according to the number of valid signal times and a preset time period.

12. The video frame processing method according to claim 11, characterized in that: Also includes: If the first write rate is less than a preset rate threshold, determining that the bus state is a busy state, and reducing the preset time period; If the first writing rate is not less than a preset rate threshold, the preset time period is increased.

13. The video frame processing method according to claim 10, characterized in that: Also includes: If the on-chip cache cannot be completely written, and the current video frame is a key video frame, non-key frame data in the on-chip cache will be read and discarded in units of frames; If the on-chip cache cannot be completely written, and the current video frame is not a key video frame, the compressed data is discarded.

14. The video frame processing method according to claim 13, characterized in that: Also includes: Get the total amount of discarded non-key frame data in the on-chip cache; If the total discarded amount reaches a preset total discarded amount threshold, reducing the third weight; After the third weight is reduced, if it is detected that a preset amount of non-key frame data is not discarded, the third weight is reset.

15. A video frame processing device, characterized in that: include: A video capture module, used to obtain a first valid write count of pixels of a current video frame, wherein the first valid write count represents a total number of pixel data having differences between the current video frame and a previous video frame; an interval adaptive module, configured to determine, according to a first preset number range, a first target preset number range in which the first valid write number is located; wherein, if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; The interval adaptive module is further used to adjust the frame interval according to the first target preset number range, and the frame interval is the number of intervals between two adjacent first valid write times; A non-key frame processing module, used for obtaining a bus state; if the bus state is a busy state and the current video frame is not a key video frame, the processing of the current video frame is stopped; otherwise, the video processing module is triggered; A video processing module, used for performing image compression processing on the current video frame; The data output control module is used to write the compressed data into the on-chip cache.

16. A server management and control device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the video frame processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video frame processing method according to any one of claims 1 to 14 are implemented.

18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the video frame processing method according to any one of claims 1 to 14 are implemented.

Images