Video frame processing method, device, equipment, medium and product

Abstract

The present invention relates to the technical field of data processing, and in particular to a video frame processing method, apparatus, device, medium and product. The method comprises: obtaining a first valid write count of a pixel point of a current video frame, and being able to identify the degree of difference between the current video frame and a previous frame; judging whether the current video frame is a key video frame based on the first valid write count, and dynamically adjusting the frame interval obtained by the next first valid write count, thereby improving the video frame processing efficiency; stopping processing the current video frame when confirming that the bus state is busy and the current frame is not a key frame, effectively avoiding the problem that non-key frames occupy bandwidth resources when bus resources are tight, thereby causing key frames to be lost due to insufficient resources; for non-busy states or key video frames, performing image compression processing and writing them into an on-chip cache, which can save bandwidth and ensure the timely transmission of key frames, thereby significantly improving the display quality of the remote end.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a video frame processing method, device, equipment, medium and product. Background Art

[0002] Typically, video data is processed using the video processing function on the management and control chip. The video processing flow in this server management and control chip is as follows: The original video data from the host is transmitted via PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) to the VGA (Video Graphics Array) module inside the server management and control chip. The VGA generates original video data in RGB format. This data is then converted to a YUV format through color space conversion. The YUV data is then converted to blocks and compressed by the video core (e.g., H.264 or JPEG) to obtain compressed video data. The compressed data is then written to an external DDR (Double Data Rate) memory through the output control module.

[0003] In the related art, many other functional modules and software programs in the server management control chip need to occupy the memory bus and operate the memory DDR. Therefore, the output control module will not always obtain control of the bus when outputting video data, while the host-side video data is continuously generated. Since the video data cannot be output in time, a large number of frame drops occur in the output control module, especially the drop of key frames, which greatly reduces the display quality of the remote end. At the same time, the large number of non-key frames transmitted is a huge waste of the chip's memory bandwidth and bus bandwidth, resulting in low video combing efficiency and poor video quality. Summary of the Invention

[0004] The purpose of the present invention is to provide a video frame processing method, device, equipment, medium and product, which can effectively avoid the problem that non-key frames occupy bandwidth resources when bus resources are tight, thereby causing key frames to be lost due to insufficient resources; and can ensure the timely and rapid transmission of key frames, thereby significantly improving the display quality of the remote end.

[0005] In a first aspect, a video frame processing method is provided, comprising: obtaining a first valid write count of a pixel point of a current video frame, the first valid write count indicating the total number of pixel point data that differs between the current video frame and a previous video frame; determining a first target preset count range within which the first valid write count lies based on a first preset count range; wherein, if any data in the first target preset count range is greater than the maximum data in other first preset count ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; adjusting a frame interval based on the first target preset count range, the frame interval being the number of intervals between two adjacent determinations of the first valid write count; obtaining a bus status; if the bus status is a busy state and the current video frame is not a key video frame, stopping processing of the current video frame; otherwise, performing image compression processing on the current video frame, and writing the compressed data into an on-chip cache.

[0006] In a preferred example, the present invention can be further configured as follows: adjusting the frame interval according to the first target preset number range, the frame interval being the number of intervals between two adjacent first valid write times, including: if the first target preset number range is the preset number range of the minimum value, then determining that the frame interval remains unchanged, wherein the frame interval is the number of intervals between two adjacent first valid write times; if the first target preset number range is the preset number range of the intermediate value, then obtaining the average effective number of second valid write times of each of multiple consecutive video frames; adjusting the frame interval according to the second target preset number range of the second preset number range where the average effective number is located to obtain the first frame interval; if the first target preset number range is the preset number range of the largest value, then determining that the frame interval is the second frame interval, the second frame interval is smaller than the frame interval, and smaller than the first frame interval.

[0007] In a preferred example, the present invention may be further configured to: adjust the frame interval according to a second target preset number range of the second preset number range in which the average effective number of times is located, including: determining an adjustment coefficient according to the second target preset number range of the second preset number range in which the average effective number of times is located; and obtaining a first frame interval according to the adjustment coefficient and the frame interval;

[0008] And / or, when the first preset number range and the second preset number range are the same, the method also includes: if the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, obtaining multiple consecutive average effective times; if there is a target average effective number that is less than the minimum value of the first target preset number range, increasing the frame interval.

[0009] In a preferred example, the present invention can be further configured as follows: before determining whether the current video frame is a key video frame based on the first valid write count, it also includes: obtaining the third valid write count of each of the previous multiple video frames; determining whether the current video frame is a key video frame based on the third valid write count of each of the previous multiple video frames; if it is a key video frame, determining in advance that the frame interval is the second frame interval.

[0010] In a preferred example, the present invention can be further configured as follows: determining whether the current video frame is a key video frame based on the third valid write times of each of the previous multiple video frames, including: determining whether the valid write times gradually increase based on the third valid write times of each of the previous multiple video frames; if it gradually increases, and the first valid write times of the current video frame is within the set range of the preset maximum times threshold, determining that the current video frame is a key video frame; otherwise, determining that the current video frame is not a key video frame.

[0011] In a preferred example, the present invention can be further configured as follows: based on the third valid write times of each of the previous multiple video frames, determining whether the third valid write times gradually increases, including: obtaining the difference in the third valid write times of two adjacent video frames; for the two adjacent number differences, determining whether the first value corresponding to the previous number difference is less than the second value corresponding to the subsequent number difference, and obtaining a determination result; the first value is obtained based on the previous number difference and the corresponding first weight, and the second value is obtained based on the subsequent number difference and the corresponding second weight; based on the determination results of all the adjacent number differences, determining whether the third valid write times gradually increases, wherein if all the determination results are yes, it is determined that the third valid write times gradually increases.

[0012] In a preferred example, the present invention can be further configured as follows: it also includes: if the current video frame is determined to be a key video frame based on the first valid write count and the third valid write count, then the second weight is increased; if the current video frame is determined not to be a key video frame based on the first valid write count, and the current video frame is determined to be a key video frame based on the third valid write count, then the first weight is increased.

[0013] In a preferred example, the present invention can be further configured as follows: if the bus state is a busy state and the current video frame is not a key video frame, then the processing of the current video frame is stopped, including: if the bus state is a busy state and based on the first valid write count and the third valid write count, it is determined that the current video frame is not a key video frame, then the processing of the current video frame is stopped.

[0014] In a preferred example, the present invention can be further configured as follows: before writing the compressed data into the on-chip cache, it also includes: obtaining the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; determining whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame; if it can be completely written into the on-chip cache, writing the compressed data into the on-chip cache.

[0015] In a preferred example, the present invention can be further configured as follows: determining whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame, including: determining a third value based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the third weight; determining a fourth value based on the compressed length of the key frame and the second write rate of the frame data into the on-chip cache; determining whether the compressed data can be completely written into the on-chip cache based on the sizes of the third and fourth values, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

[0016] In a preferred example, the present invention can be further configured to: obtain a first write rate of a double data rate memory, including: obtaining the number of times a write data valid signal of an interactive interface of the double data rate memory and a write data enable indication signal output by a bus of the double data rate memory are simultaneously valid within a preset time period; and determining the first write rate of the double data rate memory based on the number of signal valid times and the preset time period.

[0017] In a preferred example, the present invention can be further configured as follows: it also includes: if the first write rate is less than the preset rate threshold, determining that the bus state is a busy state and reducing the preset time period; if the first write rate is not less than the preset rate threshold, increasing the preset time period.

[0018] In a preferred example, the present invention can be further configured as follows: it also includes: if the on-chip cache cannot be completely written and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if the on-chip cache cannot be completely written and the current video frame is not a key video frame, the compressed data will be discarded.

[0019] In a preferred example, the present invention can be further configured as follows: it also includes: obtaining the total amount of discarded non-critical frame data in the on-chip cache; if the total amount of discarded data reaches a preset total amount of discarded data threshold, lowering the third weight; after lowering the third weight, if it is detected that a preset amount of non-critical frame data has not been discarded, resetting the third weight.

[0020] In a second aspect, a video frame processing device is provided, including: a video capture module, used to obtain a first valid write count of a pixel point of a current video frame, the first valid write count indicating the total number of pixel point data with differences between the current video frame and the previous video frame; an interval adaptation module, used to determine a first target preset count range within which the first valid write count lies based on a first preset count range; wherein, if any data in the first target preset count range is greater than the maximum data in other first preset count ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; the interval adaptation module is further used to adjust the frame interval based on the first target preset count range, the frame interval being the number of intervals between two adjacent determinations of the first valid write count; a non-key frame processing module, used to obtain a bus status; if the bus status is a busy state and the current video frame is not a key video frame, the processing of the current video frame is stopped; otherwise, the video processing module is triggered; the video processing module is used to perform image compression processing on the current video frame; and a data output control module is used to write the compressed data into an on-chip cache.

[0021] In a third aspect, a server management and control device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the video frame processing method described in any one of the first aspects.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the program code is loaded and executed by a processor to implement the video frame processing method as described in any one of the first aspects.

[0023] In a fifth aspect, a computer program product is provided, comprising a computer program or instructions, which, when executed by a processor, implements the video frame processing method as described in any one of the first aspects.

[0024] In summary, the video frame processing method provided by the present invention includes the following beneficial technical effects: obtaining the first valid write count of the pixel point of the current video frame, which can identify the degree of difference between the current video frame and the previous frame; judging whether the current video frame is a key video frame based on the number range of the first valid write count; and, since in monitoring mode, since the previous and subsequent frames of video basically do not change much, comparing each video frame with the previous video frame to determine the valid write count increases meaningless processing of the chip, resulting in low video frame processing efficiency. This solution also dynamically adjusts the frame interval obtained by the next first valid write count based on the preset number range of the first valid write count, thereby improving video frame processing efficiency; when it is confirmed that the bus state is busy and the current frame is not a key frame, processing of the current video frame is stopped, effectively avoiding the problem that non-key frames occupy bandwidth resources when bus resources are tight, thereby causing key frames to be lost due to insufficient resources; for non-busy or key video frames, image compression processing is performed and written to the on-chip cache, which can save bandwidth and ensure timely transmission of key frames, thereby significantly improving the display quality of the remote end.

[0025] In addition, the present invention also provides a video frame processing device, equipment, medium and product, all of which have the above-mentioned beneficial technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0027] Figure 1 This is a schematic diagram of the implementation of the video function in the server management control chip of the related technology.

[0028] Figure 2 This is a hardware processing flow chart of a video data output control module in related technology.

[0029] Figure 3 This is a schematic diagram of the BLOCK conversion process.

[0030] Figure 4 It is a flowchart of a video processing method provided by an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of coefficient correspondence provided by an embodiment of the present invention.

[0032] Figure 6 The present invention provides a schematic diagram of a video hardware processing flow of a server management and control device.

[0033] Figure 7 This is a hardware flow diagram of a data output control module provided by an embodiment of the present invention.

[0034] Figure 8 It is a structural diagram of a video processing device provided by an embodiment of the present invention.

[0035] Figure 9 It is a structural diagram of a server management and control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] The terms "including" and "having," as used in the present description and accompanying drawings, and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements and may include steps or elements that are not listed.

[0038] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0039] In order to better understand and illustrate the solutions of the embodiments of the present invention, some technical terms involved in the embodiments of the present invention are briefly explained below.

[0040] Server management and control equipment: refers to the server field. This chip is used to monitor the server status (temperature, fan, main CPU operation status, etc.). It also has an important function, which is to transmit local video information to the remote end through the network for remote display and monitoring.

[0041] SoC: System on Chip, or SoC for short. It's generally defined as a system that integrates a microprocessor, analog IP cores, digital IP cores, and memory (or an off-chip memory control interface) on a single chip. It's typically custom-built or a standard product for a specific application.

[0042] Compressed video controller: The original image information output by the local graphics card needs to be compressed before being transmitted to the remote end through the network. The purpose is to reduce the amount of video data and reduce the network bandwidth.

[0043] See also Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of the implementation of the video function in the server management control chip of the related technology. Figure 2 This is the hardware processing flow of the video data output control module in the related technology.

[0044] The processing process of the video processing function in the related art is:

[0045] 1: The original video data from the server host (operating system interface) is transmitted via PCIe to the VGA module inside the server management control chip. The VGA generates raw video data in RGB format (in the data generation process, the VGA needs to interact with the off-chip DDR).

[0046] 2: The video capture module obtains video data in frames through the DVI (Digital Visual Interface) interface of VGA.

[0047] 3: Color space conversion module (RGB2YUV module), according to the matrix conversion formula, realizes color space conversion and converts RGB format video data into YUV format video data.

[0048] The matrix conversion formula is as follows:

[0049] .

[0050] .

[0051] .

[0052] 4: The block generation conversion module realizes the BLOCK conversion of YUV data. According to the YUV444 or YUV422 / YUV420 format, the size of BLOCK is , the specific BLOCK conversion process is as follows Figure 3 .

[0053] 5: The video core compression module receives YUV data in BLOCK format and then performs video compression. There are many compression formats, the most common ones are JPEG / H.264, etc. The compressed data is output to the video output control module.

[0054] 6: The video output control module receives compressed video data, generates control logic, and writes the compressed data into DDR.

[0055] like Figure 2As shown, the compressed data cache control submodule controls the writing of compressed data to the on-chip cache, which is typically implemented using deep RAM. The write status cache control submodule stores the write status of each frame of compressed data in the on-chip cache, i.e., complete write and incomplete write. The read control submodule controls the reading of compressed data from the on-chip cache. The interface timing control submodule generates the AXI (Advanced eXtensible Interface) interface protocol to write compressed data to the off-chip DDR. The frame status generation submodule stores the status information of each frame written to the DDR, such as the frame header DDR address, frame trailer DDR address, and frame length, for the software driver to read.

[0056] 7: The network module driver reads the DDR to obtain compressed data and sends it to the remote end through the network for display.

[0057] 8: Remote users have two operation modes, one is monitoring mode and the other is management mode. In monitoring mode, users do not care about the specific information of each frame, while in management mode, users want to obtain the video screen of the local operating system as soon as possible.

[0058] In the related art, many other functional modules and software programs in the server management control chip need to occupy the memory bus and operate the memory DDR. Therefore, the output control module will not always obtain control of the bus when outputting video data, while the host-side video data is continuously generated. Since the video data cannot be output in time, a large number of frame drops occur in the output control module, especially the drop of key frames, which greatly reduces the display quality of the remote end. At the same time, the large number of non-key frames transmitted is a huge waste of the chip's memory bandwidth and bus bandwidth.

[0059] In this context, it is necessary and urgent to optimize the video frame loss function of the server management control chip, reduce the frame loss rate of key frames, discard some non-key frames, improve the remote display quality and human-computer interaction experience, and at the same time reduce the memory bandwidth and bus bandwidth occupied by the video function, and improve the overall performance of the chip.

[0060] Based on this, the present invention relates to a hardware design method for optimizing key frame processing of a server management control chip, aiming to improve the drawbacks of traditional solutions, reduce the key frame loss rate, improve the display quality of the remote end, reduce the memory bandwidth and bus bandwidth occupied by video functions, optimize the overall chip performance, and improve user experience.

[0061] Specifically, Figure 4 A flowchart of a video frame processing method provided in an embodiment of the present invention is provided. The method provided in an embodiment of the present invention can be executed by a server management control device. The video frame processing method includes:

[0062] S101: Obtain a first valid write count of a pixel point in a current video frame.

[0063] The first valid write count indicates the total number of pixel data that differ between the current video frame and the previous video frame. This can be achieved by detecting the write address and write data interface of the VGA and counting the number of valid write addresses and write data in one frame, updata_cnt, as the first valid write count.

[0064] In an embodiment of the present invention, a video capture module is used to obtain the first valid write count of a pixel point in the current video frame. The video capture module has a new data comparison function, which detects the behavior of the interface bus between the VGA module and the DDR (VGA display memory) in units of frames. Figure 5 The A space of the off-chip DDR is the video memory space of the VGA. The behavior of the VGA is to only update the data of the same pixel position in the current frame that has changed compared to the previous frame. For the data of the same pixel position in the previous and next frames that has not changed, the VGA will not update its data. For example, if the pixel data of the 10th row and 12th column of the 1st frame is consistent with the pixel data of the 10th row and 12th column of the 0th frame, then when the VGA updates the video memory data of the 1st frame, the pixel data will not be updated, that is, the VGA will not send the write address and write data of the pixel position. Compared with the related art, the video capture module of the present invention has a new function of detecting the write address and write data interface of the VGA in units of frames, and counting the number of valid write addresses and write data in 1 frame, updata_cnt. This value reflects the changed pixel values of the subsequent frame compared to the previous frame. The frame as a unit is also a new function, that is, the data comparison function of the capture module needs to use the first and last addresses of the VGA video memory as the range to detect and calculate the updata_cnt described above. The value of updata_cnt is passed to Figure 5 The key frame identification module in the video frame can at least determine whether the current video frame is a key video frame.

[0065] S102: Determine a first target preset number range within which a first valid writing number is located according to the first preset number range.

[0066] Among them, if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; among them, the key video frame refers to a frame with significant changes in the video sequence.

[0067] In an embodiment of the present invention, the interval adaptation module can determine whether a video frame is a key frame based on the first valid write count. In some embodiments, the first valid write count can be compared with a preset key frame determination threshold. If the number is greater than the key frame determination threshold, the current video frame is determined to be a key frame.

[0068] Specifically, the first preset number range is a pre-set range and can be composed of multiple thresholds, such as (0, a1], [a1, a2), [a2, +∞), a1<a2; through the adaptive adjustment module, the first target preset number range is determined according to the first valid write number; if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the range it is in is the largest range such as [a2, +∞), indicating that the current video frame is a key video frame; otherwise, it is considered that the current video frame is not a key video frame.

[0069] As can be seen, in this embodiment of the present invention, the first valid write count is compared with multiple preset count ranges to determine the first target preset count range to which the first valid write count belongs. If the data within this range is significantly greater than the maximum value of the other ranges, that is, the range with the largest value, the current video frame is determined to be a key video frame, which can more accurately identify key frames containing significant visual changes.

[0070] S103 , adjusting the frame interval according to the first target preset number range, where the frame interval is the number of intervals between two adjacent first valid write times.

[0071] Furthermore, the adaptive adjustment module can also include adaptively adjusting the frame interval for comparing previous and next frames. Because VGA interface behavior detection and data comparison are not required for every frame, in real-world server management and control chip applications, especially when the local server is in monitoring mode, the server host operating system interface remains largely unchanged. That is, there will be no significant changes between previous and next frames, or even between a dozen or even dozens of frames. Therefore, there is no need to compare and detect every frame with the previous one, as this would increase chip power consumption and result in meaningless processing, reducing chip performance.

[0072] Therefore, the present invention designs an interval adaptation module that calculates and outputs the frame interval number (frame_period) used by the video capture module for data detection and comparison. This is achieved by first using a default frame interval (N), which can be either the default value or user-configured via registers. Typically, the default value for N is 8 to 15, such as 8, 10, 11, or 15. Simultaneously, the capture module's output, updata_cnt, is detected.

[0073] Specifically, if the first target preset number range is the preset number range of the minimum value, the frame interval is determined to be unchanged; if the first target preset number range is the preset number range of the middle value, the average effective number of the second effective write times of each of the multiple consecutive video frames is obtained; according to the second target preset number range of the second preset number range where the average effective number is located, the frame interval is adjusted to obtain the first frame interval; if the first target preset number range is the preset number range of the maximum value, the frame interval is determined to be the second frame interval; wherein the frame interval is the number of intervals between two adjacent determinations of the first effective write times, and the second frame interval is less than the frame interval, and less than the first frame interval.

[0074] Among them, the preset number range of the minimum value represents the range of the minimum value. For example, the preset number range includes: (0, a0], [a0, a1), [a1, a2), [a2, +∞), where a0<a1<a2, then the preset number range of the minimum value is (0, a0]; the maximum preset number range is [a2, +∞); the preset number range of the intermediate value includes: [a0, a1), [a1, a2).

[0075] The adjustment strategy of the N value is that when the value of the first valid write count updata_cnt is less than or equal to the threshold a0, that is, it belongs to the preset count range with the smallest value, N is not adjusted; when the value of the first valid write count updata_cnt is greater than the threshold a2, that is, it belongs to the preset count range with the largest value, it means that the change between frames is very large, and N is directly set to the second frame interval, which can be 1, that is, the data capture module requires continuous comparison and detection of adjacent frames; when the value of the first valid write count updata_cnt is greater than the threshold a0 and less than or equal to the threshold a1, or when it is greater than the threshold a1 and less than or equal to the threshold a2, it starts to continuously detect the average value updata_cnt_average of the values of multiple frames of updata_cnt as the average valid number, and adjusts the frame interval according to the second target preset number range of the second preset number range where the average valid number is located to obtain the first frame interval.

[0076] The second preset number range includes multiple ranges. The second preset number range can be the same as or different from the first preset number range. The present invention is not limited thereto, as long as the purpose of the present invention can be achieved. Exemplarily, the second preset number range includes: (0, a10], [a10, a11), [a11, a12), [a12, +∞).

[0077] It is understood that the first frame interval is greater than the second frame interval and less than the default frame interval N; and the greater the average effective number of times, the smaller the first frame interval. The embodiment of the present invention does not limit the number of consecutive multiple video frames, and the user can set it according to actual needs.

[0078] S104, obtaining the bus status; if the bus status is busy and the current video frame is not a key video frame, stopping processing the current video frame.

[0079] The bus state includes a busy state and a non-busy state. In an embodiment of the present invention, the non-key frame processing module obtains a feedback signal (bus_state) from the data output control module, i.e., the DDR bus state. When the signal bus_state = 2'b10, it indicates that the current DDR bus state is very busy and the data output control module cannot write data to the DDR in a timely manner. At the same time, if it is determined that the current video frame is not a key video frame, the non-key frame processing module will output a control signal to the RGB2YUV module to stop subsequent processing of the current non-key frame. In other words, the signal stop_process_cur_frame is pulled high to stop processing the current video frame. In other words, the current non-key frame does not undergo subsequent RGB2YUV, block processing, compression, transmission, and other processing. This reduces the amount of data written to the DDR by the data output control module, reduces the bus occupancy of the chip video function without affecting the remote end video display quality, and improves the overall performance of the chip.

[0080] Furthermore, this embodiment of the present invention uses real-time frame-by-frame detection. When the bus_state is not 2'b10 (busy) or the current frame is a key frame, the aforementioned frame drop mode is exited, and the current video frame is compressed and transmitted normally. This function allows non-key frames to be bypassed for video compression and other processing when the system bus is extremely busy. This also includes compressing both non-key frames and key frames when the system bus is normal, or compressing key frames normally when the system bus is busy.

[0081] S105: Otherwise, perform image compression processing on the current video frame.

[0082] If the bus state is busy and the current video frame is a key video frame, image compression processing is performed on the current video frame; if the bus state is non-busy, regardless of whether the current video frame is a key video frame or not, image compression processing is performed on the current video frame through the video processing module.

[0083] The video processing module includes: RGBYUV module, block generation module and video core compression module.

[0084] S106 , writing the compressed data into the on-chip cache.

[0085] In the embodiment of the present invention, the compressed data is written into the on-chip cache through the data output control module.

[0086] It can be seen that in the embodiment of the present invention, the first valid write count of the pixel point of the current video frame is obtained, and the degree of difference between the current video frame and the previous frame can be identified; based on the number range of this first valid write count, it is determined whether the current video frame is a key video frame; and, since in monitoring mode, the previous and subsequent frames of video basically do not change much, comparing each video frame with the previous video frame to determine the valid write count increases meaningless processing of the chip, and the video frame processing efficiency is low. This solution also dynamically adjusts the frame interval obtained by the next first valid write count according to the preset number range of the first valid write count, thereby improving the video frame processing efficiency; when it is confirmed that the bus state is busy and the current frame is not a key frame, the processing of the current video frame is stopped, effectively avoiding the problem that non-key frames occupy bandwidth resources when bus resources are tight, thereby causing key frames to be lost due to insufficient resources; for non-busy states or key video frames, image compression processing is performed and written to the on-chip cache, which can save bandwidth and ensure timely transmission of key frames, thereby significantly improving the display quality of the remote end.

[0087] In one feasible method, the frame interval is adjusted according to the second target preset number range of the second preset number range in which the average effective number of times is located, including: determining an adjustment coefficient according to the second target preset number range of the second preset number range in which the average effective number of times is located; and obtaining the first frame interval according to the adjustment coefficient and the frame interval.

[0088] In an embodiment of the present invention, different second preset number ranges are provided with different adjustment coefficients, and the adjustment coefficient is less than 1. If the value of the second preset number range (i.e., the second target preset number range) where the average effective number is located is larger, it means that the difference between the previous and next frames is larger, and it may tend to be a key frame, then the corresponding adjustment coefficient is smaller, so as to reduce the frame interval and perform comparisons more frequently to ensure that no key frames are missed.

[0089] In one possible embodiment, the adjustment coefficient can be determined based solely on the second target preset number range within the second preset number range within which the average effective number of times falls. The larger the value of the second preset number range, the smaller the adjustment coefficient. For example, each second preset number range corresponds to an adjustment coefficient, and the first frame interval can be directly determined by multiplying the adjustment coefficient by the frame interval.

[0090] In another possible embodiment, each first preset number range corresponds to a first coefficient, and each second preset number range corresponds to a second coefficient adjustment coefficient. The first frame interval can be obtained by calculating the first coefficient * adjustment coefficient * frame interval, wherein the larger the value of the first preset number range, the smaller the first coefficient.

[0091] For example, see Figure 5 When the value of updata_cnt is greater than threshold a0 and less than or equal to threshold a1, the average valid number of updata_cnt values in multiple frames, updata_cnt_average, is continuously detected. If the value of updata_cnt_average is greater than threshold a10 and less than or equal to threshold a11, N is adjusted to a smaller value by N*coefficient 1 (coefficient 1 is the adjustment coefficient of (a0, a1])*coefficient 3, where coefficient 1 ≤ 1 and coefficient 3 < 1. If the value of updata_cnt_average is greater than threshold a11 and less than or equal to threshold a12, N is adjusted to a smaller value by N*coefficient 1*coefficient 4, where coefficient 4 < coefficient 3 < 1. If the value of updata_cnt_average is greater than threshold a12, N is adjusted to the second frame interval.

[0092] When the value of updata_cnt is greater than threshold a1 and less than or equal to threshold a2, the average value updata_cnt_average of the updata_cnt values of multiple frames is continuously detected. If the value of updata_cnt_average is greater than threshold a11 and less than or equal to threshold a12, N is continuously adjusted to a smaller value by N*coefficient 2 (coefficient 2 is the adjustment coefficient for [a1, a2], which is less than the adjustment coefficient for [a0, a1])*adjustment coefficient 6; adjustment coefficient 6 < 1. If the value of updata_cnt_average is greater than threshold a10 and less than or equal to threshold a11, N is continuously adjusted to a smaller value by N*coefficient 2*coefficient 5. If the value of updata_cnt_average is greater than threshold a12, N is adjusted to the second frame interval, where coefficient 5 is greater than coefficient 6.

[0093] In the embodiment of the present invention, the user can set coefficient 1, coefficient 2, coefficient 3, coefficient 4, coefficient 5 and coefficient 6 according to actual needs, as long as the purpose of the present invention can be achieved.

[0094] Furthermore, the setting of the frame interval N in the embodiment of the present invention can be synchronously detected and set in two directions. When the first preset number range and the second preset number range are the same, that is, when the value of updata_cnt_average reaches a threshold higher than the first valid write number, the result of continued detection is that updata_cnt_average changes toward a smaller threshold, then N is synchronously adjusted to increase the N value.

[0095] Specifically, when the first preset number range and the second preset number range are the same, the method also includes: if the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, then obtaining multiple consecutive average effective times; if there is a target average effective number that is less than the minimum value of the first target preset number range, then increasing the frame interval.

[0096] Exemplarily, when the value of updata_cnt_average reaches a threshold value higher than the first valid write count, and the result of continued detection is that updata_cnt_average changes toward a smaller threshold value, N is synchronously adjusted to increase the value of N.

[0097] When the value of updata_cnt is greater than the threshold a1 and less than or equal to the threshold a2, start continuously detecting the average value updata_cnt_average of the updata_cnt values of multiple frames. If the value of updata_cnt_average starts to be less than the threshold a1, adjust N in the direction of increase or keep N unchanged and continue detecting.

[0098] In an embodiment of the present invention, the bidirectional detection function enables this solution to achieve rapid identification and processing from non-key frames to key frames, thereby ensuring that the remote end can obtain key frames with high quality, and can also achieve recovery from key frames to non-key frames. On the basis of ensuring video quality, it minimizes the occupation of the system bus and memory bus by the video function, thereby improving the overall performance of the chip.

[0099] Furthermore, before determining whether the current video frame is a key video frame based on the first valid write count, it also includes: obtaining the third valid write count of each of the previous multiple video frames; determining whether the current video frame is a key video frame based on the third valid write count of each of the previous multiple video frames; if it is a key video frame, determining in advance that the frame interval is the second frame interval.

[0100] If it is not a key video frame, no operation is required.

[0101] In the embodiment of the present invention, key frames are predicted by a key frame prediction module, and the basic principle is to determine the number of valid write times corresponding to each of a plurality of previous video frames.

[0102] Specifically, based on the rate of change of the valid write count updata_cnt of multiple video frames, it is possible to predict in advance whether updata_cnt is about to reach a maximum threshold, such as a2. If the prediction result is that the maximum threshold is about to be reached, the frame interval N value is adjusted in advance, that is, the number of frame intervals for data detection and comparison is adjusted in advance.

[0103] It can be seen that in an embodiment of the present invention, before judging based on the first valid write count, the third valid write count of the previous multiple video frames is additionally considered; the attribute parameters of the current frame are predicted by analyzing historical data; if the prediction result shows that the current frame is a key video frame, the frame interval is adjusted to the second frame interval in advance, further reducing the risk of key frame loss.

[0104] In one achievable embodiment, determining whether a current video frame is a key video frame based on the third valid write counts of each of the plurality of prior video frames includes: determining whether the valid write counts are gradually increasing based on the third valid write counts of each of the plurality of prior video frames; if they are gradually increasing and the first valid write count of the current video frame is within a set range of a preset maximum write count threshold, determining that the current video frame is a key video frame; otherwise, determining that the current video frame is not a key video frame. If |updata_cnt-a2| < a preset value, where the preset value is user-configurable, then the first valid write count of the current video frame is within the set range of the preset maximum write count threshold.

[0105] Specifically, based on the third valid write times of each of the previous multiple video frames, determine whether the third valid write times gradually increases, including: obtaining the difference in the third valid write times of two adjacent video frames; for the two adjacent number differences, determine whether the first value corresponding to the previous number difference is less than the second value corresponding to the subsequent number difference, and obtain a determination result; the first value is obtained based on the previous number difference and the corresponding first weight, and the second value is obtained based on the subsequent number difference and the corresponding second weight; based on the determination results of all the adjacent number differences, determine whether the third valid write times gradually increases, wherein if all the determination results are yes, it is determined that the third valid write times gradually increases.

[0106] In an embodiment of the present invention, the difference in the third effective write count updata_cnt of M frames (M is less than N) is detected. The difference is increasing, and the difference between the updata_cnt of the latest frame and a2 is less than a preset value. Here, M is configurable, and the preset value is configurable. The specific formula (hardware judgment logic) is as follows: (updata_cnt_1 - updata_cnt_0) × the first first weight < (updata_cnt_2 - updata_cnt_1) × the first second weight; (updata_cnt_2 - updata_cnt_1) × the second first weight < (updata_cnt_3 - updata_cnt_2) × the second second weight; (updata_cnt_3 - updata_cnt_2) × the third first weight < (updata_cnt_4 - updata_cnt_3) × the third second weight; …; (updata_cnt_M - 2 - updata_cnt_M - 3) × the (M - 2)th first weight < (updata_cnt_M - 1 - updata_cnt_M - 2) × the (M - 2)th second weight; and |updata_cnt_M - 1 - Threhold_2| < Threhold_pre, where updata_cnt_M - 1 is the effective write count of the current video frame. Among them, the first weight and the second weight marked in the above formula (hardware judgment logic) can adopt default values such as 1, or can be configured by the user. At the same time, in the embodiment of the present invention, the above parameters are adaptively adjusted according to the correctness of the prediction result. When all the above formulas are satisfied, it is determined that the current video frame is a key frame, the frame interval N value is adjusted in advance, and the current frame is processed as a key frame, while waiting for the output result of the interval adaptive module.

[0107] Further, the video frame processing method further includes: if it is determined that the current video frame is a key video frame according to both the first effective write count and the third effective write count, then increase the second weight; if it is determined that the current video frame is not a key video frame according to the first effective write count, and it is determined that the current video frame is a key video frame according to the third effective write count, then increase the first weight.

[0108] If the interval adaptive module also determines that a frame is a keyframe after a few frames, that is, the value of updata_cnt is greater than threshold a2, then the keyframe prediction mechanism is considered correct. In this case, the second weight is increased. The increased value can be customized by the user. This increases the probability that the above formula is fully true, thereby increasing the probability of determining a keyframe, thereby ensuring that the above formula can continue to determine a keyframe with a high probability. If the interval adaptive module determines that a frame is not a keyframe after a few frames, and the value of updata_cnt does not exceed threshold a2, then the judgment in this step is considered incorrect. In this case, the first weight is increased to reduce the probability that the above formula is fully true, thereby reducing the probability of determining a keyframe. This ensures that the above formula reduces the probability of determining a keyframe.

[0109] In the embodiment of the present invention, the weights are adjusted in the above manner, so that the determination result of the key frame prediction mechanism is more accurate.

[0110] Accordingly, if the bus state is busy and the current video frame is not a key video frame, then processing of the current video frame is stopped, including: if the bus state is busy and, based on the first valid write count and the third valid write count, it is determined that the current video frame is not a key video frame, then processing of the current video frame is stopped. When the feedback signal bus_state from the post-stage data output control module indicates that the current DDR bus state is very busy and the data output control module is unable to write data to the DDR in a timely manner, and at the same time, the results of the interval adaptation module and the key frame prediction module both indicate that the current frame is a non-key frame, the non-key frame processing module outputs a control signal to the RGB2YUV module to stop subsequent processing of the current non-key frame.

[0111] It can be seen that in the embodiment of the present invention, when the bus state is busy and both key frames are determined not to be key video frames, the video processing is stopped, which can more accurately determine the attributes of the current video frame.

[0112] See also Figure 6 , Figure 6 This is a schematic diagram of the video hardware processing flow of a server management and control device provided by an embodiment of the present invention; it optimizes the function of the video capture module, adds an interval adaptation module, a key frame prediction module, and a non-key frame processing module, realizes adaptive prediction and recognition of key frames, dynamically adjusts the discrimination parameters, and improves the discrimination accuracy.

[0113] Furthermore, see Figure 7 , Figure 7This is a hardware flow diagram of a data output control module provided by an embodiment of the present invention; the embodiment of the present invention also adds a key frame analysis submodule, a bus state prediction submodule, and a frame loss statistics submodule in the data output control module, and improves the read control submodule; it not only realizes adaptive pre-judgment and recognition of key frames, dynamically adjusts the discrimination parameters, and improves the discrimination accuracy, but also improves the complete write rate of the on-chip storage space of key frames, realizes autonomous frame loss for non-key frames, reduces the memory bandwidth and bus bandwidth occupied by video functions, and improves the overall performance of the chip.

[0114] A possible implementation of an embodiment of the present application, before writing the compressed data to the on-chip cache, also includes: obtaining the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data writing to the on-chip cache, and the compressed length of the key frame; determining whether the compressed data can be completely written to the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data writing to the on-chip cache, and the compressed length of the key frame; if the compressed data can be completely written to the on-chip cache, writing the compressed data to the on-chip cache.

[0115] In an embodiment of the present invention, the key frame analysis and judgment submodule in the data output control module receives the output result of the previous key frame prediction module. If the current video frame is judged to be a key frame, the key frame prediction module outputs a control signal to the key frame analysis and judgment submodule, that is, the key_frame signal is pulled high. After the key frame analysis and judgment submodule receives this signal, the data output control module enters the urgent mode (urgent_mode).

[0116] After the data output control module enters emergency mode, the read control submodule receives the outputs of the bus status determination submodule and the compressed data cache control module to predict whether the key frame can be completely written to the on-chip cache. The output of the compressed data cache control submodule is the remaining space in the on-chip cache, and the output of the bus status determination submodule is the first write rate of the double data rate memory. If the prediction result indicates that the key frame can be completely written to the on-chip cache, normal reading is performed.

[0117] If the on-chip cache cannot be completely written, and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if the on-chip cache cannot be completely written, and the current video frame is not a key video frame, the compressed data will be discarded. Specifically, if the read control submodule predicts that the key frame cannot be completely written to the on-chip cache, the non-key frame data in the on-chip cache will be read in advance in units of frames and discarded. A frame information table is pre-set in the read control submodule, which records the address space range of the non-key frames in the on-chip cache, that is, these non-key frames read out are not written to the external DDR.

[0118] It can be seen that in an embodiment of the present invention, the compressed data is written to the on-chip cache only when it is determined that the compressed data can be completely written to the on-chip cache. If it cannot be completely written to the on-chip cache and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if it cannot be completely written to the on-chip cache and the current video frame is not a key video frame, the compressed data will be discarded, thereby reducing the frame loss rate of the key frames and improving the display quality of the remote end.

[0119] Specifically, a possible implementation method of an embodiment of the present invention determines whether the compressed data can be completely written into the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the compressed length of the key frame, including: determining a third value based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data into the on-chip cache, and the third weight; determining a fourth value based on the compressed length of the key frame and the second write rate of the frame data into the on-chip cache; determining whether the compressed data can be completely written into the on-chip cache based on the size of the third value and the fourth value, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

[0120] The calculation process is as follows:

[0121] 1: The compressed data cache control submodule outputs the remaining space space_can_use of the on-chip cache, that is, the space to be written.

[0122] 2: The bus status judgment submodule outputs the first write rate wr_ddr_rate of the double data rate memory within a preset time period T. T can use a default value or can be configured by the user through a register.

[0123] 3: The rate at which frame data is written to the on-chip buffer is fixed (depending on the module clock frequency). The second write rate at which frame data is written to the on-chip buffer is recorded as wr_buffer_speed.

[0124] 4: That is, the compressed length of the key frame is frame_compress_lenth. The judgment formula (hardware judgment logic) is as follows: according to the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data written to the on-chip cache, and the third weight PARAM0, the third value is determined, wherein the third value = space_can_use / (wr_buffer_speed-wr_ddr_rate×PARAM0); wherein PARAM0 is an adjustable parameter and the default value is 1; according to the compressed length of the key frame and the second write rate of the frame data written to the on-chip cache, the fourth value is determined; the fourth value = rame_compress_lenth / wr_buffer_speed; if the third value is greater than the fourth value, it is determined that the current key frame can be completely written to the on-chip cache; if the third value is not greater than the fourth value, it is determined that the current key frame cannot be completely written to the on-chip cache.

[0125] It can be seen that in an embodiment of the present invention, the third value is determined based on the cache margin, DRAM write rate, cache write rate and weight; the fourth value is determined based on the key frame compression length and the cache write rate; and the two values are compared. If the third value is larger, the compressed data can be completely stored in the cache.

[0126] In one possible implementation of an embodiment of the present application, obtaining a first write rate of a double data rate memory includes: obtaining the number of times a write data valid signal of an interactive interface of the double data rate memory and a write data enable indication signal output by a bus of the double data rate memory are simultaneously valid within a preset time period; and determining the first write rate of the double data rate memory based on the number of signal valid times and the preset time period.

[0127] The bus status determination submodule detects, counts, and determines the bus status, outputting two data points: bus_state, which is sent to the preceding non-keyframe processing module, and wr_ddr_rate, which is sent to the read control module. The bus status detection process counts the number of times valid_cnt the write data valid signal wr_data_vld from the data output control module's interface with the DDR (typically AXI) and the write data ready indicator (wr_ready) from the double data rate memory's bus output are simultaneously valid (indicating that data has been effectively written to the DDR), within a preset time period. Here, wr_ddr_rate = valid_cnt / preset time period T.

[0128] In one implementation, if the first write rate is less than a preset rate threshold, the bus state is determined to be busy and the preset time period is reduced; if the first write rate is not less than the preset rate threshold, the preset time period is increased.

[0129] When the first write rate wr_ddr_rate is less than the preset rate threshold, indicating that the DDR write rate is very slow, system bus resources are tight, and the data output control module cannot write data to the external DDR in a timely manner, bus_state is set to 2'b10, where the preset rate threshold is the bus state threshold. At the same time, the preset time period is reduced to further monitor the bus state and shorten the bus state detection cycle to promptly detect state changes. The specific reduction value can be set according to the user's configuration parameters.

[0130] When the first write rate wr_ddr_rate is not less than the preset rate threshold, indicating that the DDR write rate meets the preset requirements, the data output control module can promptly write data to the off-chip DDR. At this time, bus_state is set to 2'b01, and the preset time period is increased, appropriately relaxing the bus status detection period. The specific increase value can be set according to the user's configuration parameters.

[0131] A possible implementation method of an embodiment of the present application also includes: obtaining the total amount of discarded non-critical frame data in the on-chip cache; if the total amount of discarded data reaches a preset total amount of discarded data threshold, lowering the third weight; after lowering the third weight, if it is detected that a preset amount of non-critical frame data has not been discarded, resetting the third weight.

[0132] In an embodiment of the present invention, a frame loss statistics submodule receives the total number of continuously discarded non-critical frames (drop_cnt) from the on-chip cache and read by the read control submodule. When the cumulative total number of drop_cnt reaches a preset total number of discarded frames threshold (i.e., the total number of discarded frames exceeds the preset total number of discarded frames threshold), indicating that a large number of continuously discarded non-critical frames are being affected, the third weight is adjusted to reduce the frame loss rate. Furthermore, after drop_cnt detects a preset number (one or two frames) of non-critical frames without loss, it automatically resets. Re-testing restores the third weight to 1.

[0133] As can be seen, in this embodiment of the present invention, when the amount of discarded non-key frame data in the on-chip buffer reaches a certain amount, the third weight used to determine key frames is reduced, reducing the discard of non-key frames and thus balancing video quality. Furthermore, when it is detected that a preset amount of non-key frame data has not been discarded, the third weight is reset to its initial state, improving the efficiency and quality of overall video transmission.

[0134] Based on any of the above embodiments, an embodiment of the present invention optimizes the functions of the video capture module, adds an interval adaptation module, a key frame prediction module, and a non-key frame processing module, and improves the data output control module. A key frame parsing submodule, a bus state prediction submodule, and a frame loss statistics submodule are added to the data output control module, and the read control submodule is improved. In this embodiment of the present invention, adaptive prediction and recognition of key frames are achieved, and discrimination parameters are dynamically adjusted to improve discrimination accuracy. At the same time, the complete write rate of the on-chip storage space of key frames is increased, and autonomous frame loss is achieved for non-key frames, thereby reducing the memory bandwidth and bus bandwidth occupied by the video function and improving the overall performance of the chip.

[0135] Specifically, the embodiment of the present invention implements adaptive prediction of key frames, proposes a prediction formula (hardware judgment logic) for key frames and adaptive adjustment of parameters, which greatly improves the accuracy of prediction.

[0136] The embodiment of the present invention realizes adaptive adjustment of the detection interval, realizes rapid discrimination and processing from non-key frames to key frames, thereby ensuring that the remote end can obtain key frames with high quality, and can also realize recovery from key frames to non-key frames. On the basis of ensuring video quality, it minimizes the occupation of the system bus and memory bus by the video function, thereby improving the overall performance of the chip.

[0137] The embodiment of the present invention processes non-key frames in the on-chip cache in advance according to prediction information of key frames and information such as bus status, thereby improving the complete writing rate of key frames in the on-chip cache space and improving the display quality of the remote end.

[0138] The embodiment of the present invention automatically detects the continuous frame loss rate of non-key frames and autonomously adjusts the judgment calculation logic, thereby greatly reducing the impact of non-key frame loss processing on remote display.

[0139] The following is an introduction to a device provided by an embodiment of the present invention. The device described below and the method described above can be referenced to each other. The device of this embodiment is set in a server management control device. Figure 8 , Figure 81 is a block diagram of a device according to one embodiment of the present invention, comprising: a video capture module 210 for obtaining a first valid write count of pixels of a current video frame, the first valid write count representing the total number of pixel data points having differences between the current video frame and a previous video frame; an interval adaptation module 220 for determining a first target preset count range within which the first valid write count falls based on a first preset count range; wherein if any data point in the first target preset count range is greater than the maximum data point in any other first preset count range, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; the interval adaptation module 220 is further configured to adjust a frame interval based on the first target preset count range, the frame interval being the number of intervals between two adjacent determinations of the first valid write count; a non-key frame processing module 230 for obtaining a bus status; if the bus status is busy and the current video frame is not a key video frame, stopping processing of the current video frame; otherwise, triggering the video processing module; a video processing module 240 for performing image compression on the current video frame; and a data output control module 250 for writing the compressed data into an on-chip cache.

[0140] In one achievable manner, the interval adaptation module 220 is further used to: if the first target preset number range is the preset number range of the minimum value, determine that the frame interval remains unchanged, wherein the frame interval is the number of intervals between two adjacent determinations of the first effective write number; if the first target preset number range is the preset number range of the intermediate value, obtain the average effective number of the second effective write number of each of multiple consecutive video frames; adjust the frame interval according to the second target preset number range of the second preset number range where the average effective number is located to obtain the first frame interval; if the first target preset number range is the preset number range of the maximum value, determine that the frame interval is the second frame interval, and the second frame interval is smaller than the frame interval and smaller than the first frame interval.

[0141] In one achievable manner, the interval adaptation module 220 is further configured to: determine an adjustment coefficient according to a second target preset number range within a second preset number range where the average effective number of times falls; and obtain a first frame interval according to the adjustment coefficient and the frame interval.

[0142] In one feasible manner, the interval adaptation module 220 is also used to: if the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, obtain multiple consecutive average effective times; if there is a target average effective number that is less than the minimum value of the first target preset number range, increase the frame interval.

[0143] In one feasible manner, it also includes: a key frame prediction module, which is used to: obtain the third valid write times of each of the previous multiple video frames; determine whether the current video frame is a key video frame based on the third valid write times of each of the previous multiple video frames; if it is a key video frame, determine in advance that the frame interval is the second frame interval.

[0144] In one feasible manner, the key frame prediction module is specifically used to: determine whether the effective write count is gradually increasing based on the third effective write counts of each of the previous multiple video frames; if it is gradually increasing, and the first effective write count of the current video frame is within the set range of the preset maximum count threshold, then determine that the current video frame is a key video frame; otherwise, determine that the current video frame is not a key video frame.

[0145] In one feasible manner, the key frame prediction module is specifically used to: obtain the difference in the number of third effective write times of two adjacent video frames; for the two adjacent number differences, determine whether the first value corresponding to the previous number difference is less than the second value corresponding to the subsequent number difference, and obtain a determination result; the first value is obtained based on the previous number difference and the corresponding first weight, and the second value is obtained based on the subsequent number difference and the corresponding second weight; based on the determination results of all adjacent number differences, determine whether the third effective write number gradually increases, wherein if all the determination results are yes, it is determined that the third effective write number gradually increases.

[0146] In one feasible manner, the key frame prediction module is specifically used to: if the current video frame is determined to be a key video frame based on the first valid write count and the third valid write count, then increase the second weight; if the current video frame is determined not to be a key video frame based on the first valid write count, and the current video frame is determined to be a key video frame based on the third valid write count, then increase the first weight.

[0147] In one achievable manner, the non-key frame processing module 230 is configured to stop processing the current video frame if the bus state is busy and the current video frame is determined not to be a key video frame based on the first valid write count and the third valid write count.

[0148] In one achievable manner, the data output control module 250 includes: a read control submodule, configured to: obtain the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data written to the on-chip cache, and the compressed length of the key frame; determine whether the compressed data can be completely written to the on-chip cache based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data written to the on-chip cache, and the compressed length of the key frame; and if the compressed data can be completely written to the on-chip cache, write the compressed data to the on-chip cache.

[0149] In one implementable manner, the data output control module 250 includes: a read control submodule, specifically used to: determine a third value based on the remaining space of the on-chip cache, the first write rate of the double data rate memory, the second write rate of the frame data written to the on-chip cache, and the third weight; determine a fourth value based on the compressed length of the key frame and the second write rate of the frame data written to the on-chip cache; determine whether the compressed data can be completely written to the on-chip cache based on the sizes of the third value and the fourth value, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written to the on-chip cache.

[0150] In one achievable embodiment, the data output control module 250 further includes a bus status determination submodule configured to obtain a number of times that a write data valid signal of the interactive interface of the double data rate memory and a write data enable indication signal outputted by the bus of the double data rate memory are simultaneously valid within a preset time period; and determine a first write rate of the double data rate memory based on the number of signal valid times and the preset time period.

[0151] In one achievable manner, the data output control module 250 further includes: a bus status judgment submodule, which is also used to: if the first write rate is less than the preset rate threshold, determine that the bus status is a busy state and reduce the preset time period; if the first write rate is not less than the preset rate threshold, increase the preset time period.

[0152] In one feasible manner, the read control submodule is further used to: if the on-chip cache cannot be completely written and the current video frame is a key video frame, the non-key frame data in the on-chip cache will be read and discarded in units of frames; if the on-chip cache cannot be completely written and the current video frame is not a key video frame, the compressed data will be discarded.

[0153] In one achievable manner, the data output control module 250 further includes: a frame loss statistics submodule, which is used to: obtain the total amount of discarded non-critical frame data in the on-chip cache; if the total amount of discarded data reaches a preset total amount of discarded data threshold, reduce the third weight; after reducing the third weight, if it is detected that a preset amount of non-critical frame data has not been discarded, reset the third weight.

[0154] Figure 8 The description of the features in the corresponding embodiment can be found in Figure 4 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0155] Figure 9 A structural diagram of a server management and control device provided by an embodiment of the present invention, such as Figure 9As shown, the server management control device includes: a memory 60 for storing computer programs; a processor 61 for implementing the steps of the video frame processing method in the above embodiment when executing the computer program.

[0156] The server management and control device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0157] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing content required to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor for handling computational operations related to machine learning.

[0158] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the video frame processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc.

[0159] In some embodiments, the server management and control device may further include a display screen 62 , an input and output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0160] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation to the server management control device, and may include more or fewer components than shown in the figure.

[0161] It is understandable that if the video frame processing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the current technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and executes all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, a magnetic disk, or an optical disk, and other media that can store program code.

[0162] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned video frame processing method are implemented.

[0163] An embodiment of the present invention further provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned video frame processing method when executed by a processor.

[0164] The above describes in detail the video frame processing method, apparatus, device, medium, and product provided by the embodiments of the present invention. The various embodiments are described in a progressive manner throughout this specification, with each embodiment focusing on the differences from the other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple. For relevant details, refer to the method description.

[0165] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0166] The above is a detailed introduction to the video frame processing method, device, equipment, medium and product provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A video frame processing method, characterized in that: include: Obtaining a first valid write count of pixels of a current video frame, where the first valid write count represents a total number of pixel data having differences between the current video frame and a previous video frame; determining, based on a first preset number range, a first target preset number range in which the first valid write number falls; wherein, if any data in the first target preset number range is greater than the maximum data in other first preset number ranges, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; Adjusting the current frame interval according to the first target preset number range, where the frame interval is the number of intervals between two adjacent first valid write times; Obtaining a bus state; if the bus state is busy and the current video frame is not a key video frame, stopping processing the current video frame; Otherwise, performing image compression processing on the current video frame and writing the compressed data into an on-chip cache; Before determining, according to the first preset number range, a first target preset number range within which the first valid write number falls, the method further includes: Obtaining a third valid write count of each of the previous plurality of video frames; determining whether the current video frame is a key video frame according to respective third valid write times of a plurality of previous video frames; If it is a key video frame, the current frame interval is determined in advance to be the second frame interval.

2. The video frame processing method according to claim 1, wherein: Adjusting the current frame interval according to the first target preset number range includes: If the first target preset number range is the preset number range of the minimum value, determining that the current frame interval remains unchanged; If the first target preset number range is a preset number range of intermediate values, obtaining an average effective number of second effective write times of each of the plurality of consecutive video frames; adjusting the current frame interval according to a second target preset number range of the second preset number range within which the average effective number falls, to obtain a first frame interval; If the first target preset number range is the preset number range with the largest value, it is determined that the current frame interval is the second frame interval which is smaller than the first frame interval.

3. The video frame processing method according to claim 2, wherein: Adjusting the current frame interval according to a second target preset number range of the second preset number range within which the average effective number of times falls includes: determining an adjustment coefficient according to a second target preset number range within the second preset number range in which the average effective number of times falls; Obtaining a first frame interval according to the adjustment coefficient and the current frame interval; And / or, when the first preset number range and the second preset number range are the same, the method further includes: If the minimum value of the first target preset number range is not greater than the minimum value of the second target preset number range, obtaining a plurality of consecutive average effective times; If there is a target average effective number of times that is less than the minimum value of the first target preset number range, the current frame interval is increased.

4. The video frame processing method according to claim 1, wherein: Determining whether the current video frame is a key video frame according to the third valid write times of each of the plurality of previous video frames includes: determining whether the valid writing times gradually increase based on respective third valid writing times of a plurality of previous video frames; If it gradually increases and the first valid writing times of the current video frame is within the set range of the preset maximum times threshold, the current video frame is determined to be a key video frame; otherwise, the current video frame is determined not to be a key video frame.

5. The video frame processing method according to claim 4, wherein: Determining whether the third valid write count gradually increases based on respective third valid write counts of a plurality of previous video frames includes: Obtain the difference between the third valid write times of two adjacent video frames; For two adjacent number of times difference values, determining whether a first value corresponding to the previous number of times difference value is less than a second value corresponding to the next number of times difference value, and obtaining a determination result; the first value is obtained based on the previous number of times difference value and a corresponding first weight, and the second value is obtained based on the next number of times difference value and a corresponding second weight; According to all the determination results of the differences between two adjacent times, it is determined whether the third effective writing number gradually increases. If all the determination results are yes, it is determined that the third effective writing number gradually increases.

6. The video frame processing method according to claim 5, characterized in that: Also includes: If it is determined that the current video frame is a key video frame according to the first valid writing times and the third valid writing times, then the second weight is increased; If it is determined that the current video frame is not a key video frame according to the first valid writing number, and if it is determined that the current video frame is a key video frame according to the third valid writing number, the first weight is increased.

7. The video frame processing method according to claim 1, wherein: If the bus state is a busy state and the current video frame is not a key video frame, stopping processing the current video frame includes: If the bus state is a busy state, and according to the first valid write count and the third valid write count, it is determined that the current video frame is not a key video frame, then processing of the current video frame is stopped.

8. The video frame processing method according to any one of claims 1 to 7, characterized in that: Before writing the compressed data to the on-chip cache, it also includes: Obtaining remaining space of an on-chip buffer, a first write rate of the double data rate memory, a second write rate of writing frame data into the on-chip buffer, and a compressed length of a key frame; determining whether the compressed data can be completely written into the on-chip cache based on remaining space in the on-chip cache, a first write rate of the double data rate memory, a second write rate of writing the frame data into the on-chip cache, and a compressed length of the key frame; If the data can be completely written into the on-chip cache, the compressed data is written into the on-chip cache.

9. The video frame processing method according to claim 8, wherein: Determining whether the compressed data can be completely written into the on-chip cache according to remaining space of the on-chip cache, a first write rate of the double data rate memory, a second write rate of writing the frame data into the on-chip cache, and a compressed length of the key frame includes: determining a third value according to a remaining space of the on-chip cache, a first write rate of the double data rate memory, a second write rate of writing frame data into the on-chip cache, and a third weight; determining a fourth value according to a compressed length of the key frame and a second write rate of writing frame data into the on-chip cache; Whether the compressed data can be completely written into the on-chip cache is determined based on the sizes of the third and fourth values, wherein when the third value is greater than the fourth value, it indicates that the compressed data can be completely written into the on-chip cache.

10. The video frame processing method according to claim 8, wherein: Obtaining a first write rate of a double data rate memory, comprising: Obtaining a number of times that a write data valid signal of an interactive interface of the double data rate memory and a write data permission indication signal output by a bus of the double data rate memory are simultaneously valid within a preset time period; A first write rate of the double data rate memory is determined according to the number of valid signal times and a preset time period.

11. The video frame processing method according to claim 10, wherein: Also includes: If the first write rate is less than a preset rate threshold, determining that the bus state is a busy state and reducing the preset time period; If the first writing rate is not less than a preset rate threshold, the preset time period is increased.

12. The video frame processing method according to claim 9, wherein: Also includes: If the on-chip cache cannot be fully written, and the current video frame is a key video frame, non-key frame data in the on-chip cache will be read and discarded in units of frames; If the compressed data cannot be completely written into the on-chip cache and the current video frame is not a key video frame, the compressed data is discarded.

13. The video frame processing method according to claim 12, wherein: Also includes: Get the total amount of discarded non-key frame data in the on-chip cache; If the total discarded amount reaches a preset total discarded amount threshold, reducing the third weight; After reducing the third weight, if it is detected that a preset amount of non-key frame data is not discarded, the third weight is reset.

14. A video frame processing device, characterized in that: include: A video capture module, configured to obtain a first valid write count of pixels of a current video frame, where the first valid write count represents a total number of pixel data having differences between the current video frame and a previous video frame; an interval adaptive module, configured to determine, based on a first preset number range, a first target preset number range within which the first valid write count falls; wherein if any data in the first target preset number range is greater than a maximum data in any other first preset number range, the current video frame is a key video frame; otherwise, the current video frame is not a key video frame; The interval adaptive module is further configured to adjust the current frame interval according to the first target preset number range, where the frame interval is the number of intervals between two adjacent first valid write times; A non-key frame processing module is used to obtain a bus state; if the bus state is busy and the current video frame is not a key video frame, the processing of the current video frame is stopped; otherwise, the video processing module is triggered; A video processing module, configured to perform image compression processing on the current video frame; A data output control module, used to write compressed data into the on-chip cache; The key frame prediction module is used to obtain the third valid write times of each of the previous multiple video frames; determine whether the current video frame is a key video frame based on the third valid write times of each of the previous multiple video frames; if it is a key video frame, determine in advance that the current frame interval is the second frame interval.

15. A server management and control device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the video frame processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the video frame processing method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the video frame processing method according to any one of claims 1 to 13 are implemented.

Images