Video encoding and decoding dynamic frequency modulation method and its architecture, equipment and medium

Abstract

The present application discloses a method for dynamic frequency modulation of video encoding and decoding and its architecture, equipment, and medium, and relates to the field of video encoding and decoding technology; the method comprises: performing performance testing and processing according to a preset frequency modulation gear to create an optimal frequency index table; receiving at least one video to be processed, and when the dynamic frequency modulation function is enabled, obtaining video information of the video to be processed and determining a video processing method; recording or updating the received video information to the node information of the corresponding user node; obtaining the node information of each user node; performing matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determining the maximum frequency value among multiple user frequencies as the target frequency; and the hardware encoding and decoding module encoding and decoding the video to be processed according to the target frequency and the video processing method. The present application can dynamically adjust the operating frequency of the hardware encoding and decoding module, thereby reducing power consumption and improving encoding and decoding performance.

Description

Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to a video coding and decoding dynamic frequency modulation method and its architecture, device, and medium. Background Art

[0002] With the rapid development of integrated circuit technology and multimedia audio and video technology, video has become an important and indispensable information medium in people's daily lives and work. Electronic products with video image acquisition, compression, transmission, decompression, and display functions are widely used in various industries, such as dashcams, sports DVs, security building monitoring systems, network TV set-top boxes, and tablet computers. Video images, however, have a relatively large information capacity. For example, for 4K@30fps+8-bit+YUV420 video, a single image is 11.86MB, a 1-second image is 355MB, and a 1-minute image is 2.136GB. Directly storing and transmitting raw image data consumes significant storage and network bandwidth resources. To reduce resource consumption, video encoders are often used to compress images captured from cameras, with compression ratios exceeding 100 times. The compressed video stream is then transmitted over the network or stored locally, saving network bandwidth and storage resources. On the user side, a video decoder must first decompress the video stream data to restore the video image, which is then output to a display device such as an LCD for display. Because the compression and decompression algorithms for video images are complex and require significant computing resources, dedicated video codec digital integrated circuits are typically used to process image encoding and decoding. Video codec digital integrated circuits (hereinafter referred to as video codec modules) can operate at multiple clock frequencies. Increasing the clock frequency within a certain range improves the processing power of the video codec module, but also increases power consumption. Finding the optimal clock frequency for the hardware module is crucial, balancing performance and power consumption to meet the requirements of the existing use case while also reducing power consumption. Finding a reliable optimal clock frequency for the hardware module is a pressing issue. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the prior art. To this end, this application proposes a video codec dynamic frequency modulation method and its architecture, device, and medium, which can dynamically adjust the operating frequency of the hardware codec module based on a constructed optimal frequency index table and video information of the video to be processed, thereby reducing power consumption and improving codec performance.

[0004] In a first aspect, an embodiment of the present application provides a video codec dynamic frequency modulation method, which is applied to a video codec dynamic frequency modulation architecture, wherein the video codec dynamic frequency modulation architecture includes: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially communicatively connected; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; and the hardware layer includes a hardware codec module;

[0005] The method comprises:

[0006] Through the hardware layer, a performance test is performed according to the preset frequency modulation gear to create an optimal frequency index table;

[0007] When the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains video information of the video to be processed and determines a video processing method;

[0008] The driving layer records or updates the received video information into the node information of the corresponding user node through the information update module;

[0009] The driver layer obtains node information of each user node through the frequency selection control module; performs matching processing based on the node information and the optimal frequency index table to determine the user frequency of each user node; determines the maximum frequency value among the multiple user frequencies as the target frequency; and sets the operating frequency of the hardware codec module to the target frequency;

[0010] The hardware encoding and decoding module performs encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

[0011] In a second aspect, an embodiment of the present application provides a video codec dynamic frequency modulation architecture, comprising: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially communicatively connected; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a hardware codec module;

[0012] The hardware layer is used to: perform performance testing according to the preset frequency modulation gear and create an optimal frequency index table;

[0013] The codec layer is used to: receive at least one video to be processed sent by the application end, and when the dynamic frequency modulation function is enabled, obtain video information of the video to be processed and determine a video processing method;

[0014] The driving layer is used to: record or update the received video information into the node information of the corresponding user node through the information update module;

[0015] The driver layer is further configured to: obtain node information of each user node through the frequency selection control module; perform matching processing based on the node information and the optimal frequency index table to determine the user frequency of each user node; determine the maximum frequency value among the multiple user frequencies as the target frequency; and set the operating frequency of the hardware codec module to the target frequency;

[0016] The hardware encoding and decoding module is further configured to perform encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

[0017] In a third aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video encoding and decoding dynamic frequency modulation method as described in any one of the embodiments of the first aspect.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video encoding and decoding dynamic frequency modulation method as described in any one of the embodiments of the first aspect.

[0019] The embodiments of the present application include: a video codec dynamic frequency modulation architecture includes: an application layer, a codec layer, a driver layer, and a hardware layer that are communicatively connected in sequence; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a hardware codec module. In the process of video encoding and decoding using the video codec dynamic frequency modulation architecture, first, through the hardware layer, performance testing is performed according to the preset frequency modulation gear, and an optimal frequency index table is created to provide a reliable reference for subsequent dynamic frequency modulation; secondly, when the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains the video information of the video to be processed and determines the video processing method; then, the driver layer records or updates the received video information to the node information of the corresponding user node through the information update module; then, the driver layer obtains the node information of each user node through the frequency selection control module; then, matching processing is performed based on each node information and the optimal frequency index table to determine the user frequency of each user node; then, the maximum frequency value among multiple user frequencies is determined as the target frequency; finally, the operating frequency of the hardware codec module is set to the target frequency; the hardware codec module encodes and decodes the video to be processed according to the target frequency and the video processing method; thereby reducing the power consumption of codecs and improving the codec performance. That is, the solution of the embodiment of the present application can dynamically adjust the operating frequency of the hardware codec module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing the codec power consumption and improving the codec performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a structural diagram of a video codec dynamic frequency modulation architecture provided by an embodiment of the present application;

[0021] Figure 2 This is a flowchart of a video encoding and decoding dynamic frequency modulation method provided by an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a process of sending video information from a codec layer to a driver layer according to an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an optimal frequency index table provided by an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0026] It should be understood that in the description of this application, descriptions of orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0027] It should be noted that although a logical order is shown in the flowchart in the description of this application, in some cases, the steps shown or described may be performed in an order different from that in the flowchart. In the description of this application, "several" means one or more, and "more" means two or more. The description of "first" and "second" is only used to distinguish technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0029] First, some terms used in this application are explained:

[0030] Ioctl: A system call function widely used in UNIX and UNIX-like systems (such as Linux). Its full name is "input / output control". It is used for communication between device drivers and device files, allowing user space programs to send or receive control commands to kernel space device drivers.

[0031] The present application provides a method for dynamic frequency modulation of video encoding and decoding and its architecture, electronic device and computer-readable storage medium, the method comprising: creating an optimal frequency index table by performing performance testing according to a preset frequency modulation gear; receiving at least one video to be processed, and obtaining video information of the video to be processed and determining a video processing method when the dynamic frequency modulation function is enabled; recording or updating the received video information to the node information of the corresponding user node; obtaining the node information of each user node; performing matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determining the maximum frequency value among multiple user frequencies as the target frequency; and the hardware encoding and decoding module encoding and decoding the video to be processed according to the target frequency and the video processing method. The present application can dynamically adjust the operating frequency of the hardware encoding and decoding module, thereby reducing power consumption and improving encoding and decoding performance.

[0032] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, an embodiment of the present application provides a video codec dynamic frequency modulation architecture, comprising: an application layer, a codec layer, a driver layer, and a hardware layer, which are sequentially communicatively connected. The application layer includes an application terminal; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a hardware codec module. The codec layer also includes a performance monitoring module.

[0034] The hardware layer is used to perform performance testing based on preset frequency modulation levels and create an optimal frequency index table.

[0035] The codec layer is used to: receive at least one video to be processed sent by the application end, and when the dynamic frequency modulation function is enabled, obtain video information of the video to be processed and determine the video processing method.

[0036] The driver layer is used to record or update the received video information into the node information of the corresponding user node through the information update module.

[0037] The driver layer is also used to: obtain the node information of each user node through the frequency selection control module; perform matching processing based on each node information and the optimal frequency index table to determine the user frequency of each user node; determine the maximum frequency value among multiple user frequencies as the target frequency; and set the operating frequency of the hardware codec module to the target frequency.

[0038] The hardware codec module is further used to perform codec processing on the video to be processed according to the target frequency and video processing method.

[0039] The performance monitoring module is used to monitor the codec processing of the hardware codec module and send a working frequency adjustment instruction to the driver layer when an abnormal situation of insufficient codec performance is detected.

[0040] It is understood that the video codec dynamic frequency modulation software architecture of this application is mainly divided into four layers: application layer, codec layer, driver layer, and hardware layer. The frequency modulation software solution logic is implemented in the driver layer. Further, the application layer, codec layer, driver layer, and hardware layer of the video codec dynamic frequency modulation architecture are described.

[0041] The application layer includes the application end, which is also the user of the codec layer. It is responsible for calling the codec layer to implement related functions of video processing and is the user of the codec function.

[0042] The codec layer is responsible for managing the entire codec hardware module, encapsulating the underlying complex interfaces and operation logic, and providing a simple interface to the application layer to implement codec functions.

[0043] The Linux driver layer also enables multi-user management. For the hardware codec module, multiple users mean that multiple processes or threads may be using the hardware codec module simultaneously. To manage multiple users, the driver layer creates a user node that represents identity information for each user, making it easier to determine the identity of the user currently using the hardware codec module. It should be noted that after receiving video information from the user, the driver layer performs unified management and adjusts the performance of the hardware codec module by looking up the most suitable operating frequency. This allows for timely adoption of the optimal operating frequency for the hardware codec module in complex application scenarios such as multi-process, multi-thread, and multiple codec formats.

[0044] According to the video codec dynamic frequency modulation architecture provided in the embodiment of the present application, in the process of video encoding and decoding using the video codec dynamic frequency modulation architecture, first, through the hardware layer, performance testing is performed according to the preset frequency modulation gear, and an optimal frequency index table is created to provide a reliable reference for subsequent dynamic frequency modulation; secondly, when the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains the video information of the video to be processed and determines the video processing method; then, the driver layer records or updates the received video information to the node information of the corresponding user node through the information update module; then, the driver layer obtains the node information of each user node through the frequency selection control module; then, matching processing is performed based on each node information and the optimal frequency index table to determine the user frequency of each user node; then, the maximum frequency value among multiple user frequencies is determined as the target frequency; finally, the operating frequency of the hardware codec module is set to the target frequency; the hardware codec module performs codec processing on the video to be processed according to the target frequency and the video processing method; thereby reducing codec power consumption and improving codec performance. It can be seen that the video codec dynamic frequency modulation architecture of the embodiment of the present application can dynamically adjust the operating frequency of the hardware codec module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing the codec power consumption and improving the codec performance.

[0045] It should be emphasized that currently, there is no dynamic frequency modulation solution for video codec modules, and dynamic frequency modulation is usually performed on the CPU. Specifically, the CPU dynamic frequency modulation method is to obtain the CPU occupancy rate at the current frequency and determine the operating frequency based on the CPU occupancy rate. This application addresses the high power consumption in video codec application scenarios, which is specifically manifested in the low energy efficiency utilization rate when playing or encoding videos with different resolutions, formats, and frame rates. It uses a software-driven collaborative approach to improve the hardware resource utilization efficiency of video playback and encoding in different scenarios, thereby reducing power consumption, balancing the bandwidth occupancy of the video codec, reducing bus bandwidth fluctuations, and thus reducing and smoothing the DDR frequency modulation frequency.

[0046] Those skilled in the art will understand that the system structure shown in the figure does not constitute a limitation on the embodiments of the present application, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0047] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0048] Those skilled in the art will understand that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will know that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0049] Based on the above-mentioned video codec dynamic frequency modulation architecture, various embodiments of the video codec dynamic frequency modulation method of the present application are proposed below.

[0050] like Figure 2 As shown, the video codec dynamic frequency modulation method can be applied to Figure 1 The video codec dynamic frequency modulation framework shown in the figure includes: an application layer, a codec layer, a driver layer, and a hardware layer, which are sequentially connected in communication; wherein the application layer includes an application terminal; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a hardware codec module; the video codec dynamic frequency modulation method may include but is not limited to steps S110 to S150.

[0051] Step S110: Through the hardware layer, a performance test is performed according to the preset frequency modulation gear to create an optimal frequency index table.

[0052] Step S120: When the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains video information of the video to be processed, and determines a video processing method.

[0053] Step S130: The driving layer records or updates the received video information into the node information of the corresponding user node through the information update module.

[0054] Step S140: The driver layer obtains the node information of each user node through the frequency selection control module; performs matching processing based on each node information and the optimal frequency index table to determine the user frequency of each user node; determines the maximum frequency value among the multiple user frequencies as the target frequency; and sets the operating frequency of the hardware codec module to the target frequency.

[0055] Step S150: The hardware encoding and decoding module performs encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

[0056] According to some embodiments of the present application, the codec layer further includes a performance monitoring module. The method further includes: monitoring the codec processing of the hardware codec module via the performance monitoring module of the codec layer; and when an abnormality of insufficient codec performance is detected, sending an operating frequency adjustment instruction to the driver layer. This allows the operating frequency of the hardware codec module to be adjusted in a timely manner to ensure efficient codec efficiency.

[0057] Specifically, abnormal situations of insufficient codec performance include: codec timeout, insufficient frame rate, etc.

[0058] It is understandable that the monitoring module in the codec layer is a thread performance_watch_pthread responsible for monitoring the current hardware codec performance. When the hardware prompts that the codec timeout or frame rate is insufficient, it will notify the driver layer through the ioctl function to further increase the operating frequency of the hardware codec module to improve the codec performance.

[0059] Furthermore, in step S120, after the codec layer receives at least one video to be processed from the application, the codec layer further includes: obtaining a preset configuration file; determining whether the dynamic frequency modulation function is enabled based on the configuration file; if so, enabling the dynamic frequency modulation function; if not, disabling the dynamic frequency modulation function. Related processing is performed only when the dynamic frequency modulation function is enabled.

[0060] Specifically, in step S130, the video information includes: a first video encoding format, a first frame rate, and a first resolution.

[0061] In some embodiments, step S140 includes: determining a first video encoding format, a first frame rate, and a first resolution from the received video information; saving the first video encoding format, the first frame rate, and the first resolution as an array; recording or updating the array to the node information of the corresponding user node; so as to be read subsequently.

[0062] For example, Figure 3 As shown, Figure 3 This is a flow chart of a codec layer sending video information to a driver layer, provided by an embodiment of the present application.

[0063] Step S301: receiving a video to be processed through an encoder and / or decoder.

[0064] Step S302: Determine whether the dynamic frequency modulation function is enabled through the configuration file; if not, end the process; if so, jump to step S303.

[0065] Step S303: Obtain video information of the video to be processed, wherein the video information includes: a first video encoding format, a first frame rate, and a first resolution.

[0066] Step S304: sending the video information to the driver layer through the Ioctl function, so that the driver layer records or updates the received video information into the node information of the corresponding user node.

[0067] Through steps S110 to S150, in the process of video encoding and decoding using the video codec dynamic frequency modulation architecture, first, through the hardware layer, performance testing is performed according to the preset frequency modulation gear, and an optimal frequency index table is created to provide a reliable reference for subsequent dynamic frequency modulation; secondly, when the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains the video information of the video to be processed and determines the video processing method; then, the driver layer records or updates the received video information to the node information of the corresponding user node through the information update module; then, the driver layer obtains the node information of each user node through the frequency selection control module; then, matching processing is performed based on each node information and the optimal frequency index table to determine the user frequency of each user node; then, the maximum frequency value among the multiple user frequencies is determined as the target frequency; finally, the operating frequency of the hardware codec module is set to the target frequency; the hardware codec module performs codec processing on the video to be processed according to the target frequency and the video processing method; thereby reducing codec power consumption and improving codec performance. In other words, the solution of the embodiment of the present application can dynamically adjust the operating frequency of the hardware codec module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing codec power consumption and improving codec performance. It can be seen that the dynamic frequency modulation method provided by the embodiment of the present application can achieve performance management of the hardware codec module in complex scenarios, and can adopt the optimal frequency point to achieve a balance between performance and power consumption in multi-user scenarios. It can also monitor the performance of the hardware codec module in real time and make decisions to adjust the frequency to improve performance.

[0068] According to some embodiments of the present application, step S110 is further described. Step S110: a performance test is performed according to a preset frequency modulation gear, and an optimal frequency index table is created, including but not limited to steps S111 to S116.

[0069] Step S111: determining a second video encoding format, a second resolution, and a second frame rate of a test source according to a preset frequency modulation level, and setting the hardware codec module to operate at a first test frequency.

[0070] Step S112: The hardware codec module decodes the test source according to the initially preset first test frequency to obtain the decoding frame rate per second of the test source by the hardware codec module; based on the comparison result of the decoding frame rate per second and the preset frame rate threshold, frequency debugging is performed to obtain the target decoding frequency in the decoding scenario.

[0071] Step S113: The hardware codec module encodes the test film source according to the initially preset second test frequency to obtain the encoding frame rate per second of the test film source by the hardware codec module; based on the comparison result of the encoding frame rate per second and the preset frame rate threshold, frequency debugging is performed to obtain the target encoding frequency in the encoding scenario.

[0072] Step S114: determining the optimal hardware frequency required for processing the test source according to the comparison result of the target decoding frequency and the target encoding frequency.

[0073] Step S115: forming a frequency mapping relationship according to the second video encoding format, the total range of candidate pixels, and the hardware frequency; the total range of candidate pixels is determined by the second resolution and the second frame rate.

[0074] Step S116: After processing multiple test sources in different formats, an optimal frequency index table is created according to the obtained multiple frequency mapping relationships.

[0075] Specifically, in step S111, the preset FM gears include: 720p@60fps, 1080p@30fps, 1080p@60fps, and 4k@30fps. It should be noted that 720p@60fps is a specification parameter for video display and video recording, and its specific meaning is as follows: "p" in 720p stands for "progressive", that is, line-by-line scanning; "720" refers to the vertical resolution of the video is 720 pixels; "fps" is the abbreviation of "frames per second", that is, the number of frames per second. "60fps" means that 60 frames of images can be displayed or recorded per second (that is, the number of image frames that can be displayed or recorded per second). Other preset FM gears can be understood with reference to this description and will not be repeated here.

[0076] According to some embodiments of the present application, step S112 is further described, wherein, based on the comparison result of the decoding frame rate per second and the preset frame rate threshold, a frequency debugging process is performed to obtain the target decoding frequency in the decoding scenario, including: when the decoding frame rate per second is equal to the preset frame rate threshold, the first test frequency is determined as the target decoding frequency; when the decoding frame rate per second is greater than the preset frame rate threshold, the first test frequency is reduced to a third test frequency, the hardware codec module decodes the test source according to the third test frequency, and when the decoding frame rate per second is equal to the preset frame rate threshold, the third test frequency is determined as the target decoding frequency. In this way, a target decoding frequency with better decoding effect in the decoding scenario is determined.

[0077] Similarly, in step S113, frequency debugging is performed based on the comparison result between the encoding frames per second and the preset frame rate threshold, and the target encoding frequency for the encoding scenario is obtained, including but not limited to: when the encoding frames per second is equal to the preset frame rate threshold, the second test frequency is determined as the target encoding frequency; when the encoding frames per second is greater than the preset frame rate threshold, the second test frequency is reduced to a fourth test frequency, and the hardware codec module encodes the test source according to the fourth test frequency. When the encoding frames per second is equal to the preset frame rate threshold, the fourth test frequency is determined as the target encoding frequency. In this way, a target encoding frequency with better encoding effect is determined for the encoding scenario.

[0078] Specifically, the preset frame rate threshold is the sum of the number of image frames that can be displayed or recorded per second in the preset frequency modulation gear and the performance margin, and the performance margin is the product of the number of image frames that can be displayed or recorded per second and the preset percentage.

[0079] According to some embodiments of the present application, step S114 is further described, and the optimal hardware frequency required for processing the test film source is determined based on the comparison result of the target decoding frequency and the target encoding frequency, including: when the target decoding frequency is greater than or equal to the target encoding frequency, the target decoding frequency is determined as the optimal hardware frequency required for processing the test film source; when the target encoding frequency is greater than the target decoding frequency, the target encoding frequency is determined as the optimal hardware frequency required for processing the test film source.

[0080] Through steps S111 to S116, the performance test process is completed, and an optimal frequency index table is created, providing a reliable reference for subsequent dynamic frequency modulation.

[0081] An example is given to further illustrate the process of creating the optimal frequency index table provided in the embodiment of the present application.

[0082] Based on common video codec usage scenarios, four frequency adjustment levels are defined for the hardware codec module: 720p@60fps, 1080p@30fps, 1080p@60fps, and 4k@30fps. The codec performance of the hardware codec module is tested using these four frequency adjustment levels. Due to performance differences between different decoders, performance testing of different codecs is required. Based on the tested codec performance, the corresponding hardware codec module frequencies are determined for each of the four levels, ensuring that the performance margin for each level is greater than 15%.

[0083] To determine the optimal frequency, first uniformly test the second video encoding format, second resolution, and second frame rate of the source video. For example, for an H264_720P@60fps source video, assume that the frequency currently set for the hardware decoding module is A, and obtain the frame rate per second decoded by the hardware module for the source video at this time, and compare whether it can meet the 69fps with the performance margin. If it can meet the requirements, it means that the currently set frequency A meets the performance requirements. The value of frequency A can be further lowered until the hardware decoding performance is close to 69fps. This means that the hardware decoding frequency at this time is one that can both meet the decoding performance requirements and minimize the power consumption of the decoding hardware. Similarly, for other video encoding formats, combining the four gears (720p@60fps, 1080p@30fps, 1080p@60fps, 4k@30fps) can obtain an optimal frequency index table. Since the current hardware only supports H264 encoding, after obtaining the optimal frequency in the encoding scenario, it is compared with the optimal frequency in the decoding scenario, and the higher value between the two is taken as the hardware frequency.

[0084] It should be noted that regarding performance margin, for example, if the performance required by the scene is 1280x720@60fps, then the decoding performance must be greater than or equal to 1280x720@(60*1.15)fps=1280x720@(69)fps. Based on the above system measured data, an optimal frequency index table is obtained. The optimal frequency index table can be used to determine the optimal target frequency as the operating frequency based on the video encoding format and the total pixel range.

[0085] like Figure 4As shown, it can be seen that the optimal frequency index table includes a mapping relationship between multiple candidate encoding formats, candidate pixel total ranges, and hardware frequencies. Among them, the total number of pixels is the product of the resolution and the frame rate. The total number of candidate pixels is determined by the two total numbers of pixels. It can be understood that the resolution gear corresponding to a video encoding format is divided into four, and the corresponding total number of pixels can be obtained. Since the resolution of the video is not necessarily standard, as long as the total number of pixels falls within a certain interval, the hardware frequency specified by the current gear is applicable. For example, the resolution of an H264 video is 480x360, which is suitable for the gear of Index=1, and the determined target frequency is a hardware frequency of 100Mhz.

[0086] It should be noted that the optimal frequency index table in the embodiment of the present application is obtained based on testing under a single-channel situation. For scenarios of multi-channel encoding, multi-channel decoding, or multi-channel simultaneous encoding and decoding, the optimal frequency index table is still applicable.

[0087] According to some embodiments of the present application, the optimal frequency index table includes mappings between multiple candidate encoding formats, candidate pixel total ranges, and hardware frequencies. Further describing step S140, wherein matching processing is performed based on each node information and the optimal frequency index table to determine the user frequency of each user node, includes but is not limited to steps S141 to S146.

[0088] Step S141: Obtain a first video encoding format, a first frame rate, and a first resolution of video information from a node information.

[0089] Step S142: Calculate the total number of pixels per second according to the first frame rate and the first resolution.

[0090] Step S143: Determine a target encoding format from candidate encoding formats according to the first video encoding format.

[0091] Step S144: Determine the target pixel total amount range into which the total amount of pixels per second falls from the four candidate pixel total amount ranges corresponding to the target coding format.

[0092] Step S145: Determine the hardware frequency corresponding to the target pixel total amount range as the user frequency.

[0093] Step S146: Repeat the matching process until the user frequency of each user node is determined.

[0094] According to some embodiments of the present application, before matching the node information and the optimal frequency index table, the method also includes: determining whether the number of node information obtained by scanning is equal to the number of user nodes; if so, performing matching processing; if not, continuing to obtain node information.

[0095] Through steps S141 to S146, the optimal user frequency of each user node can be determined, laying the foundation for subsequently selecting the maximum frequency value from multiple user frequencies to determine as the target frequency; the determined target frequency can meet the needs of all user nodes.

[0096] Specifically, the dynamic frequency modulation strategy provided by the embodiment of the present application is further explained.

[0097] After receiving the video information sent by the codec layer, the driver layer will call the following interface to arbitrate and execute dynamic frequency modulation.

[0098] The information update module uses the update_case_load_param interface to record the current user's video information in an idle or assigned information node, load_infos. This step aims to manage the video information of different users in multi-user scenarios. The frequency selection control module uses the adjust_ve_freq_by_case interface to coordinate the video information of all users recorded in all information nodes, load_infos, and calculate the total number of decoding channels, total number of encoding channels, resolution, frame rate, and video format. The optimal frequency index table of the hardware codec module is used to find the optimal frequency for each user. By determining the optimal frequency currently required by all users, the highest frequency point is found to meet the needs of all users and is assigned to the hardware codec module.

[0099] It's understandable that the codec information used by different user nodes is recorded in corresponding information nodes in the driver, including whether it's an encoding or decoding scenario, and the corresponding resolution, frame rate, and video encoding format are all recorded and stored in an array. The adjust_ve_freq_by_case function reads this array to generate statistics for all channels currently using the hardware codec module.

[0100] The current design uses a single hardware codec module, which is time-multiplexed for each user. Therefore, the frequency is switched to a different one for each time slice allocated to the corresponding user. This high-frequency switching of the hardware operating frequency also affects performance. Therefore, adjust_ve_freq_by_case obtains the frequency corresponding to each user from the "Optimal Frequency Index Table" and then sets the highest value among all user frequencies as the target frequency for the hardware module to meet the needs of each user.

[0101] In summary, it should be emphasized that the embodiments of the present application can: create an optimal frequency index table through performance testing, and the hardware codec module implements a dynamic frequency modulation strategy based on the optimal frequency index table and the information of the actually processed video, and supports multi-threaded and multi-process scenarios; and can implement a frequency modulation strategy in complex scenarios: if there are three-way decoding, namely H264, H265, and VP9 formats, since the performance of these three decoders is different, it is necessary to unify the task load or decoder performance; wherein, unified processing refers to: determining the optimal user frequency of the three-way decoding processing respectively, and determining the maximum value from the three user frequencies as the target frequency; making the hardware codec module work at the target frequency and perform three-way decoding processing at the same time to meet the needs of each decoding processing. In addition, there is a strategy of dynamically adjusting the frequency of the hardware codec module in real time: detecting the codec performance of all channels in real time, and immediately increasing the working frequency of the hardware codec module if the codec performance is insufficient.

[0102] like Figure 5 As shown, the present invention also provides an electronic device, including:

[0103] The processor 501 may be implemented as a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0104] The memory 502 can be implemented in the form of a read-only memory, a static storage device, a dynamic storage device, or a random access memory. The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 502 and are called by the processor 501 to execute the video encoding and decoding dynamic frequency modulation method of the embodiments of this application.

[0105] Input / output interface 503, used to implement information input and output;

[0106] Communication interface 504, used to implement communication interaction between the apparatus and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0107] Bus 505 , which transmits information between various components of the device (e.g., processor 501 , memory 502 , input / output interface 503 , and communication interface 504 );

[0108] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via a bus 505 .

[0109] It is understandable that electronic devices that can be applied to the dynamic frequency modulation method of the embodiment of the present application include but are not limited to: mid-to-high-end car multimedia products on the Android / Linux platform, network set-top box products, building security monitoring central controller products, etc.

[0110] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video encoding and decoding dynamic frequency modulation method as described in the above embodiment.

[0111] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory optionally includes a memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of the above-mentioned networks include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0112] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0113] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above implementation mode. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the present application.

Claims

1. A video encoding and decoding dynamic frequency modulation method, characterized in that: Applicable to a video codec dynamic frequency modulation architecture, the video codec dynamic frequency modulation architecture includes: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially communicatively connected; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a hardware codec module; The method comprises: Through the hardware layer, a performance test is performed according to the preset frequency modulation gear to create an optimal frequency index table; When the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains video information of the video to be processed and determines a video processing method; The driving layer records or updates the received video information into the node information of the corresponding user node through the information update module; The driver layer obtains node information of each user node through the frequency selection control module; performs matching processing based on the node information and the optimal frequency index table to determine the user frequency of each user node; determines the maximum frequency value among the multiple user frequencies as the target frequency; and sets the operating frequency of the hardware codec module to the target frequency; The hardware encoding and decoding module performs encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

2. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: The performance test process is performed according to the preset frequency modulation gear to create the optimal frequency index table, including: Determine the second video encoding format, the second resolution and the second frame rate of the test source according to the preset FM gear, and set the hardware codec module to operate at the first test frequency; The hardware codec module decodes the test source according to an initially preset first test frequency to obtain a decoding frame rate per second of the test source by the hardware codec module; and performs frequency debugging based on a comparison result of the decoding frame rate per second with a preset frame rate threshold to obtain a target decoding frequency in a decoding scenario; The hardware codec module encodes the test source according to an initially preset second test frequency to obtain a frame rate per second encoded by the hardware codec module for the test source; and performs frequency debugging based on a comparison result of the encoded frame rate per second with a preset frame rate threshold to obtain a target encoding frequency in the encoding scenario; Determining an optimal hardware frequency required for processing the test source according to a comparison result of the target decoding frequency and the target encoding frequency; forming a frequency mapping relationship according to the second video encoding format, the total range of candidate pixels, and the hardware frequency; wherein the total range of candidate pixels is determined by the second resolution and the second frame rate; After processing a plurality of test sources in different formats, the optimal frequency index table is created according to the obtained plurality of frequency mapping relationships.

3. The video encoding and decoding dynamic frequency modulation method according to claim 2, characterized in that: The frequency debugging process is performed according to the comparison result of the decoding frame rate per second and the preset frame rate threshold to obtain the target decoding frequency in the decoding scenario, including: When the decoding frame rate per second is equal to a preset frame rate threshold, determining the first test frequency as a target decoding frequency; When the decoding frame rate per second is greater than the preset frame rate threshold, the first test frequency is reduced to a third test frequency, and the hardware codec module decodes the test film source according to the third test frequency. When the decoding frame rate per second is equal to the preset frame rate threshold, the third test frequency is determined as the target decoding frequency.

4. The video encoding and decoding dynamic frequency modulation method according to claim 2, characterized in that: The determining, based on the comparison result of the target decoding frequency and the target encoding frequency, the optimal hardware frequency required for processing the test source includes: When the target decoding frequency is greater than or equal to the target encoding frequency, determining the target decoding frequency as the optimal hardware frequency required for processing the test slice source; When the target encoding frequency is greater than the target decoding frequency, the target encoding frequency is determined as the optimal hardware frequency required for processing the test slice source.

5. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: The optimal frequency index table includes a mapping relationship between multiple candidate encoding formats, candidate pixel total ranges, and hardware frequencies; The performing matching processing according to the information of each node and the optimal frequency index table to determine the user frequency of each user node includes: Obtaining a first video encoding format, a first frame rate, and a first resolution of the video information from one of the node information; Calculate the total number of pixels per second according to the first frame rate and the first resolution; Determining a target encoding format from the candidate encoding formats according to the first video encoding format; Determine a target pixel total amount range into which the total amount of pixels per second falls from the four candidate pixel total amount ranges corresponding to the target coding format; Determining a hardware frequency corresponding to the target total pixel range as a user frequency; The matching process is repeated until the user frequency of each user node is determined.

6. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: Before matching the information of each node with the optimal frequency index table, the method further includes: Determining whether the number of the node information obtained by scanning is equal to the number of the user nodes; If yes, then the matching process is performed; if no, then the node information is continuously acquired.

7. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: The codec layer further includes a performance monitoring module; the method further includes: Monitoring the encoding and decoding processing of the hardware encoding and decoding module through the performance monitoring module of the encoding and decoding layer; When an abnormal situation of insufficient coding and decoding performance is detected, an operating frequency adjustment instruction is sent to the driving layer.

8. A video codec dynamic frequency modulation architecture, characterized in that: include: The application layer, codec layer, driver layer, and hardware layer are sequentially connected in communication; wherein the application layer includes an application terminal; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a hardware codec module; The hardware layer is used to: perform performance testing according to the preset frequency modulation gear and create an optimal frequency index table; The codec layer is used to: receive at least one video to be processed sent by the application end, and when the dynamic frequency modulation function is enabled, obtain video information of the video to be processed and determine a video processing method; The driving layer is used to: record or update the received video information into the node information of the corresponding user node through the information update module; The driver layer is further configured to: obtain node information of each user node through the frequency selection control module; perform matching processing based on the node information and the optimal frequency index table to determine the user frequency of each user node; determine the maximum frequency value among the multiple user frequencies as the target frequency; and set the operating frequency of the hardware codec module to the target frequency; The hardware encoding and decoding module is further configured to perform encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

9. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video codec dynamic frequency modulation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video encoding and decoding dynamic frequency modulation method according to any one of claims 1 to 7.

Images