Video coding and decoding dynamic frequency modulation method and architecture, equipment and medium thereof

Abstract

The invention discloses a dynamic frequency modulation method for video coding and decoding, an architecture thereof, equipment and a medium, and relates to the technical field of video coding and decoding. The method comprises the following steps: performing performance test processing according to a preset frequency modulation gear to create an optimal frequency index table; at least one to-be-processed video is received, and video information of the to-be-processed video is acquired and a video processing mode is determined under the condition that the dynamic frequency modulation function is enabled; recording or updating the received video information to the node information of the corresponding user node; acquiring node information of each user node; performing matching processing according to the information of each node and the optimal frequency index table, and determining the user frequency of each user node; determining the maximum frequency value in the plurality of user frequencies as a target frequency; and the hardware coding and decoding module performs coding and decoding processing on the video to be processed according to the target frequency and the video processing method. The working frequency of the video coding and decoding hardware module can be dynamically adjusted, so that the power consumption is reduced, and the coding and decoding performance is improved.

Description

Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to a video coding and decoding dynamic frequency modulation method and its architecture, device, and medium. Background Art

[0002] With the rapid development of integrated circuit technology and multimedia audio and video technology, video images have become an important and indispensable information transmission medium in people's daily life and work. Electronic products with functions such as video image acquisition, compression, transmission, decompression, and display have been widely used in various industries, such as driving recorders, sports DVs, security building monitoring, network TV set-top boxes, tablet computers, etc. The information capacity of video images is relatively large. For example, for 4k@30fps+8bit+yuv420 videos, the size of one image is 11.86MB, the size of one second of image is 355MB, and the size of one minute of image is 2.136GB. If the original data of the image is directly stored and transmitted, a large amount of storage resources and network bandwidth resources will be consumed. In order to reduce resource consumption, a video encoder is usually used to compress the image collected from the camera end, and the compression rate can reach more than 100 times. The compressed video code stream data is then transmitted through the network or stored locally, thereby saving network bandwidth resources and storage resources. For the user end, it is necessary to first use a video decoder to decompress the video code stream data, restore the video image, and finally output it to a display device such as an LCD for display. Since the compression and decompression algorithms of video images are relatively complex and require a large amount of computing resources, a dedicated video codec digital integrated circuit is generally used to process the encoding and decoding of the image. The video codec digital integrated circuit (hereinafter referred to as the video codec module) can operate at multiple different clock frequencies. Increasing the clock frequency within a certain range will improve the processing capability of the video codec module, but it will also lead to an increase in power consumption. It is necessary to find the optimal clock frequency for the hardware module to work, balance performance and power consumption, and achieve the goal of meeting the requirements of existing usage scenarios while reducing power consumption. How to find the optimal clock frequency for reliable hardware modules is an urgent problem to be solved. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a video codec dynamic frequency modulation method and its architecture, device, and medium, which can dynamically adjust the operating frequency of the video codec hardware module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing power consumption and improving codec performance.

[0004] In a first aspect, an embodiment of the present application provides a video codec dynamic frequency modulation method, which is applied to a video codec dynamic frequency modulation architecture, wherein the video codec dynamic frequency modulation architecture comprises: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially connected in communication; wherein the application layer comprises an application end; the codec layer comprises: at least one decoder and at least one encoder; the driver layer comprises a plurality of user nodes, an information update module, and a frequency selection control module; the hardware layer comprises a video codec hardware module;

[0005] The method comprises:

[0006] Through the hardware layer, a performance test process is performed according to a preset frequency modulation gear, and an optimal frequency index table is created;

[0007] When the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains video information of the video to be processed and determines a video processing method;

[0008] The driving layer records or updates the received video information into the node information of the corresponding user node through the information updating module;

[0009] The driver layer obtains node information of each user node through the frequency selection control module; performs matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determines the maximum frequency value among the multiple user frequencies as the target frequency; and sets the operating frequency of the hardware codec module to the target frequency;

[0010] The hardware encoding and decoding module performs encoding and decoding processing on the video to be processed according to the target frequency and the video processing method.

[0011] In a second aspect, an embodiment of the present application provides a video codec dynamic frequency modulation architecture, including: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially connected in communication; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a video codec hardware module;

[0012] The hardware layer is used to: perform performance test processing according to the preset frequency modulation gear and create an optimal frequency index table;

[0013] The codec layer is used to: receive at least one video to be processed sent by the application end, and when the dynamic frequency modulation function is enabled, obtain video information of the video to be processed and determine a video processing method;

[0014] The driving layer is used to: record or update the received video information into the node information of the corresponding user node through the information update module;

[0015] The driver layer is also used to: obtain node information of each user node through the frequency selection control module; perform matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determine the maximum frequency value among the multiple user frequencies as the target frequency; set the working frequency of the hardware codec module to the target frequency;

[0016] The hardware encoding and decoding module is further used to: perform encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

[0017] In a third aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video encoding and decoding dynamic frequency modulation method as described in any one of the embodiments of the first aspect.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video encoding and decoding dynamic frequency modulation method as described in any one of the embodiments of the first aspect.

[0019] The embodiments of the present application include: a video codec dynamic frequency modulation architecture includes: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially connected in communication; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a video codec hardware module. In the process of video encoding and decoding using the video codec dynamic frequency modulation architecture, first, through the hardware layer, performance test processing is performed according to the preset frequency modulation gear, and an optimal frequency index table is created to provide a reliable reference for subsequent dynamic frequency modulation; secondly, when the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains the video information of the video to be processed and determines the video processing method; then, the driver layer records or updates the received video information to the node information of the corresponding user node through the information update module; then, the driver layer obtains the node information of each user node through the frequency selection control module; then, matching processing is performed according to each node information and the optimal frequency index table to determine the user frequency of each user node; then, the maximum frequency value among multiple user frequencies is determined as the target frequency; finally, the operating frequency of the hardware codec module is set to the target frequency; the hardware codec module performs codec processing on the video to be processed according to the target frequency and the video processing method; thereby reducing the power consumption of codec and improving the codec performance. That is to say, the solution of the embodiment of the present application can dynamically adjust the operating frequency of the video codec hardware module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing the codec power consumption and improving the codec performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural diagram of a video codec dynamic frequency modulation architecture provided by an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a video encoding and decoding dynamic frequency modulation method provided by an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a process of sending video information from a coding and decoding layer to a driving layer provided by an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an optimal frequency index table provided by an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0026] It should be understood that in the description of the present application, the orientation descriptions, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0027] It should be noted that although a logical order is shown in the flowchart in the description of the present application, in some cases, the steps shown or described may be performed in an order different from that in the flowchart. In the description of the present application, a number of means one or more, and a plurality of means two or more. The description of "first" and "second" is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] First, some terms used in this application are explained:

[0030] Ioctl: is a system call function widely used in UNIX and UNIX-like systems (such as Linux). Its full name is "input / output control". It is used for communication between device drivers and device files, allowing user space programs to send or receive control commands to kernel space device drivers.

[0031] The present application provides a video codec dynamic frequency modulation method and its architecture, electronic device and computer-readable storage medium, the method comprising: creating an optimal frequency index table by performing performance test processing according to a preset frequency modulation gear; receiving at least one video to be processed, obtaining video information of the video to be processed and determining a video processing method when the dynamic frequency modulation function is enabled; recording or updating the received video information to the node information of the corresponding user node; obtaining the node information of each user node; performing matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determining the maximum frequency value among multiple user frequencies as the target frequency; the hardware codec module performs codec processing on the video to be processed according to the target frequency and the video processing method. The present application can dynamically adjust the operating frequency of the video codec hardware module, thereby reducing power consumption and improving codec performance.

[0032] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0033] like Figure 1 As shown, the embodiment of the present application provides a video codec dynamic frequency modulation architecture, including: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially connected in communication; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a video codec hardware module. The codec layer also includes a performance monitoring module.

[0034] The hardware layer is used to perform performance testing according to the preset frequency modulation gear and create an optimal frequency index table.

[0035] The codec layer is used to: receive at least one video to be processed sent by the application end, and when the dynamic frequency modulation function is enabled, obtain video information of the video to be processed and determine the video processing method.

[0036] The driving layer is used to record or update the received video information into the node information of the corresponding user node through the information update module.

[0037] The driver layer is also used to: obtain the node information of each user node through the frequency selection control module; perform matching processing based on each node information and the optimal frequency index table to determine the user frequency of each user node; determine the maximum frequency value among multiple user frequencies as the target frequency; and set the operating frequency of the hardware codec module to the target frequency.

[0038] The hardware codec module is also used to: perform codec processing on the video to be processed according to the target frequency and the video processing method.

[0039] The performance monitoring module is used to: monitor the encoding and decoding processing of the hardware encoding and decoding module; when an abnormal situation of insufficient encoding and decoding performance is detected, a working frequency adjustment instruction is sent to the driver layer.

[0040] It is understandable that the video codec dynamic frequency modulation software architecture of the present application is mainly divided into four layers: application layer, codec layer, driver layer, and hardware layer. Among them, the logic of the frequency modulation software solution is implemented in the driver layer. Further, the application layer, codec layer, driver layer, and hardware layer of the video codec dynamic frequency modulation architecture are described.

[0041] The application layer includes the application end, which is also the user of the codec layer. It is responsible for calling the codec layer to implement related functions of video processing and is the user of the codec function.

[0042] The codec layer is responsible for managing the entire codec hardware module, encapsulating the underlying complex interfaces and operation logic, and providing a simple interface for the application layer to implement codec functions.

[0043] The Linux driver layer can also implement multi-user management. For the hardware codec module, multiple users means that there may be multiple processes or multiple threads using the hardware codec module at the same time. For multi-user management, the driver layer creates a user node that can represent identity information for each user, which is helpful for knowing the identity of the user who is currently using the hardware codec module. It should be noted that the driver layer performs unified management after receiving the video information sent by the user, and adjusts the performance of the hardware codec module by looking up the table to find the most suitable working frequency. It can timely adopt the best working frequency of the hardware codec module for complex application scenarios such as multi-process, multi-thread, and multiple codec formats.

[0044] According to the video codec dynamic frequency modulation architecture provided in the embodiment of the present application, in the process of performing video coding and decoding using the video codec dynamic frequency modulation architecture, first, a performance test is performed through the hardware layer according to the preset frequency modulation gear, and an optimal frequency index table is created to provide a reliable reference for subsequent dynamic frequency modulation; secondly, when the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains the video information of the video to be processed and determines the video processing method; then, the driver layer records or updates the received video information to the node information of the corresponding user node through the information update module; then, the driver layer obtains the node information of each user node through the frequency selection control module; then, matching processing is performed according to each node information and the optimal frequency index table to determine the user frequency of each user node; then, the maximum frequency value among multiple user frequencies is determined as the target frequency; finally, the operating frequency of the hardware codec module is set to the target frequency; the hardware codec module performs codec processing on the video to be processed according to the target frequency and the video processing method; thereby reducing the power consumption of codec and improving the codec performance. It can be seen that the video codec dynamic frequency modulation architecture of the embodiment of the present application can dynamically adjust the operating frequency of the video codec hardware module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing the codec power consumption and improving the codec performance.

[0045] It should be emphasized that, at present, there is no dynamic frequency modulation scheme for video codec modules, and CPU is often used for dynamic frequency modulation; specifically, the dynamic frequency modulation of CPU is to obtain the CPU occupancy rate at the current frequency, and determine the working frequency according to the CPU occupancy rate. This application aims at the high power consumption in video codec application scenarios, which is specifically manifested as the problem of low energy efficiency utilization when playing or encoding videos with different resolutions, formats, and frame rates. It adopts software-driven collaborative method to improve the hardware resource utilization efficiency of video playback and encoding in different scenarios, thereby reducing power consumption, balancing the bandwidth occupancy of video codecs, and reducing bus bandwidth fluctuations, thereby reducing and smoothing the DDR frequency modulation frequency.

[0046] Those skilled in the art will appreciate that the system structure shown in the figure does not constitute a limitation on the embodiments of the present application, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0047] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0048] Those skilled in the art will appreciate that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0049] Based on the above-mentioned video codec dynamic frequency modulation architecture, various embodiments of the video codec dynamic frequency modulation method of the present application are proposed below.

[0050] like Figure 2 As shown, the video codec dynamic frequency modulation method can be applied to Figure 1 The video codec dynamic frequency modulation framework shown. The video codec dynamic frequency modulation architecture includes: an application layer, a codec layer, a driver layer, and a hardware layer that are sequentially connected in communication; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes multiple user nodes, an information update module, and a frequency selection control module; the hardware layer includes a video codec hardware module; the video codec dynamic frequency modulation method may include but is not limited to steps S110 to S150.

[0051] Step S110: Through the hardware layer, a performance test is performed according to the preset frequency modulation level to create an optimal frequency index table.

[0052] Step S120: When the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains video information of the video to be processed and determines a video processing method.

[0053] Step S130: The driving layer records or updates the received video information into the node information of the corresponding user node through the information updating module.

[0054] Step S140: The driver layer obtains the node information of each user node through the frequency selection control module; performs matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determines the maximum frequency value among multiple user frequencies as the target frequency; and sets the operating frequency of the hardware codec module to the target frequency.

[0055] Step S150: the hardware encoding and decoding module performs encoding and decoding processing on the video to be processed according to the target frequency and the video processing method.

[0056] According to some embodiments of the present application, the codec layer further includes a performance monitoring module; the method further includes: monitoring the codec processing of the hardware codec module through the performance monitoring module of the codec layer; when an abnormal situation of insufficient codec performance is detected, sending a working frequency adjustment instruction to the driver layer. Thus, the working frequency of the hardware codec module can be adjusted in time to ensure efficient codec efficiency.

[0057] Specifically, abnormal situations of insufficient codec performance include: codec timeout, insufficient frame rate, etc.

[0058] It is understandable that the monitoring module in the codec layer is a thread performance_watch_pthread responsible for monitoring the current hardware codec performance. When the hardware prompts that the codec timeout or frame rate is insufficient, the driver layer will be notified through the ioctl function to further increase the working frequency of the hardware codec module to improve the codec performance.

[0059] Furthermore, in step S120, after the codec layer receives at least one video to be processed sent by the application end, the method further includes: the codec layer obtains a preset configuration file; determines whether the dynamic frequency modulation function is enabled through the configuration file, and if so, starts the dynamic frequency modulation function; if not, disables the dynamic frequency modulation function. Relevant processing is performed only when the dynamic frequency modulation function is enabled.

[0060] Specifically, in step S130, the video information includes: a first video encoding format, a first frame rate, and a first resolution.

[0061] In some embodiments, step S140 includes: determining a first video encoding format, a first frame rate, and a first resolution from the received video information; saving the first video encoding format, the first frame rate, and the first resolution as an array; recording or updating the array to the node information of the corresponding user node; for subsequent reading.

[0062] For example, Figure 3 As shown, Figure 3 It is a schematic diagram of a process of sending video information from a codec layer to a driver layer provided by an embodiment of the present application.

[0063] Step S301: receiving a video to be processed through an encoder and / or a decoder.

[0064] Step S302: determine whether the dynamic frequency modulation function is enabled through the configuration file; if not, end the process; if so, jump to step S303.

[0065] Step S303: Obtain video information of the video to be processed, wherein the video information includes: a first video encoding format, a first frame rate, and a first resolution.

[0066] Step S304: Send the video information to the driver layer through the Ioctl function, so that the driver layer records or updates the received video information into the node information of the corresponding user node.

[0067] Through step S110 to step S150, in the process of video encoding and decoding using the video codec dynamic frequency modulation architecture, first, through the hardware layer, a performance test is performed according to the preset frequency modulation gear, and an optimal frequency index table is created to provide a reliable reference for subsequent dynamic frequency modulation; secondly, when the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains the video information of the video to be processed and determines the video processing method; then, the driver layer records or updates the received video information to the node information of the corresponding user node through the information update module; then, the driver layer obtains the node information of each user node through the frequency selection control module; then, matching processing is performed according to each node information and the optimal frequency index table to determine the user frequency of each user node; then, the maximum frequency value among multiple user frequencies is determined as the target frequency; finally, the operating frequency of the hardware codec module is set to the target frequency; the hardware codec module performs codec processing on the video to be processed according to the target frequency and the video processing method; thereby reducing the power consumption of codec and improving the codec performance. That is to say, the scheme of the embodiment of the present application can dynamically adjust the working frequency of the video codec hardware module based on the constructed optimal frequency index table and the video information of the video to be processed, thereby reducing the power consumption of the codec and improving the codec performance. It can be seen that the dynamic frequency modulation method provided by the embodiment of the present application can achieve the performance management of the hardware codec module in complex scenes, and can adopt the optimal frequency point in a multi-user scene to achieve a balance between performance and power consumption; it can monitor the performance of the hardware codec module in real time, and make decisions to adjust the frequency to improve performance.

[0068] According to some embodiments of the present application, step S110 is further described. Step S110: perform performance test processing according to a preset frequency modulation gear and create an optimal frequency index table, including but not limited to steps S111 to S116.

[0069] Step S111: determining the second video encoding format, the second resolution and the second frame rate of the test source according to the preset frequency modulation level, and setting the hardware codec module to operate at the first test frequency.

[0070] Step S112: The hardware codec module decodes the test film source according to the initially preset first test frequency to obtain the decoding frame rate per second of the test film source by the hardware codec module; based on the comparison result between the decoding frame rate per second and the preset frame rate threshold, frequency debugging is performed to obtain the target decoding frequency in the decoding scenario.

[0071] Step S113: The hardware codec module encodes the test film source according to the initially preset second test frequency, and obtains the encoding frame rate per second of the test film source by the hardware codec module; based on the comparison result of the encoding frame rate per second and the preset frame rate threshold, frequency debugging is performed to obtain the target encoding frequency in the encoding scenario.

[0072] Step S114: determining the optimal hardware frequency required for processing the test source according to the comparison result between the target decoding frequency and the target encoding frequency.

[0073] Step S115: forming a frequency mapping relationship according to the second video encoding format, the total range of candidate pixel points, and the hardware frequency; the total range of candidate pixel points is determined by the second resolution and the second frame rate.

[0074] Step S116: after processing a plurality of test sources in different formats, an optimal frequency index table is created according to the obtained plurality of frequency mapping relationships.

[0075] Specifically, in step S111, the preset FM gears include: 720p@60fps, 1080p@30fps, 1080p@60fps, and 4k@30fps. It should be noted that 720p@60fps is a specification parameter for video display and video recording, and its specific meaning is as follows: "p" in 720p stands for "progressive", that is, line-by-line scanning; "720" means that the vertical resolution of the video is 720 pixels; "fps" is the abbreviation of "frames per second", that is, the number of frames per second. "60fps" means that 60 frames of images can be displayed or recorded per second (that is, the number of image frames that can be displayed or recorded per second). Other preset FM gears can be understood with reference to this description and will not be repeated here.

[0076] According to some embodiments of the present application, step S112 is further described, wherein, according to the comparison result of the decoding frame rate per second and the preset frame rate threshold, a frequency debugging process is performed to obtain a target decoding frequency in a decoding scenario, including: when the decoding frame rate per second is equal to the preset frame rate threshold, the first test frequency is determined as the target decoding frequency; when the decoding frame rate per second is greater than the preset frame rate threshold, the first test frequency is reduced to a third test frequency, and the hardware codec module decodes the test source according to the third test frequency, and when the decoding frame rate per second is equal to the preset frame rate threshold, the third test frequency is determined as the target decoding frequency. In this way, a target decoding frequency with a better decoding effect in a decoding scenario is determined.

[0077] Similarly, in step S113, frequency debugging is performed based on the comparison result between the encoding frame rate per second and the preset frame rate threshold, and the target encoding frequency in the encoding scenario is obtained, including but not limited to: when the encoding frame rate per second is equal to the preset frame rate threshold, the second test frequency is determined as the target encoding frequency; when the encoding frame rate per second is greater than the preset frame rate threshold, the second test frequency is reduced to the fourth test frequency, and the hardware encoding and decoding module encodes the test source according to the fourth test frequency. When the encoding frame rate per second is equal to the preset frame rate threshold, the fourth test frequency is determined as the target encoding frequency. In this way, a target encoding frequency with better encoding effect in the encoding scenario is determined.

[0078] Specifically, the preset frame rate threshold is the sum of the number of image frames that can be displayed or recorded per second in the preset frequency modulation gear and the performance margin, and the performance margin is the product of the number of image frames that can be displayed or recorded per second and the preset percentage.

[0079] According to some embodiments of the present application, step S114 is further described, and the optimal hardware frequency required for processing the test film source is determined based on the comparison result of the target decoding frequency and the target encoding frequency, including: when the target decoding frequency is greater than or equal to the target encoding frequency, the target decoding frequency is determined as the optimal hardware frequency required for processing the test film source; when the target encoding frequency is greater than the target decoding frequency, the target encoding frequency is determined as the optimal hardware frequency required for processing the test film source.

[0080] Through step S111 to step S116, the performance test process is completed, and the optimal frequency index table is created to provide a reliable reference for the subsequent implementation of dynamic frequency modulation.

[0081] An example is given to further illustrate the process of creating the optimal frequency index table provided in the embodiment of the present application.

[0082] According to common video codec usage scenarios, four frequency adjustment gears are set for adjusting the frequency of the hardware codec module: 720p@60fps, 1080p@30fps, 1080p@60fps, and 4k@30fps. Combined with the above four frequency adjustment gears of the hardware codec module, the codec performance of the hardware codec module is tested. Since the performance of different decoders varies, performance tests need to be performed on different codecs. The hardware codec module frequencies corresponding to the four gears are determined based on the tested codec performance numbers, and it is ensured that the performance margin of the hardware codec module frequency corresponding to each gear must be greater than 15%.

[0083] For the determination of the optimal frequency, firstly, the second video encoding format, the second resolution and the second frame rate of the test source are unified. For example, it is a H264_720P@60fps source. Assume that the frequency currently set for the hardware decoding module is A, and obtain the frame rate of the hardware module decoding the source per second at this time, and compare whether it can meet the 69fps with the performance margin. If it can meet, it means that the currently set frequency A meets the performance requirements. The value of frequency A can be further lowered until the hardware decoding performance is close to 69fps, which means that the hardware decoding frequency at this time is a frequency that can meet the decoding performance requirements and minimize the power consumption of the decoding hardware. By analogy, for other video encoding formats, an optimal frequency index table can be obtained by combining 4 gears (720p@60fps, 1080p@30fps, 1080p@60fps, 4k@30fps). Since the current hardware only supports H264 encoding, after obtaining the optimal frequency in the encoding scenario, it is compared with the optimal frequency in the decoding scenario, and the higher value between the two is taken as the hardware frequency.

[0084] It should be noted that regarding the performance margin, for example, if the performance required by the scene is 1280x720@60fps, then the decoding performance is greater than or equal to 1280x720@(60*1.15)fps=1280x720@(69)fps. Based on the above system measured data, an optimal frequency index table is obtained. The optimal frequency index table can be used to determine the optimal target frequency as the operating frequency based on the video encoding format and the total pixel range.

[0085] like Figure 4 As shown, it can be seen that the optimal frequency index table includes a mapping relationship between multiple candidate encoding formats, candidate pixel total ranges, and hardware frequencies. Among them, the total number of pixels is the product of the resolution and the frame rate. The total range of candidate pixel points is determined by two pixel totals. It can be understood that the resolution gear corresponding to a video encoding format is divided into four, and the corresponding total number of pixels can be obtained. Since the resolution of the video is not necessarily standard, as long as the total number of pixels falls within a certain interval, the hardware frequency specified by the current gear is applicable. For example, the resolution of an H264 video is 480x360, which is suitable for the gear of Index=1, and the determined target frequency is a hardware frequency of 100Mhz.

[0086] It should be noted that the optimal frequency index table in the embodiment of the present application is obtained based on testing under a single-channel case. For scenarios of multi-channel encoding, multi-channel decoding, or multi-channel encoding and decoding, the optimal frequency index table is still applicable.

[0087] According to some embodiments of the present application, the optimal frequency index table includes a mapping relationship between multiple candidate encoding formats, candidate pixel total ranges, and hardware frequencies. Further describing step S140, wherein matching processing is performed based on each node information and the optimal frequency index table to determine the user frequency of each user node, including but not limited to steps S141 to S146.

[0088] Step S141: Obtain a first video encoding format, a first frame rate, and a first resolution of video information from a node information.

[0089] Step S142: Calculate the total number of pixels per second according to the first frame rate and the first resolution.

[0090] Step S143: determining a target encoding format from candidate encoding formats according to the first video encoding format.

[0091] Step S144: Determine the target pixel total amount range into which the total amount of pixels per second falls from the four candidate pixel total amount ranges corresponding to the target coding format.

[0092] Step S145: Determine the hardware frequency corresponding to the target pixel total amount range as the user frequency.

[0093] Step S146: Repeat the matching process until the user frequency of each user node is determined.

[0094] According to some embodiments of the present application, before matching each node information with the optimal frequency index table, the method also includes: determining whether the amount of node information obtained by scanning is equal to the number of user nodes; if so, performing matching processing; if not, continuing to obtain node information.

[0095] Through steps S141 to S146, the optimal user frequency of each user node can be determined, laying a foundation for subsequently selecting the maximum frequency value from multiple user frequencies to determine as the target frequency; the determined target frequency can meet the needs of all user nodes.

[0096] Specifically, the dynamic frequency modulation strategy provided by the embodiment of the present application is further explained.

[0097] After receiving the video information sent by the codec layer, the driver layer will call the following interface to arbitrate and execute dynamic frequency modulation.

[0098] The information update module records the current user's video information to the idle or updated information node load_infos assigned to the user through the update_case_load_param interface. The purpose of this step is to manage the video information of different users in multi-user scenarios. The frequency selection control module coordinates the video information of all users recorded in all information nodes load_infos through the adjust_ve_freq_by_case interface, and counts the total number of decoding channels, the total number of encoding channels, resolution, frame rate, and video format. According to the optimal frequency index table of the hardware codec module, the optimal frequency that each user needs to use is found. By judging the optimal frequency currently required by all users, the highest frequency point is found to meet the needs of all users, and it is set to the hardware codec module.

[0099] It is understandable that the codec information used by different user nodes has corresponding information nodes in the driver to save and record, including whether it is an encoding scene or a decoding scene, and the corresponding resolution, frame rate and video encoding format will be recorded and saved in the array. In the adjust_ve_freq_by_case function, this array will be read to count the status of all channels currently using the hardware codec module.

[0100] Currently, there is only one hardware codec module designed, and each user uses this module in time-sharing mode. Therefore, a frequency is switched when each time slice is assigned to the corresponding user. Such high-frequency switching of the hardware operating frequency also affects the performance. Therefore, adjust_ve_freq_by_case obtains the frequency corresponding to each user in the "best frequency index table", and then takes the highest value of all user frequencies as the target frequency for the hardware module to meet the usage needs of each user.

[0101] In summary, it should be emphasized that the embodiments of the present application can: create an optimal frequency index table through performance testing, and the hardware codec module implements a dynamic frequency modulation strategy based on the optimal frequency index table and the information of the actually processed video, and supports multi-threaded and multi-process scenarios; and can implement frequency modulation strategies in complex scenarios: if there are three decoding channels, namely H264, H265, and VP9 formats, since the performance of these three decoders is different, it is necessary to unify the task load or decoder performance; wherein, unified processing refers to: determining the optimal user frequency of the three-channel decoding processing respectively, and determining the maximum value from the three user frequencies as the target frequency; making the hardware codec module work at the target frequency and perform three-channel decoding processing at the same time to meet the needs of each decoding processing. Another strategy is to dynamically adjust the frequency of the hardware codec module in real time: detect the codec performance of all channels in real time, and if the codec performance is insufficient, immediately increase the working frequency of the hardware codec module.

[0102] like Figure 5 As shown, the present invention also provides an electronic device, including:

[0103] The processor 501 may be implemented by a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0104] The memory 502 may be implemented in the form of a read-only memory, a static storage device, a dynamic storage device, or a random access memory. The memory 502 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the video encoding and decoding dynamic frequency modulation method of the embodiment of this application;

[0105] Input / output interface 503, used to implement information input and output;

[0106] The communication interface 504 is used to realize the communication interaction between the present apparatus and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0107] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0108] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0109] It is understandable that electronic devices that can be applied to the dynamic frequency modulation method of the embodiment of the present application include but are not limited to: mid-to-high-end car multimedia products on the Android / Linux platform, network set-top box products, building security monitoring central controller products, etc.

[0110] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video encoding and decoding dynamic frequency modulation method as described in the above embodiment.

[0111] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0112] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0113] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present application. These equivalent deformations or substitutions are all included in the scope defined by the present application.

Claims

1. A video encoding and decoding dynamic frequency modulation method, characterized in that: Applied to a video codec dynamic frequency modulation architecture, the video codec dynamic frequency modulation architecture comprises: an application layer, a codec layer, a driver layer, and a hardware layer which are sequentially connected in communication; wherein the application layer comprises an application end; the codec layer comprises: at least one decoder and at least one encoder; the driver layer comprises a plurality of user nodes, an information update module, and a frequency selection control module; the hardware layer comprises a video codec hardware module; The method comprises: Through the hardware layer, a performance test process is performed according to a preset frequency modulation gear, and an optimal frequency index table is created; When the dynamic frequency modulation function is enabled, the codec layer receives at least one video to be processed sent by the application end, obtains video information of the video to be processed and determines a video processing method; The driving layer records or updates the received video information into the node information of the corresponding user node through the information updating module; The driver layer obtains node information of each user node through the frequency selection control module; performs matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determines the maximum frequency value among the multiple user frequencies as the target frequency; and sets the operating frequency of the hardware codec module to the target frequency; The hardware encoding and decoding module performs encoding and decoding processing on the video to be processed according to the target frequency and the video processing method.

2. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: The process of performing performance testing according to the preset frequency modulation gear and creating an optimal frequency index table includes: Determine the second video encoding format, the second resolution and the second frame rate of the test source according to the preset frequency modulation gear, and set the hardware codec module to operate at the first test frequency; The hardware codec module decodes the test source according to the initially preset first test frequency to obtain the decoding frame rate per second of the test source by the hardware codec module; and performs frequency debugging according to the comparison result of the decoding frame rate per second and the preset frame rate threshold to obtain the target decoding frequency in the decoding scenario; The hardware codec module encodes the test film source according to the initially preset second test frequency, and obtains the encoding frame rate per second of the test film source by the hardware codec module; and performs frequency debugging processing according to the comparison result of the encoding frame rate per second and the preset frame rate threshold, and obtains the target encoding frequency in the encoding scenario; Determining the optimal hardware frequency required for processing the test film source according to a comparison result of the target decoding frequency and the target encoding frequency; A frequency mapping relationship is formed according to the second video encoding format, the total range of candidate pixels, and the hardware frequency; the total range of candidate pixels is determined by the second resolution and the second frame rate; After processing a plurality of test film sources in different formats, the optimal frequency index table is created according to the obtained plurality of frequency mapping relationships.

3. The video encoding and decoding dynamic frequency modulation method according to claim 2, characterized in that: The frequency debugging process is performed according to the comparison result of the decoding frame rate per second and the preset frame rate threshold to obtain the target decoding frequency in the decoding scenario, including: When the decoding frame rate per second is equal to a preset frame rate threshold, determining the first test frequency as a target decoding frequency; When the decoding frame rate per second is greater than the preset frame rate threshold, the first test frequency is reduced to a third test frequency, and the hardware codec module decodes the test film source according to the third test frequency. When the decoding frame rate per second is equal to the preset frame rate threshold, the third test frequency is determined as the target decoding frequency.

4. The video encoding and decoding dynamic frequency modulation method according to claim 2, characterized in that: The step of determining the optimal hardware frequency required for processing the test source according to the comparison result between the target decoding frequency and the target encoding frequency includes: When the target decoding frequency is greater than or equal to the target encoding frequency, determining the target decoding frequency as the optimal hardware frequency required for processing the test slice source; When the target encoding frequency is greater than the target decoding frequency, the target encoding frequency is determined as the optimal hardware frequency required for processing the test slice source.

5. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: The optimal frequency index table includes a mapping relationship between multiple candidate encoding formats, candidate pixel total ranges, and hardware frequencies; The matching process is performed according to the information of each node and the optimal frequency index table to determine the user frequency of each user node, including: Acquire a first video encoding format, a first frame rate, and a first resolution of the video information from one of the node information; Calculate the total number of pixels per second according to the first frame rate and the first resolution; Determining a target encoding format from the candidate encoding formats according to the first video encoding format; Determine the target pixel total amount range into which the total amount of pixels per second falls from the four candidate pixel total amount ranges corresponding to the target coding format; Determining a hardware frequency corresponding to the target total pixel range as a user frequency; The matching process is repeated until the user frequency of each user node is determined.

6. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: Before matching the information of each node with the optimal frequency index table, the method further includes: Determine whether the number of the node information obtained by scanning is equal to the number of the user nodes; If yes, the matching process is performed; if no, the node information is continuously acquired.

7. The video encoding and decoding dynamic frequency modulation method according to claim 1, characterized in that: The codec layer also includes a performance monitoring module; the method also includes: Monitoring the encoding and decoding processing of the hardware encoding and decoding module through the performance monitoring module of the encoding and decoding layer; When an abnormal situation of insufficient coding and decoding performance is detected, a working frequency adjustment instruction is sent to the driving layer.

8. A video codec dynamic frequency modulation architecture, characterized in that: include: The application layer, the codec layer, the driver layer, and the hardware layer are sequentially connected in communication; wherein the application layer includes an application end; the codec layer includes: at least one decoder and at least one encoder; the driver layer includes a plurality of user nodes, an information update module, and a frequency selection control module; the hardware layer includes a video codec hardware module; The hardware layer is used to: perform performance test processing according to the preset frequency modulation gear and create an optimal frequency index table; The codec layer is used to: receive at least one video to be processed sent by the application end, and when the dynamic frequency modulation function is enabled, obtain video information of the video to be processed and determine a video processing method; The driving layer is used to: record or update the received video information into the node information of the corresponding user node through the information update module; The driver layer is also used to: obtain node information of each user node through the frequency selection control module; perform matching processing according to each node information and the optimal frequency index table to determine the user frequency of each user node; determine the maximum frequency value among the multiple user frequencies as the target frequency; set the working frequency of the hardware codec module to the target frequency; The hardware encoding and decoding module is further used to: perform encoding and decoding processing on the video to be processed according to the target frequency and the video processing mode.

9. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the video codec dynamic frequency modulation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video encoding and decoding dynamic frequency modulation method according to any one of claims 1 to 7.

Images