Video encoder device based on V4L2 and control method thereof

Abstract

The invention discloses a video encoder device based on V4L2. The coding state maintenance module, the code rate control module, the reference frame management module, the register content generation and analysis module, the video frame acquisition module and the video stream packaging module are functional modules realized by software and are located in a Linux user space. And the V4L2 video equipment driver framework is a function module realized by software and is positioned in a Linux kernel space. The hardware video encoder module is a hardware module. According to the method, the work of most of control software of the stateless hardware video encoder is realized in the Linux user space, interaction with the V4L2 video equipment driver framework located in the Linux kernel space is carried out only by transmitting the information of the register block, the realization difficulty of the control software is reduced, the software function debugging and upgrading are more convenient, and the user experience is improved. And when problems occur, the stability of the whole Linux system is not influenced generally. And meanwhile, control software of the Linux user space can be realized in a closed source manner, so that intellectual property can be protected.

Description

Technical Field

[0001] The present application relates to a digital video encoding technology, and in particular to a video encoder device based on V4L2. Background Art

[0002] V4L2 (Video For Linux 2) is a driver framework for video device management in the Linux kernel, which provides a unified interface for applications to access underlying video devices. V4L2 allows applications to access and control video devices through a unified interface, enabling developers to write video processing applications that are independent of hardware devices, improving the portability and reusability of the code and accelerating the development process of video processing software.

[0003] From the perspective of hardware implementation, hardware video encoders can be divided into stateful encoders and stateless encoders. Stateful hardware video encoders are generally composed of hardware accelerator modules and microcontroller modules. Stateless hardware video encoders generally do not contain microcontroller modules, but only consist of hardware accelerator modules. Many control functions of stateless hardware video encoders need to be completed by control software running on the CPU.

[0004] On embedded devices based on the Linux kernel, when designing a stateless hardware video encoder based on the V4L2 video device driver framework, the control software (part or all of the work such as maintenance of video stream encoding status information, bit rate control, reference frame management, video stream packaging, etc.) is generally implemented as a Linux kernel module, which brings the following disadvantages.

[0005] First, when an application controls a stateless hardware video encoder through the V4L2 video device driver framework, it needs to predefine a large number of V4L2 control primitives that vary according to different video encoding standards and are closely related to the syntax of the video encoding standards, which is a lot of work.

[0006] Second, since the Linux kernel follows the GPL open source agreement, this requires that the control software (bitrate control, reference frame management, etc.) must also follow the GPL open source agreement. This is very unfavorable for protecting the manufacturer's core algorithms and technologies, and is not conducive to protecting the manufacturer's software intellectual property rights, and may lead to the loss of core algorithms and technologies.

[0007] Third, the upgrade and debugging of the control software of the V4L2-based stateless hardware video encoder are bound to the upgrade and debugging of the Linux kernel, which is not only inconvenient but also easily affects the stability of the entire Linux system once a problem occurs. Summary of the invention

[0008] The technical problem to be solved by this application is to develop a video encoder device based on the V4L2 video device driver framework, including a stateless hardware video encoder, so as to overcome the shortcomings of the existing control software upgrade and debugging inconvenience, while protecting the manufacturer's core algorithms and technologies.

[0009] In order to solve the above technical problems, the present application proposes a video encoder device based on V4L2, including an encoding state maintenance module, a bit rate control module, a reference frame management module, a register content generation and analysis module, a video frame acquisition module, a video stream packaging module, a V4L2 video device driver framework, and a hardware video encoder module. The encoding state maintenance module, the bit rate control module, the reference frame management module, the register content generation and analysis module, the video frame acquisition module, and the video stream packaging module are functional modules implemented by software and are located in the Linux user space. The V4L2 video device driver framework is a functional module implemented by software and is located in the Linux kernel space. The hardware video encoder module is a hardware module. The encoding state maintenance module is used to generate encoding tool configuration information according to the user's encoding configuration and the encoding state of the current video stream, and provide it to the register content generation and analysis module; it is also used to update the video encoding state according to the encoding state feedback information provided by the register content generation and analysis module, which is used for the encoding of subsequent video frames. The rate control module is used to generate rate control information according to the rate control requirements of the user, and provide it to the register content generation and analysis module; it is also used to update the control parameters of the rate control algorithm according to the encoding status feedback information provided by the register content generation and analysis module, which is used for the encoding of subsequent video frames. The reference frame management module is used to generate reference frame configuration information according to the image group parameters set by the user, and provide it to the register content generation and analysis module; it is also used to update the control parameters of the reference frame management algorithm according to the encoding status feedback information provided by the register content generation and analysis module, which is used for the encoding of subsequent video frames. The register content generation and analysis module is used to collect encoding tool configuration information, rate control information, and reference frame configuration information before encoding the video frame, and use this information to generate a control register group corresponding to each video frame to be encoded by the hardware video encoder module, and provide it to the V4L2 video device driver framework; it is also used to parse the content of the state register group of the encoded video data provided by the V4L2 video device driver framework when the video frame encoding is completed and generate encoding status feedback information, and provide it to the encoding status maintenance module, the rate control module and the reference frame management module. The video acquisition module is used to provide the video frame to be encoded to the V4L2 video device driver framework. The video stream packaging module is used to package the coded video data provided by the V4L2 video device driver framework and generate corresponding header information to generate a coded video stream. The V4L2 video device driver framework is used to transmit the video frame to be encoded and the corresponding control register group to the hardware video encoder module, and after the video frame encoding is completed, the coded video data and the corresponding status register group are collected from the hardware video encoder module and provided to the video stream packaging module and the register content generation and analysis module respectively.The hardware video encoder module is used to encode the video frame to be encoded under the control of the control register group of each video frame to be encoded; after the video frame encoding is completed, the encoded video data and the corresponding status register group are generated and provided to the V4L2 video device driver framework.

[0010] Furthermore, the V4L2 video device driver framework is the only interface for functional modules implemented by other software to access the hardware video encoder module.

[0011] Furthermore, the hardware video encoder module is a stateless hardware video encoder.

[0012] Furthermore, the register content generation and parsing module and the V4L2 video device driver framework transmit the control register group information used in encoding each video frame and the status register group information corresponding to the encoded video data obtained when the video frame encoding is completed through a shared command cache; the shared command cache is a cache area on the memory, which can be accessed by both the Linux user space and the Linux kernel space through memory mapping.

[0013] Furthermore, before encoding each video frame, the register content generation and analysis module writes the control register group information used in encoding the video frame into the shared command cache, and the V4L2 video device driver framework reads the control register group information in the shared command cache and transmits it to the hardware video encoder for use in encoding. After encoding of a video frame is completed, the V4L2 video device driver framework collects the status register group information corresponding to the encoded video data obtained when encoding of the video frame is completed from the hardware video encoder, and writes it into the shared command cache, and the register content generation and analysis module reads and analyzes the status register group information in the shared command cache to generate the encoding status feedback information corresponding to the video frame.

[0014] Furthermore, the V4L2 video device driver framework pre-allocates one or more v4l2_buffer structures for software encapsulation of the video frames to be encoded; each v4l2_buffer structure is only used to encapsulate one video frame to be encoded; when the number of video frames to be encoded exceeds the number of v4l2_buffer structures, the v4l2_buffer structures are recycled, that is, the v4l2_buffer structures corresponding to the encoded video frames are recycled and used to encapsulate the next video frame to be encoded and used in its encoding process.

[0015] Furthermore, each v4l2_buffer structure has a one-to-one correspondence with a shared command buffer, and the shared command buffer is used to store the control register group information and status register group information of the video frame to be encoded encapsulated in the corresponding v4l2_buffer structure.

[0016] Furthermore, a shared command buffer is composed of two parts: a shared command buffer header and a shared command buffer load; the content of the shared command buffer header is generated when the shared command buffer is allocated; the content of the shared command buffer load is generated before and after encoding each video frame to be encoded; the shared command buffer load part includes one or more subframe register group information. When encoding a video frame to be encoded, if it is not pre-divided into multiple subframes, the entire video frame is regarded as a subframe, and the shared command buffer load part of the shared command buffer only stores one subframe register group information; if the video frame to be encoded is pre-divided into multiple subframes for encoding, the shared command buffer load part of the shared command buffer stores a corresponding number of multiple subframe register group information, and each subframe corresponds to one subframe register group information.

[0017] Furthermore, in the video frame encoding process, if the number of subframes divided by a certain video frame to be encoded is less than or equal to the maximum capacity of the subframe register group of the shared command cache corresponding to the v4l2_buffer structure that encapsulates the video frame, each subframe will correspond to a subframe register group information in the shared command cache, and the actual capacity of the subframe register group in the shared command cache is equal to the number of subframes divided by the video frame to be encoded. If the number of subframes divided by a certain video frame to be encoded is greater than the maximum capacity of the subframe register group of the shared command cache corresponding to the v4l2_buffer structure that encapsulates the video frame, the old shared command cache is destroyed and a new shared command cache is reallocated, and the new shared command cache satisfies that the maximum capacity of the subframe register group is greater than or equal to the number of subframes divided by the video frame to be encoded, and is used to save the subframe register group information corresponding to each subframe of the video frame to be encoded.

[0018] The present application also proposes a control method for a video encoder device based on V4L2, comprising the following steps. Step S1: The video frame acquisition module provides the video frame to be encoded to the V4L2 video device driver framework. Step S2: Before encoding each video frame, the encoding state maintenance module generates the encoding tool configuration information corresponding to the video frame according to the user's encoding configuration and the encoding state of the current video stream; the rate control module generates the rate control information corresponding to the video frame according to the user's rate control requirements; the reference frame management module generates the reference frame configuration information corresponding to the video frame according to the image group parameters set by the user. Step S3: The register content generation and analysis module collects the encoding tool configuration information, rate control information and reference frame configuration information corresponding to the video frame, and uses this information to generate the control register group used to control the hardware video encoder module when encoding the video frame, and provides it to the V4L2 video device driver framework. Step S4: The V4L2 video device driver framework transmits the video frame and the corresponding control register group to the hardware video encoder module. Step S5: The hardware video encoder module performs video encoding on the video frame under the control of its corresponding control register group; after the encoding of the video frame is completed, the hardware video encoder module generates its corresponding encoded video data and status register group. Step S6: The V4L2 video device driver framework collects the encoded video data and status register group corresponding to the video frame, and provides them to the video stream packaging module and the register content generation and analysis module respectively. Step S7: The video stream packaging module packages the video encoding data and generates corresponding header information to generate an encoded video stream; at the same time, the register content generation and analysis module analyzes the content of the status register group and generates the encoding status feedback information corresponding to the video frame, and provides it to the encoding status maintenance module, the bit rate control module and the reference frame management module. Step S8: The encoding status maintenance module updates the encoding status of the video stream according to the encoding status feedback information of the video frame, the bit rate control module updates the control parameters of the bit rate control algorithm according to the encoding status feedback information of the video frame, and the reference frame management module updates the control parameters of the reference frame management algorithm according to the encoding status feedback information of the video frame, all of which are used for the encoding of subsequent video frames.

[0019] The technical effect achieved by this application is: most of the control software of the stateless hardware video encoder is implemented in the Linux user space, and only relies on the transmission of register group information to interact with the V4L2 video device driver framework located in the Linux kernel space. There is no need to predefine a large number of V4L2 control primitives, which greatly reduces the difficulty of implementing the control software. At the same time, since most of the control software is implemented in the Linux user space, it is not necessary to comply with the GPL open source agreement and can be implemented in a closed source manner, which is conducive to protecting the manufacturer's software intellectual property rights and core technology algorithms. In addition, most of the control software is implemented in the Linux user space, which makes software function debugging and upgrading more convenient, and when problems occur, it generally does not affect the stability of the entire Linux system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural diagram of a video encoder device based on V4L2 proposed in this application.

[0021] Figure 2 It is a schematic diagram of the format of the shared command cache proposed in this application.

[0022] Figure 3 It is a flowchart of a control method of a video encoder device based on V4L2 proposed in this application.

[0023] Explanation of the reference numerals in the figure: encoding state maintenance module 11, bit rate control module 12, reference frame management module 13, register content generation and analysis module 14, video frame acquisition module 15, video stream packaging module 16, V4L2 video device driver framework 17, hardware video encoder module 18. DETAILED DESCRIPTION

[0024] See also Figure 1 The V4L2-based video encoder device proposed in this application is mainly composed of a coding state maintenance module 11, a bit rate control module 12, a reference frame management module 13, a register content generation and analysis module 14, a video frame acquisition module 15, a video stream packaging module 16, a V4L2 video device driver framework 17, and a hardware video encoder module 18. The first seven are software-implemented functional modules, and the last one is a hardware module. Among them, the V4L2 video device driver framework 17 is located in the Linux kernel space and is implemented by the Linux kernel module. The remaining six software-implemented functional modules are all located in the Linux user space, and can be implemented without the Linux kernel module, and do not have to comply with the GPL open source agreement, and can be implemented in a closed source manner. Figure 1 In FIG. 8 , the solid line with an arrow represents the flow of the to-be-encoded video frame and its control information, and the dotted line with an arrow represents the flow of the encoded video data and its feedback information.

[0025] The encoding state maintenance module 11 is used to generate encoding tool configuration information according to the user's encoding configuration and the encoding state of the current video stream, and provide it to the register content generation and analysis module 14. The encoding state maintenance module 11 is also used to update the video encoding state according to the encoding state feedback information provided by the register content generation and analysis module 14, which is used for the encoding process of subsequent video frames.

[0026] The bit rate control module 12 is used to generate bit rate control information according to the bit rate control requirements of the user, and provide it to the register content generation and analysis module 14. The bit rate control module 12 is also used to update the control parameters of the bit rate control algorithm according to the encoding state feedback information provided by the register content generation and analysis module 14, which is used in the subsequent video frame encoding process.

[0027] The reference frame management module 13 is used to generate reference frame configuration information according to the image group parameters set by the user, and provide it to the register content generation and analysis module 14. The reference frame management module 13 is also used to update the control parameters of the reference frame management algorithm according to the encoding state feedback information provided by the register content generation and analysis module 14, which is used in the subsequent video frame encoding process.

[0028] The register content generation and analysis module 14 is used to collect the encoding tool configuration information provided by the encoding state maintenance module 11, the rate control information provided by the rate control module 12, and the reference frame configuration information provided by the reference frame management module 13 before encoding the video frame, and use this information to generate a control register group for controlling the hardware video encoder module 18 to encode the video frame to be encoded, and provide it to the V4L2 video device driver framework 17. The register content generation and analysis module 14 is also used to analyze the content of the state register group of the encoded video data provided by the V4L2 video device driver framework 17 when the video frame encoding is completed and generate encoding state feedback information, and provide it to the encoding state maintenance module 11, the rate control module 12, and the reference frame management module 13 at the same time, so as to be used by the encoding state maintenance module 11 to update the video encoding state, the rate control module 12 to update the control parameters of the rate control algorithm, and the reference frame management module 13 to update the control parameters of the reference frame management algorithm.

[0029] The video acquisition module 15 is used to provide the video frames to be encoded to the V4L2 video device driver framework 17 for use in the video encoding process.

[0030] The video stream packaging module 16 is used to package the encoded video data provided by the V4L2 video device driver framework 17 and generate corresponding header information to generate an encoded video stream.

[0031] The V4L2 video device driver framework 17 is the only interface for the functional modules implemented by other software to access the hardware video encoder module 18. It is used to transmit the video frame to be encoded provided by the video acquisition module 15 and the control register group corresponding to the video frame to be encoded provided by the register content generation and analysis module 14 to the hardware video encoder module 18 for encoding, and after the video frame encoding is completed, the encoded video data and the corresponding status register group are collected from the hardware video encoder module 18 and provided to the video stream packaging module 16 and the register content generation and analysis module 14 respectively.

[0032] The hardware video encoder module 18 is a stateless hardware video encoder, which is used to complete the actual video encoding work. Under the control of the control register group of each video frame to be encoded provided by the V4L2 video device driver framework 17, the hardware video encoder module 18 encodes the video frame to be encoded provided by the V4L2 video device driver framework 17; after the encoding of the video frame is completed, the encoded video data and the corresponding state register group are generated and provided to the V4L2 video device driver framework 17.

[0033] In the present application, the register content generation and analysis module 14 located in the Linux user space and the V4L2 video device driver framework 17 located in the Linux kernel space transmit the control register group information required for encoding each video frame and the status register group information corresponding to the encoded video data obtained when the video frame encoding is completed through the "shared command cache" mechanism. The shared command cache is a cache area on the memory of the embedded device based on the Linux kernel, which is used to store the information of the control register group and the status register group when the video frame is encoded. The shared command cache is accessed by the register content generation and analysis module 14 located in the Linux user space and the V4L2 video device driver framework 17 located in the Linux kernel space through memory mapping (mmap for short). For example, the shared command cache is allocated in the Linux kernel space, can be accessed in the Linux kernel space, and is mapped to the Linux user space through memory mapping for access by the Linux user space.

[0034] Before encoding each video frame, the register content generation and parsing module 14 writes the control register group information required for encoding the video frame into the shared command cache, and then the V4L2 video device driver framework 17 reads the control register group information in the shared command cache and transmits it to the hardware video encoder 18 for use in encoding.

[0035] After a video frame is encoded, the V4L2 video device driver framework 17 collects the status register group information corresponding to the encoded video data obtained when the video frame encoding is completed from the hardware video encoder 18, and writes it to the shared command cache for the register content generation and analysis module 14 to read and parse, thereby generating the encoding status feedback information corresponding to the video frame.

[0036] The V4L2 video device driver framework 17 pre-allocates one or more v4l2_buffer structures to be used for software encapsulation of the video frames to be encoded, so that the V4L2 video device driver framework 17 can manage them. Each v4l2_buffer structure is only used to encapsulate one video frame to be encoded. Each v4l2_buffer structure has a unique serial number, which is used to identify and distinguish different v4l2_buffer structures. Assuming that N v4l2_buffer structures are allocated, the serial number is 0, 1, 2, ..., N-1. When the number of video frames to be encoded exceeds the number of v4l2_buffer structures, these v4l2_buffer structures will be recycled, that is, the v4l2_buffer structures corresponding to the encoded video frames are recycled, used to encapsulate the next video frame to be encoded, and used for its encoding process.

[0037] Each v4l2_buffer structure has a one-to-one correspondence with a shared command buffer, which is used to store the control register group information and status register group information of the video frame to be encoded encapsulated in the corresponding v4l2_buffer structure. The serial number of the shared command buffer is equal to the serial number of the corresponding v4l2_buffer structure. Therefore, the number of v4l2_buffer structures is consistent with the number of shared command buffers.

[0038] See also Figure 2 A shared command buffer consists of a shared command buffer header and a shared command buffer payload. The content of the shared command buffer header is generated when the shared command buffer is allocated; the content of the shared command buffer payload is generated before and after encoding each video frame to be encoded. Figure 2 The "subframe register group" in includes the control register group and status register group corresponding to the encoding of a subframe of the video frame to be encoded. When encoding the video frame to be encoded, if it is not pre-divided into multiple subframes, the entire video frame is regarded as a subframe, and only one subframe register group information needs to be saved in the shared command cache; if the video frame to be encoded is pre-divided into multiple subframes for encoding, multiple subframe register group information will be saved in the shared command cache, and each subframe will correspond to one subframe register group information.

[0039] Figure 2 In the shared command cache, the meanings of the various fields are as follows. The shared command cache serial number is a unique identifier of a shared command cache, which is the same as the serial number of the v4l2_buffer structure corresponding to the shared command cache. The shared command cache size refers to the memory size occupied by the shared command cache, in bytes. The shared command cache offset refers to the offset used by the shared command cache when performing memory mapping. The maximum capacity of the subframe register group refers to the maximum number of subframe register groups that can be saved in the shared command cache, that is, the maximum number of subframes that the shared command cache supports and can divide the video frame to be encoded. The actual capacity of the subframe register group refers to the number of subframe register groups actually saved in the shared command cache, that is, the number of subframes actually divided when the video frame to be encoded is encoded. Coding auxiliary control information means that in addition to the register group information, the shared command cache can also pass some coding auxiliary control information to control the state of the hardware video encoder. For example, auxiliary control information for DVFS (Dynamic Voltage and Frequency Scaling) can be passed. The subframe register group information refers to the information of the control register group and the status register group corresponding to a subframe of the to-be-encoded video frame when encoding. If the to-be-encoded video frame is divided into several subframes for encoding, there will be several subframe register group information in the shared command buffer, and each subframe corresponds to one subframe register group information.

[0040] During the video frame encoding process, if the number of subframes divided by a certain video frame to be encoded is less than or equal to the maximum capacity of the subframe register group of the shared command cache corresponding to the v4l2_buffer structure that encapsulates the video frame, each subframe will correspond to a subframe register group information in the shared command cache, and the actual capacity of the subframe register group in the shared command cache is equal to the number of subframes divided by the video frame to be encoded. If the number of subframes divided by a certain video frame to be encoded is greater than the maximum capacity of the subframe register group of the shared command cache corresponding to the v4l2_buffer structure that encapsulates the video frame, the old shared command cache is destroyed and a new shared command cache is reallocated, and it satisfies that the maximum capacity of the subframe register group is greater than or equal to the number of subframes divided by the video frame to be encoded, and is used to save the subframe register group information corresponding to each subframe of the video frame to be encoded.

[0041] The shared command cache mechanism described above can flexibly support the situation where the video frame to be encoded is divided into one or more sub-frames for encoding, and there is no need to extend or modify the existing V4L2 video device driver framework 17, making software function expansion and upgrading more convenient.

[0042] See also Figure 3The control method of the video encoder device based on V4L2 proposed in this application includes the following steps. Figure 3 The method shown corresponds to Figure 1 The device shown.

[0043] Step S1: the video frame acquisition module 15 provides the video frame to be encoded to the V4L2 video device driver framework 17.

[0044] Step S2: Before encoding each video frame, the encoding status maintenance module 11 generates the encoding tool configuration information corresponding to the video frame according to the user's encoding configuration and the encoding status of the current video stream; the bit rate control module 12 generates the bit rate control information corresponding to the video frame according to the user's bit rate control requirements; the reference frame management module 13 generates the reference frame configuration information corresponding to the video frame according to the image group parameters set by the user.

[0045] Step S3: The register content generation and analysis module 14 collects the encoding tool configuration information, bit rate control information and reference frame configuration information corresponding to the video frame, and uses this information to generate a control register group required for encoding the video frame and used to control the hardware video encoder module 18, and provides it to the V4L2 video device driver framework 17.

[0046] Step S4: the V4L2 video device driver framework 17 transmits the video frame and the corresponding control register set to the hardware video encoder module 18.

[0047] Step S5: The hardware video encoder module 18 performs video encoding on the video frame under the control of its corresponding control register group. After the encoding of the video frame is completed, the hardware video encoder module 18 generates its corresponding encoded video data and status register group.

[0048] Step S6: the V4L2 video device driver framework 17 collects the encoded video data and the status register group corresponding to the video frame, and provides them to the video stream packaging module 16 and the register content generation and analysis module 14 respectively.

[0049] Step S7: The video stream packaging module 16 packages the video encoding data and generates corresponding header information to generate an encoded video stream. At the same time, the register content generation and analysis module 14 analyzes the content of the status register group and generates encoding status feedback information corresponding to the video frame, and provides it to the encoding status maintenance module 11, the bit rate control module 12 and the reference frame management module 13.

[0050] Step S8: The encoding status maintenance module 11 updates the encoding status of the video stream according to the encoding status feedback information of the video frame, the bit rate control module 12 updates the control parameters of the bit rate control algorithm according to the encoding status feedback information of the video frame, and the reference frame management module 13 updates the control parameters of the reference frame management algorithm according to the encoding status feedback information of the video frame, all of which are used for the subsequent encoding process of the video frame.

[0051] The steps S1 to S4 are performed before encoding each video frame, and the steps S6 to S8 are performed after encoding each video frame.

[0052] Compared with the prior art, the present application has the following beneficial technical effects.

[0053] First, in this application, the encoding state maintenance module 11, the bit rate control module 12 and the reference frame management module 13 generate a control register group for controlling the encoding process of the video frame through the register content generation and analysis module 14, and receive the status register group generated when the video frame encoding is completed for updating the video stream encoding state and algorithm control parameters. When these four modules 11, 12, 13, 14 located in the Linux user space interact with the V4L2 video device driver framework 17 located in the Linux kernel space, they only need to transfer the control register group information and the status register group information, without pre-defining a large number of V4L2 control primitives that vary according to different encoding standards, which greatly reduces the difficulty of implementing the control software.

[0054] Second, in this application, only the V4L2 video device driver framework 17 is located in the Linux kernel space. The encoding state maintenance module 11, the bit rate control module 12, the reference frame management module 13, the register content generation and analysis module 14, the video frame acquisition module 15, and the video stream packaging module 16 are all located in the Linux user space, and do not need to comply with the GPL open source agreement, and can be implemented in a closed source manner, which is conducive to protecting the manufacturer's software intellectual property rights and core technology algorithms.

[0055] Third, in the present application, the encoding status maintenance module 11, the bit rate control module 12, the reference frame management module 13, the register content generation and analysis module 14, the video frame acquisition module 15, and the video stream packaging module 16 are all located in the Linux user space, rather than in the Linux kernel space. Therefore, the functional debugging and upgrading of the control software are more convenient and are no longer bound to the upgrading and debugging of the Linux kernel. When problems occur, they generally do not affect the stability of the entire Linux system.

[0056] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A video encoder device based on V4L2, characterized in that: It includes encoding status maintenance module, bit rate control module, reference frame management module, register content generation and parsing module, video frame acquisition module, video stream packaging module, V4L2 video device driver framework, and hardware video encoder module; The encoding state maintenance module, bit rate control module, reference frame management module, register content generation and analysis module, video frame acquisition module, and video stream packaging module are functional modules implemented by software and are located in the Linux user space; The V4L2 video device driver framework is a functional module implemented by software and is located in the Linux kernel space; The hardware video encoder module is a hardware module; The encoding state maintenance module is used to generate encoding tool configuration information according to the user's encoding configuration and the encoding state of the current video stream, and provide it to the register content generation and analysis module; it is also used to update the video encoding state according to the encoding state feedback information provided by the register content generation and analysis module, for encoding of subsequent video frames; The bit rate control module is used to generate bit rate control information according to the bit rate control requirements of the user, and provide it to the register content generation and analysis module; it is also used to update the control parameters of the bit rate control algorithm according to the encoding state feedback information provided by the register content generation and analysis module, for encoding of subsequent video frames; The reference frame management module is used to generate reference frame configuration information according to the image group parameters set by the user, and provide it to the register content generation and analysis module; It is also used to update the control parameters of the reference frame management algorithm based on the encoding status feedback information provided by the register content generation and parsing module for encoding subsequent video frames; The register content generation and analysis module is used to collect encoding tool configuration information, bit rate control information, and reference frame configuration information before video frame encoding, and use this information to generate a control register group of the hardware video encoder module corresponding to each video frame to be encoded, and provide it to the V4L2 video device driver framework; it is also used to parse the content of the state register group of the encoded video data provided by the V4L2 video device driver framework when the video frame encoding is completed and generate encoding state feedback information, and provide it to the encoding state maintenance module, the bit rate control module and the reference frame management module at the same time; The video acquisition module is used to provide the video frame to be encoded to the V4L2 video device driver framework; The video stream packaging module is used to package the coded video data provided by the V4L2 video device driver framework and generate corresponding header information to generate a coded video stream; The V4L2 video device driver framework is used to transmit the video frame to be encoded and the corresponding control register group to the hardware video encoder module, and after the video frame encoding is completed, the encoded video data and the corresponding status register group are collected from the hardware video encoder module and provided to the video stream packaging module and the register content generation and analysis module respectively; The hardware video encoder module is used to encode the video frame to be encoded under the control of the control register group of each video frame to be encoded; after the video frame encoding is completed, the encoded video data and the corresponding status register group are generated and provided to the V4L2 video device driver framework.

2. The V4L2-based video encoder device according to claim 1, characterized in that: The V4L2 video device driver framework is the only interface for other software-implemented function modules to access the hardware video encoder module.

3. The V4L2-based video encoder device according to claim 1, characterized in that: The hardware video encoder module belongs to a stateless hardware video encoder.

4. The V4L2-based video encoder device according to claim 1, characterized in that: The register content generation and parsing module and the V4L2 video device driver framework transmit the control register group information used in encoding each video frame and the status register group information corresponding to the encoded video data obtained when the video frame encoding is completed through a shared command cache; the shared command cache is a cache area on the memory, which can be accessed by both the Linux user space and the Linux kernel space through memory mapping.

5. The V4L2-based video encoder device according to claim 4, characterized in that: Before encoding each video frame, the register content generation and parsing module writes the control register group information used in encoding the video frame into the shared command buffer. The V4L2 video device driver framework reads the control register group information in the shared command buffer and transmits it to the hardware video encoder for use in encoding. After a video frame is encoded, the V4L2 video device driver framework collects the status register group information corresponding to the encoded video data obtained when the video frame encoding is completed from the hardware video encoder, and writes it to the shared command cache. The register content generation and parsing module reads and parses the status register group information in the shared command cache to generate the encoding status feedback information corresponding to the video frame.

6. The V4L2-based video encoder device according to claim 4, characterized in that: The V4L2 video device driver framework pre-allocates one or more v4l2_buffer structures for software encapsulation of video frames to be encoded; each v4l2_buffer structure is only used to encapsulate one video frame to be encoded; when the number of video frames to be encoded exceeds the number of v4l2_buffer structures, the v4l2_buffer structure is recycled, that is, the v4l2_buffer structure corresponding to the encoded video frame is recycled and used to encapsulate the next video frame to be encoded and used in its encoding process.

7. The V4L2-based video encoder device according to claim 6, characterized in that: Each v4l2_buffer structure has a one-to-one correspondence with a shared command buffer, which is used to store the control register group information and status register group information of the video frame to be encoded encapsulated in the corresponding v4l2_buffer structure.

8. The V4L2-based video encoder device according to claim 7, characterized in that: A shared command buffer consists of a shared command buffer header and a shared command buffer load. The content of the shared command buffer header is generated when the shared command buffer is allocated. The content of the shared command buffer load is generated before and after encoding each video frame to be encoded. The shared command buffer load part includes one or more sub-frame register group information. When encoding a video frame to be encoded, if it is not pre-divided into multiple sub-frames, the entire video frame is regarded as a sub-frame, and the shared command cache load part of the shared command cache only stores one copy of the sub-frame register group information; if the video frame to be encoded is pre-divided into multiple sub-frames for encoding, the shared command cache load part of the shared command cache stores a corresponding number of copies of the sub-frame register group information, and each sub-frame corresponds to one copy of the sub-frame register group information.

9. The V4L2-based video encoder device according to claim 8, characterized in that During the video frame encoding process, if the number of subframes divided into a certain video frame to be encoded is less than or equal to the maximum capacity of the subframe register group of the shared command cache corresponding to the v4l2_buffer structure encapsulating the video frame, each subframe will correspond to a subframe register group information in the shared command cache, and the actual capacity of the subframe register group in the shared command cache is equal to the number of subframes divided into the video frame to be encoded; If the number of subframes divided by a certain video frame to be encoded is greater than the maximum capacity of the subframe register group of the shared command cache corresponding to the v4l2_buffer structure that encapsulates the video frame, the old shared command cache is destroyed and a new shared command cache is reallocated. The new shared command cache satisfies that the maximum capacity of the subframe register group is greater than or equal to the number of subframes divided by the video frame to be encoded, and is used to save the subframe register group information corresponding to each subframe of the video frame to be encoded.

10. A control method for a video encoder device based on V4L2, characterized in that: The method comprises the following steps: Step S1: the video frame acquisition module provides the video frame to be encoded to the V4L2 video device driver framework; Step S2: before encoding each video frame, the encoding state maintenance module generates encoding tool configuration information corresponding to the video frame according to the user's encoding configuration and the encoding state of the current video stream; the rate control module generates rate control information corresponding to the video frame according to the user's rate control requirement; the reference frame management module generates reference frame configuration information corresponding to the video frame according to the image group parameters set by the user; Step S3: the register content generation and analysis module collects the encoding tool configuration information, bit rate control information and reference frame configuration information corresponding to the video frame, and uses this information to generate a control register group for controlling the hardware video encoder module required for encoding the video frame, and provides it to the V4L2 video device driver framework; Step S4: the V4L2 video device driver framework transmits the video frame and the corresponding control register group to the hardware video encoder module; Step S5: the hardware video encoder module performs video encoding on the video frame under the control of its corresponding control register group; after the encoding of the video frame is completed, the hardware video encoder module generates its corresponding encoded video data and status register group; Step S6: the V4L2 video device driver framework collects the encoded video data and the status register group corresponding to the video frame, and provides them to the video stream packaging module and the register content generation and analysis module respectively; Step S7: the video stream packaging module packages the video coding data and generates corresponding header information to generate a coded video stream; at the same time, the register content generation and analysis module analyzes the content of the status register group and generates coding status feedback information corresponding to the video frame, and provides it to the coding status maintenance module, the bit rate control module and the reference frame management module; Step S8: The encoding status maintenance module updates the encoding status of the video stream according to the encoding status feedback information of the video frame, the bit rate control module updates the control parameters of the bit rate control algorithm according to the encoding status feedback information of the video frame, and the reference frame management module updates the control parameters of the reference frame management algorithm according to the encoding status feedback information of the video frame, all of which are used for the encoding of subsequent video frames.

Images