Heterogeneous chip platform compatible audio and video coding and decoding system and method

Abstract

The invention discloses a heterogeneous chip platform compatible audio and video coding and decoding system and method, relates to the field of audio and video coding and decoding, and aims to reduce or avoid the problem that upper applications cannot be used due to change of a chip platform. The system comprises an interface layer and an implementation layer, the interface layer comprises a common interface, an encoder interface and a decoder interface; the public interface is used for pre-defining an image format, a code stream format and a storage format; the encoder interface is used for encoding the audio and video data into code stream data based on the public interface; the decoder interface is used for decoding the code stream data into audio and video data based on the public interface; the realization layer is used for realizing functions of a common interface, an encoder interface and a decoder interface based on different chip platforms; and the chip platform comprises an Intel SoC (System on Chip) x86 platform, an Nvidia Jetson arm platform and an Nvidia GPGPU (General Purpose Graphics Processing Unit) x86 platform.

Description

Technical Field

[0001] The present invention relates to the field of audio and video coding and decoding, and in particular to an audio and video coding and decoding system and method compatible with heterogeneous chip platforms. Background Art

[0002] When chip manufacturers release new chips, they usually provide users (encoders or decoders) with matching software development kits. The development kits provided by the same manufacturer on different platforms may also be different. For example, Nvidia provides a CUDA development kit with the same interface on the server x86 platform and the edge device arm platform, but the audio and video encoding / decoding development kit is unique to each platform.

[0003] Therefore, when the same manufacturer provides a development kit for a new chip on the same platform, it may abandon support for some old chips. This will cause unexpected errors when programs written for old chips are migrated to new chips. For example: Intel released the "MediaSDK" series of development kits on the x86 platform to make it easier for developers to use chip encoding / decoding capabilities. When Alder Lake was released in 2021, a new "Intel® oneAPI Video Processing Library" development kit (oneVPL for short) was released in conjunction with it. Subsequent chips will only support oneVPL. This will result in the upper-level applications taking a lot of time to re-adapt when programs running on old chips are ported to new chips due to the huge differences in the development kits. This type of adaptation involving basic functions is full of risks and uncertainties. Summary of the invention

[0004] In response to at least one problem existing in the prior art, the present application provides an audio and video codec system and method compatible with heterogeneous chip platforms, which can reduce or avoid the problem of upper-layer applications becoming unusable due to changes in the chip platform.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions.

[0006] In a first aspect, the present invention provides an audio and video encoding and decoding system compatible with heterogeneous chip platforms, including an interface layer and an implementation layer; The interface layer includes a common interface, an encoder interface, and a decoder interface; Public interface for pre-defining image format, code stream format and storage format; The encoder interface is used to encode the audio and video data into bit stream data based on the public interface; A decoder interface, used to decode the bitstream data into audio and video data based on a public interface; The implementation layer is used to implement the functions of the public interface, encoder interface and decoder interface based on different chip platforms; Chip platforms include Intel SoC x86 platform, Nvidia Jetson arm platform and Nvidia GPGPU x86 platform.

[0007] Furthermore, the application layer also includes: The application layer is used to call the encoder interface or decoder interface to perform encoding or decoding on different chip platforms.

[0008] Furthermore, the public interface is also used to pre-set a global shared device identifier pointing to a target chip in the chip platform; The encoder interface is also used to operate the target chip to encode the audio and video data into bit stream data based on the public interface and the global shared device identifier; The decoder interface is also used to operate the target chip to decode the code stream data into audio and video data based on the public interface and the global shared device identifier.

[0009] Furthermore, the storage format includes a host memory HostMemory and a device memory DeviceMemory.

[0010] In a second aspect, the present invention provides an audio and video encoding and decoding method compatible with heterogeneous chip platforms, based on the above system implementation, including an encoder initialization process and an encoder operation process: The encoder initialization process is as follows: Call the public interface through the encoder interface to create hardware device instances and storage allocators; The hardware device instance is the target chip pointed to by the global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; Create a hardware session instance and encoder instance through the encoder interface; A hardware session instance includes at least one encoding session; The encoder operation process is as follows: The encoder instance applies for cache through the storage allocator; The encoder instance writes audio and video data into the cache and sends encoding processing instructions to the hardware device instance; After receiving the encoding processing instruction, the hardware device instance processes the audio and video data in the buffer and outputs the bit stream data.

[0011] Furthermore, it also includes: During the operation of the encoder, detect whether the device instance has any abnormality; if the abnormality is as expected, restore the encoder to the encoder initialization state and continue to run; Device instances include hardware instances and caches; Expectations include the hardware instance becoming unresponsive and / or running out of buffer space.

[0012] In a third aspect, the present invention provides an audio and video encoding and decoding method compatible with heterogeneous chip platforms, based on the above system implementation, including a decoder initialization process and a decoder operation process: The decoder initialization process is as follows: Call the public interface through the decoder interface to create a hardware device instance and storage allocator; The hardware device instance is the target chip pointed to by the global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; Create a hardware session instance and a decoder instance through the decoder interface; The hardware session instance includes at least one decoding session; The decoder runs as follows: The decoder instance applies for cache through the storage allocator; The decoder instance writes the bitstream data into the cache and sends a decoding processing instruction to the hardware device instance; After receiving the decoding processing instruction, the hardware device instance processes the code stream data in the buffer and outputs the audio and video data.

[0013] Furthermore, it also includes: During the operation of the decoder, detect whether the device instance has any abnormality; if the abnormality is as expected, restore the decoder to the state of encoder initialization and continue to operate; Device instances include hardware instances and caches; Expectations include the hardware instance becoming unresponsive and / or running out of buffer space.

[0014] In a fourth aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned audio and video encoding and decoding methods compatible with heterogeneous chip platforms are implemented.

[0015] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned audio and video encoding and decoding methods compatible with heterogeneous chip platforms.

[0016] In a sixth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the audio and video encoding and decoding method compatible with heterogeneous chip platforms.

[0017] The beneficial effects of the present invention are: It can be seen from the above technical solution that the present application provides an audio and video codec system and method compatible with heterogeneous chip platforms, and designs a set of API interfaces that are compatible with the codec capabilities of heterogeneous chips from various manufacturers. This set of API interfaces shields the differences in system development kits for different platforms. Users do not need to access specific platform-specific API calls and can seamlessly migrate to different platforms. When the chip platform is switched or upgraded, the upper-level application only needs to be recompiled and linked on the relevant platform to achieve overall smooth migration, greatly reducing software coupling, and can reduce or avoid the problem of upper-level applications being unusable due to changes in the chip platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the structure of an audio and video codec system compatible with heterogeneous chip platforms in an embodiment of the present application; Figure 2 This is a schematic diagram of the structure of the interface layer in the audio and video encoding and decoding system compatible with heterogeneous chip platforms in an embodiment of the present application; Figure 3 A schematic diagram of the flow of an audio and video encoding and decoding method compatible with heterogeneous chip platforms in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0021] Taking into account the problems existing in the prior art, the present application provides an audio and video codec system and method compatible with heterogeneous chip platforms. It can be seen from the above technical solution that the present application provides an audio and video codec system and method compatible with heterogeneous chip platforms, and designs a set of API interfaces that are compatible with the codec capabilities of heterogeneous chips from various manufacturers. This set of API interfaces shields the differences in system development kits for different platforms. Users do not need to access specific platform-specific API calls and can seamlessly migrate to different platforms. When the chip platform is switched or upgraded, the upper-level application only needs to be recompiled and linked on the relevant platform to achieve overall smooth migration, greatly reducing the degree of software coupling, and can reduce or avoid the problem of upper-level applications being unusable due to changes in the chip platform.

[0022] In order to reduce or avoid the problem that the upper layer application cannot be used due to the change of the chip platform, the present application provides an embodiment of the audio and video codec system compatible with heterogeneous chip platforms, see Figure 1 , an audio and video codec system compatible with heterogeneous chip platforms, including interface layer 1 and implementation layer 2; Interface layer 1 includes a common interface, an encoder interface and a decoder interface; Public interface for pre-defining image format, code stream format and storage format; The encoder interface is used to encode the audio and video data into bit stream data based on the public interface; A decoder interface, used to decode the bitstream data into audio and video data based on a public interface; Implementation layer 2 is used to implement the functions of the common interface, encoder interface and decoder interface based on different chip platforms; Chip platforms include Intel SoC x86 platform, Nvidia Jetson arm platform and Nvidia GPGPU x86 platform.

[0023] Optionally, in this embodiment, a set of API interfaces is designed to be compatible with the encoding / decoding capabilities of heterogeneous chips from various manufacturers. This set of API interfaces shields the differences in system development kits for different platforms. Based on the interface of the present invention, users (encoders or decoders) do not need to contact specific platform-specific API calls and can seamlessly migrate to different platforms. Currently, Intel SoC (x86 platform), Nvidia Jetson (arm platform), and Nvidia GPGPU (x86 platform) are supported.

[0024] like Figure 1 As shown, an intermediate layer is added between the implementation layer (bottom-level system development package) and the application layer (upper-level application).

[0025] The middle layer describes the definition of the encoding / decoding related interfaces. The upper layer applications use the capabilities provided by the chip according to the interface definitions. The bottom layer implements each interface function according to the system development kit. The main purpose of the interface layer is to provide a set of efficient and easy-to-use interfaces for the upper layer applications, and it must ensure that these interfaces are easy to implement in the bottom system development kit. This requires that the interface definition only involves the most basic common functions, and the unique functions of each chip cannot be reflected in the interface layer.

[0026] The interface layer can be divided into several parts: 1. Public interface definition; 2. Encoder interface definition; 3. Decoder interface definition. The underlying implementation layer only needs to select the appropriate system development package and implement related functions according to the interface definition.

[0027] Among them, the public interface is defined as follows: Multiple open standards specify the formats of image data and bitstream data. Each chip manufacturer selectively supports certain types of data formats based on actual conditions. Therefore, the interface layer needs to clearly define which image formats and bitstream formats a certain type of chip supports.

[0028] like Figure 2 As shown in the figure, "ColorFormat" defines common color formats. The input / output image data format must be specified during the encoding / decoding process. "CodecFormat" defines common bitstream formats, of which AVC and HEVC are commonly known as H.264 and H.265 in the field of audio and video encoding. The design also provides single image encoding / decoding, such as "CODEC_JPEG". "MemoryType" defines the data storage method. The enumeration value definitions of the above functions can be used by each chip platform to increase the supported formats as needed.

[0029] Among them, the encoder interface is defined as follows: The process of compressing recognizable image data into a bit stream is called encoding, and the encoder is the specific presentation of this process. The chip encoding unit (hardware) reads the image data from the buffer and outputs the compressed bit stream. In this process, the chip encoding unit plays the role of a consumer, that is, it consumes image data and outputs bit stream data. The encoder needs to prepare the image data required by the chip encoding unit and save the encoded bit stream data in time. Since the chip design uses interrupts to asynchronously notify whether the task is completed, the system SDK layer is reflected through various events to describe whether image data is needed, whether encoding is completed, and whether some abnormal conditions occur during the encoding process.

[0030] like Figure 2As shown in the figure, the "Encoder" interface class is an abstraction of the encoder. This class is designed as a base class and provides two functions: "Initialize()" and "Run()". The "Run()" function is used to implement specific encoding processes on each platform, which will be explained later. The "Initialize()" function is used to describe the encoder initialization process. Since the chip encoding unit must specify the input image format and the encoded bitstream format before use, the input parameter of this function, in addition to the "Context" mentioned above, also accepts an important parameter class "EncoderOptions". The "ColorFormat" and "CodecFormat" member variables of this parameter class control the output and input formats of the encoder. During initialization, the image data storage method is determined by setting "MemoryType", which can avoid changes during the encoding process and greatly reduce the complexity of encoder implementation. The parameters required for the encoder to run have taken effect. In order to solve the two problems of providing image data to the encoder and receiving the bitstream data after encoding, two callback functions "DataProvider()" and "ResultProvider()" can be added to the member variables of the "EncoderOptions" class.

[0031] During the running process, the encoder calls the "DataProvider()" callback function to obtain the required image data from it. The first two parameters passed into the function are the pointer to the writable buffer and the maximum number of bytes in the writable buffer; "OutSize" is the outgoing parameter, which indicates the number of bytes actually written to the buffer by the user (encoder or decoder). If all image data has been written to the buffer and there is no more data to be encoded, set the value of "OutSize" to 0. The return type of the function is int. Returning a non-zero value indicates that the encoder has an abnormal situation and needs to stop encoding. When the encoder consumes a part of the image data, it will call the "ResultProvider()" callback function to output the encoded bitstream data. The user (encoder or decoder) listens to the encoder output event by setting this callback function, copies the bitstream data, and transfers it to the subsequent business logic module for processing. The bitstream data will continue to be output until all image data in the buffer is consumed. "EncoderResult" defines the data structure of the output bitstream, and "Data" and "DataSize" indicate the storage address and data size of the output bitstream data.

[0032] Among them, the decoder interface is defined as follows: The working mode of the decoder is exactly the opposite of that of the encoder: the chip decoding unit obtains the bitstream data from the buffer, decodes the bitstream and outputs the image data. There is a slight difference: the image data output by the decoder needs to specify the width and height of the image. The decoding function design also needs to provide bitstream data to the decoder and notify the user (encoder or decoder) that there is image data output. Figure 2 As shown, the decoder interface class design is given. This definition is similar to the encoder, but the data transmission direction is exactly the opposite.

[0033] The functions of the two interfaces "Initialize()" and "Run()" defined in the "Decoder" class are similar to those of "Encoder", so they are not described in detail. "DecoderOptions" defines the parameters required for the decoder to run. "ColorFormat" and "CodecFormat" specify the output image data format and the input code stream format respectively, and "MemoryType" specifies how to store image data when the decoder outputs. The decoder obtains the code stream data through the "DecoderProvider()" callback function. Similarly, the three parameters of this function represent the buffer pointer, buffer length, and the actual length of the code stream bytes written to the buffer. If there is no code stream data, set the value of "OutSize" to 0. The function returns a non-zero value indicating that the decoder has an abnormal situation and needs to stop decoding. The decoder notifies the user (encoder or decoder) that there is image data output by calling the "ResultProvider()" callback function. The output image data structure is defined by "DecoderResult". Each time a single "DecoderResult" instance is output, it means that a complete image has been decoded. The member fields "Data" and "DataSize" represent the storage address and number of bytes of the image respectively, and the member fields "Width" and "Height" represent the width and height of the image. It should be noted that the storage method of the image data is determined by "DecoderOptions:MemoryType".

[0034] The present application provides an embodiment of an audio and video codec system compatible with heterogeneous chip platforms, which also includes an application layer 3: Application layer 3 is used to call the encoder interface or decoder interface to perform encoding or decoding on different chip platforms.

[0035] Optionally, in this embodiment, the middle layer describes the encoding / decoding related interface definition, and the application layer (upper layer application follows) the interface definition to use the capabilities provided by the chip. The upper layer application only focuses on what functions the interface provides and does not need to know which type of system development kit provides the function.

[0036] In an embodiment of an audio and video codec system compatible with heterogeneous chip platforms provided by the present application, the public interface is also used to pre-set a global shared device identifier pointing to a target chip in the chip platform; The encoder interface is also used to operate the target chip to encode the audio and video data into bit stream data based on the public interface and the global shared device identifier; The decoder interface is also used to operate the target chip to decode the code stream data into audio and video data based on the public interface and the global shared device identifier.

[0037] Optionally, in this embodiment, the public interface definition further includes: defining a global shared device identifier, ie, a "global shared" instance.

[0038] like Figure 2 As shown in the figure, "Context" is an abstraction of the "global shared" instance when each chip platform is running. Some chip platforms require a process-level "handle" when encoding / decoding, which is shared by multiple encoding / decoding instances in the same process.

[0039] For example, when using the Nvidia GPGPU device on the x86 platform, you need to create a global "CUdevice" instance first, and the subsequent encoding / decoding functions access the functions provided by the target chip unit through this instance, but the Intel Soc on the x86 platform does not need such a global instance. Therefore, in order to be compatible with the functions of different chip platforms, the interface layer still needs to provide a conceptually unique "Context" interface class, and the specific implementation of each platform determines whether this interface class works.

[0040] In an embodiment of an audio and video codec system compatible with heterogeneous chip platforms provided by the present application, the storage format includes a host memory HostMemory and a device memory DeviceMemory.

[0041] Optionally, in this embodiment, two types of data storage methods are summarized for the heterogeneous chip platform: Host Memory and Device Memory.

[0042] HostMemory has low cost, large capacity, and relatively slow data reading and writing; DeviceMemory has high cost, small capacity, and fast data reading and writing speed. For SoCs without DRAM such as Intel Soc and Nvidia Jetson, HostMemory interacts with the chip. For Nvidia GPGPU with GPU DRAM, optional HostMemory or DeviceMemory interacts with the chip. Users can flexibly choose the appropriate data reading and writing method according to the usage scenario.

[0043] For technical effects, the present application provides an embodiment of an audio and video encoding and decoding method for heterogeneous chip platforms compatible with all or part of the content of the audio and video encoding and decoding system for heterogeneous chip platforms, including an encoder initialization process and an encoder operation process: S1. The encoder initialization process is as follows: S101, calling the public interface through the encoder interface to create a hardware device instance and a storage allocator; The hardware device instance is the target chip pointed to by the global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; S102, creating a hardware session instance and an encoder instance through an encoder interface; A hardware session instance includes at least one encoding session; S2, the encoder operation process is as follows: S201, the encoder instance applies for cache through the storage allocator; S202, the encoder instance writes the audio and video data into the cache, and sends an encoding processing instruction to the hardware device instance; S203: After receiving the encoding processing instruction, the hardware device instance processes the audio and video data in the buffer and outputs code stream data.

[0044] Optionally, in this embodiment, the multi-chip platform interface is implemented as two modules: initialization and runtime.

[0045] like Figure 2 As shown in the figure, the process starting with "Initialize()" represents the encoder initialization process. It can be roughly divided into several processes: "Create hardware device instance" → "Create storage allocator" → "Create hardware session instance" → "Create encoding unit instance".

[0046] The hardware device instance in "Create a hardware device instance" refers to the globally unique driver handle object in the process. Basically, the chip platform presents the hardware-supported functions through such an instance. The user (coder) chooses to create a suitable instance according to various compilation options and operating systems (Linux, Windows).

[0047] When the chip unit is working, it needs to access a specific function cache area to store data reading and writing during the encoding process. This function is provided by "Create Storage Allocator".

[0048] The creation type of the storage allocator is specified by the external frame_memory_type. It is worth noting that different SoCs and operating systems may not fully support it. The cache allocated by the storage allocator determines whether the encoded data obtained by the external user (encoder) is stored on the HOST side or the DEVICE side.

[0049] The encoding process supports processing video streams with multiple resolutions simultaneously. In this case, "Create Hardware Session Instance" will create a session for each video stream to ensure that multiple video streams do not interfere with each other when encoded simultaneously.

[0050] The last step of initialization is to determine whether the session is a decoder or an encoder. Once the specific function is determined, the entire initialization work is ready.

[0051] The process started by "Run()" fully presents the internal working steps of the encoder, which is generally divided into two stages: "Loop execution: outside" and "Loop execution: inside".

[0052] When the encoder is running, if the outer loop is in a normal state (a successful "reset" is a normal state), it will immediately enter the "loop execution: inner" stage (referred to as the inner loop).

[0053] After the inner loop phase begins, the "hardware unit cache is requested" is first performed. This function is actually completed by the storage allocator. In order to improve performance, the storage allocator will pre-create a cache pool containing multiple caches. The used or unused caches will be saved here. If the cache pool is insufficient, the cache will be created. When the encoder requests a cache, the storage allocator obtains a cache from the cache pool and passes it to the user (encoder). The user (encoder) writes data to the cache through the "DataProvider()" callback function and "notifies" the hardware unit that there is new data to be processed. After receiving the "notification", the hardware unit consumes the cache data and outputs the result. When the hardware unit has a result output, it will "notify" the user (encoder) through the "ResultProvider()" callback function. It should be noted that not every time data is written to the cache, there will be a corresponding result output. This is related to the specific image format and video stream format. When the user (encoder) writes data to the cache, the inner loop will be suspended and in the "synchronous waiting" state. If the inner loop in the "synchronous waiting" state is resumed from suspension but no result is received, the inner loop will repeat the steps of "applying for hardware unit cache", "user (encoder) writing data", and "synchronous waiting" until a result is generated or an exception occurs. If an exception occurs, the inner loop will terminate and transfer process control to the outer loop, which will determine the subsequent steps. The hardware unit "wait" for input data, the user (encoder) "wait" for output results, and the "notification" between the hardware unit and the user (encoder) all occur asynchronously. Figure 3 This is indicated by the dashed arrow in .

[0054] In an embodiment of an audio and video encoding and decoding method compatible with heterogeneous chip platforms of the present application, the method further includes: During the operation of the encoder, detect whether the device instance has any abnormality; if the abnormality is as expected, restore the encoder to the encoder initialization state and continue to run; Device instances include hardware instances and caches; Expectations include the hardware instance becoming unresponsive and / or running out of buffer space.

[0055] Optionally, in this embodiment, the process started by "Run()" completely presents the internal working steps of the encoder, which is generally divided into two stages: "loop execution: outside" and "loop execution: inside".

[0056] The main function of "Loop Execution: External" (referred to as external loop) is to detect whether various device instances have abnormalities. If the abnormal state is within the expected range, the corresponding "reset" operation is performed to restore the encoder to the initial state and continue to run. For example: when the hardware instance loses response, you can try to "wake up" again; when the buffer space is insufficient, the storage allocator can perform "scan / release / merge" operations in advance and try to allocate the buffer multiple times. When resetting the hardware instance fails, or an undefined exception occurs, or the encoding process ends normally, the external loop will terminate and return a flag bit to describe the termination reason.

[0057] The implementation method provided in this embodiment adopts a two-stage loop, in which the outer loop stage processes various hardware instance abnormal conditions, and the inner loop stage processes the encoding / decoding function under normal conditions, thereby giving the program a "self-repairing" capability and improving robustness.

[0058] For technical effects, the present application provides an embodiment of an audio and video encoding and decoding method for heterogeneous chip platforms, which is implemented based on all or part of the contents of the audio and video encoding and decoding system for heterogeneous chip platforms, including a decoder initialization process and a decoder operation process: S1. The decoder initialization process is as follows: S101, calling a public interface through a decoder interface to create a hardware device instance and a storage allocator; The hardware device instance is the target chip pointed to by the global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; S102, creating a hardware session instance and a decoder instance through a decoder interface; The hardware session instance includes at least one decoding session; S2, the decoder operation process is as follows: S201, the decoder instance applies for a cache through a storage allocator; S202, the decoder instance writes the audio and video data into the cache, and sends a decoding processing instruction to the hardware device instance; S203: After receiving the decoding processing instruction, the hardware device instance processes the audio and video data in the buffer and outputs code stream data.

[0059] Optionally, in this embodiment, the multi-chip platform interface is implemented as two modules: initialization and runtime.

[0060] like Figure 2As shown, the process starting with "Initialize()" represents the decoder initialization process. It can be roughly divided into several processes: "Create hardware device instance" → "Create storage allocator" → "Create hardware session instance" → "Create decoding unit instance".

[0061] The hardware device instance in "Create a hardware device instance" refers to the globally unique driver handle object in the process. Basically, the chip platform presents the hardware-supported functions through such an instance. The user (decoder) chooses to create a suitable instance according to various compilation options and operating systems (Linux, Windows).

[0062] When the chip unit is working, it needs to access a specific function cache area to store data reading and writing during the decoding process. This function is provided by "Create Storage Allocator".

[0063] The creation type of the storage allocator is specified by the external frame_memory_type. It is worth noting that different SoCs and operating systems may not fully support it. The cache allocated by the storage allocator determines whether the decoded data obtained by the external user (decoder) is stored on the HOST side or the DEVICE side.

[0064] The decoding process supports processing video streams with multiple resolutions simultaneously. In this case, "Create Hardware Session Instance" will create a session for each video stream to ensure that multiple video streams do not interfere with each other when being decoded simultaneously.

[0065] The last step of initialization is to determine whether the session is a decoder or a decoder. Once the specific function is determined, the entire initialization work is ready.

[0066] The process started by "Run()" fully presents the internal working steps of the decoder, which is generally divided into two stages: "Loop execution: outside" and "Loop execution: inside".

[0067] When the decoder is running, if the outer loop is in a normal state ("reset" success belongs to the normal state), it will immediately enter the "loop execution: inner" stage (referred to as inner loop). After the inner loop stage starts, "apply for hardware unit cache" is performed first. This function is actually completed by the storage allocator. In order to improve performance, the storage allocator will pre-create a cache pool containing multiple caches. The used or unused caches will be saved here. If the cache pool is insufficient, the cache will be created again. When the decoder applies for cache, the storage allocator obtains a cache from the cache pool and passes it to the user (decoder). The user (decoder) writes data to the cache through the "DataProvider()" callback function and "notifies" the hardware unit that there is new data to be processed. After receiving the "notification", the hardware unit consumes the cache data and outputs the result. When the hardware unit has a result output, it will "notify" the user (decoder) through the "ResultProvider()" callback function. It should be noted that not every time data is written to the cache, there will be a corresponding result output. This is related to the specific image format and video stream format. When the user (decoder) writes data to the cache, the inner loop will be suspended and in the "synchronous waiting" state. If the inner loop in the "synchronous waiting" state is resumed from suspension but no result is received, the inner loop will repeat the steps of "applying for hardware unit cache", "user (decoder) writing data", and "synchronous waiting" until a result is generated or an exception occurs. If an exception occurs, the inner loop will terminate and transfer process control to the outer loop, which will determine the subsequent steps. The hardware unit "wait" for input data, the user (decoder) "wait" for output results, and the "notification" between the hardware unit and the user (decoder) all occur asynchronously. Figure 3 This is indicated by the dashed arrow in .

[0068] In an embodiment of an audio and video encoding and decoding method compatible with heterogeneous chip platforms of the present application, the method further includes: During the operation of the decoder, detect whether the device instance has any abnormality; if the abnormality is as expected, restore the decoder to the state of encoder initialization and continue to operate; Device instances include hardware instances and caches; Expectations include the hardware instance becoming unresponsive and / or running out of buffer space.

[0069] Optionally, in this embodiment, the process started by "Run()" completely presents the internal working steps of the decoder, which is generally divided into two stages: "loop execution: outside" and "loop execution: inside".

[0070] The main function of "Loop Execution: External" (referred to as external loop) is to detect whether various device instances have abnormalities. If the abnormal state is within the expected range, the corresponding "reset" operation is performed to restore the decoder to the initial state and continue to run. For example: when the hardware instance loses response, you can try to "wake up" again; when the cache space is insufficient, the storage allocator can perform "scan / release / merge" operations in advance and try to allocate the cache multiple times. When resetting the hardware instance fails, or an undefined exception occurs, or the decoding process ends normally, the external loop will terminate and return a flag bit to describe the termination reason.

[0071] The implementation method provided in this embodiment adopts a two-stage loop, in which the outer loop stage processes various hardware instance abnormal conditions, and the inner loop stage processes the encoding / decoding function under normal conditions, thereby giving the program a "self-repairing" capability and improving robustness.

[0072] From the hardware level, in order to achieve full product traceability, have intelligent account period management capabilities, and be able to make accurate consumption predictions based on historical data, the present application provides an embodiment of an electronic device for implementing all or part of the content of the audio and video encoding and decoding method compatible with heterogeneous chip platforms, and the electronic device specifically includes the following content: Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used for information transmission between the audio and video codec system compatible with heterogeneous chip platforms and the core business system, user terminal and related databases and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the audio and video codec method compatible with heterogeneous chip platforms and the embodiment of the audio and video codec system compatible with heterogeneous chip platforms in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0073] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0074] In practical applications, part of the audio and video encoding and decoding method for heterogeneous chip platform compatibility can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0075] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed system.

[0076] Figure 4 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 4 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Figure 4 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0077] In one embodiment, the audio and video encoding and decoding method function for heterogeneous chip platform compatibility can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: Including encoder initialization process and encoder operation process: S1. The encoder initialization process is as follows: S101, calling the public interface through the encoder interface to create a hardware device instance and a storage allocator; The hardware device instance is the target chip pointed to by the global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; S102, creating a hardware session instance and an encoder instance through an encoder interface; A hardware session instance includes at least one encoding session; S2, the encoder operation process is as follows: S201, the encoder instance applies for cache through the storage allocator; S202, the encoder instance writes the audio and video data into the cache, and sends an encoding processing instruction to the hardware device instance; S203: After receiving the encoding processing instruction, the hardware device instance processes the audio and video data in the buffer and outputs code stream data.

[0078] From the above description, it can be seen that the electronic device provided in the embodiment of the present application is designed with a set of API interfaces that are compatible with the encoding and decoding capabilities of heterogeneous chips from various manufacturers. This set of API interfaces shields the differences in system development kits for different platforms. Users do not need to contact specific platform-specific API calls and can seamlessly migrate to different platforms. When the chip platform is switched or upgraded, the upper-level application only needs to be recompiled and linked on the relevant platform to achieve overall smooth migration, greatly reducing the degree of software coupling, and can reduce or avoid the problem of upper-level applications being unusable due to changes in the chip platform.

[0079] In another embodiment, the audio and video codec system compatible with heterogeneous chip platforms can be configured separately from the central processing unit 9100. For example, the audio and video codec system compatible with heterogeneous chip platforms can be configured as a chip connected to the central processing unit 9100, and the audio and video codec method function compatible with heterogeneous chip platforms can be implemented through the control of the central processing unit.

[0080] like Figure 4 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 4 In addition, the electronic device 9600 may also include Figure 4 For components not shown, reference may be made to the prior art.

[0081] like Figure 4 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor system and / or logic system, which receives input and controls the operation of various components of the electronic device 9600.

[0082] The memory 9140 may be, for example, a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or one or more of other suitable systems. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0083] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or touch input system. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0084] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of system. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0085] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0086] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0087] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0088] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the audio and video encoding and decoding method for heterogeneous chip platforms in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the audio and video encoding and decoding method for heterogeneous chip platforms in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented: Including encoder initialization process and encoder operation process: S1. The encoder initialization process is as follows: S101, calling the public interface through the encoder interface to create a hardware device instance and a storage allocator; The hardware device instance is the target chip pointed to by the global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; S102, creating a hardware session instance and an encoder instance through an encoder interface; A hardware session instance includes at least one encoding session; S2, the encoder operation process is as follows: S201, the encoder instance applies for cache through the storage allocator; S202, the encoder instance writes the audio and video data into the cache, and sends an encoding processing instruction to the hardware device instance; S203: After receiving the encoding processing instruction, the hardware device instance processes the audio and video data in the buffer and outputs code stream data.

[0089] From the above description, it can be seen that the computer program product provided in the embodiment of the present application designs a set of API interfaces that are compatible with the encoding and decoding capabilities of heterogeneous chips from various manufacturers. This set of API interfaces shields the differences in system development kits for different platforms. Users do not need to contact specific platform-specific API calls and can seamlessly migrate to different platforms. When the chip platform is switched or upgraded, the upper-level application only needs to be recompiled and linked on the relevant platform to achieve overall smooth migration, greatly reducing the degree of software coupling, and can reduce or avoid the problem of upper-level applications being unusable due to changes in the chip platform.

[0090] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0092] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An audio and video codec system compatible with heterogeneous chip platforms, characterized in that: It includes interface layer (1) and implementation layer (2); The interface layer (1) includes a public interface, an encoder interface and a decoder interface; The public interface is used to predefine the image format, code stream format and storage format; The encoder interface is used to encode the audio and video data into code stream data based on the common interface; The decoder interface is used to decode the code stream data into audio and video data based on the common interface; The implementation layer (2) is used to implement the functions of the common interface, the encoder interface and the decoder interface based on different chip platforms respectively; The chip platforms include Intel SoC x86 platform, Nvidia Jetson arm platform and Nvidia GPGPU x86 platform.

2. The audio and video codec system compatible with heterogeneous chip platforms according to claim 1, characterized in that: Also includes the application layer (3): The application layer (3) is used to call the encoder interface or the decoder interface to perform encoding or decoding on different chip platforms.

3. The audio and video codec system compatible with heterogeneous chip platforms according to claim 1 or 2, characterized in that: The public interface is also used to pre-set a global shared device identifier pointing to a target chip in the chip platform; The encoder interface is further used to operate the target chip to encode the audio and video data into bit stream data based on the public interface and the global shared device identifier; The decoder interface is also used to operate the target chip to decode the code stream data into audio and video data based on the public interface and the global shared device identifier.

4. The audio and video encoding and decoding system compatible with heterogeneous chip platforms according to claim 3, characterized in that: The storage format includes a host memory HostMemory and a device memory DeviceMemory.

5. An audio and video encoding and decoding method compatible with heterogeneous chip platforms, characterized in that: The system implementation according to claim 4 includes an encoder initialization process and an encoder operation process: The encoder initialization process is as follows: Call the public interface through the encoder interface to create hardware device instances and storage allocators; The hardware device instance is a target chip pointed to by a global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; Create a hardware session instance and encoder instance through the encoder interface; The hardware session instance includes at least one encoding session; The encoder operation process is as follows: The encoder instance applies for cache through the storage allocator; The encoder instance writes the audio and video data into the buffer and sends an encoding processing instruction to the hardware device instance; After receiving the encoding processing instruction, the hardware device instance processes the audio and video data in the buffer and outputs code stream data.

6. The audio and video encoding and decoding method compatible with heterogeneous chip platforms according to claim 5, characterized in that: Also includes: During the operation of the encoder, detect whether the device instance has any abnormality; if the abnormality is as expected, restore the encoder to the encoder initialization state and continue to run; The device instance includes a hardware instance and a cache; The expectations include the hardware instance becoming unresponsive and / or running out of buffer space.

7. An audio and video encoding and decoding method compatible with heterogeneous chip platforms, characterized in that: The system implementation according to claim 4 includes a decoder initialization process and a decoder operation process: The decoder initialization process is as follows: Call the public interface through the decoder interface to create a hardware device instance and storage allocator; The hardware device instance is a target chip pointed to by a global shared device identifier; The storage allocator is used to allocate the cache in HostMemory or the cache in DeviceMemory; Create a hardware session instance and a decoder instance through the decoder interface; The hardware session instance includes at least one decoding session; The decoder operates as follows: The decoder instance applies for cache through the storage allocator; The decoder instance writes the bitstream data into the buffer and sends a decoding processing instruction to the hardware device instance; After receiving the decoding processing instruction, the hardware device instance processes the bit stream data in the buffer and outputs audio and video data.

8. The audio and video encoding and decoding method for heterogeneous chip platform compatibility according to claim 7, characterized in that: Also includes: During the operation of the decoder, detect whether the device instance has any abnormality; if the abnormality is as expected, restore the decoder to the state of encoder initialization and continue to operate; The device instance includes a hardware instance and a cache; The expectations include the hardware instance becoming unresponsive and / or running out of buffer space.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the audio and video encoding and decoding method compatible with heterogeneous chip platforms as described in any one of claims 5 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the audio and video encoding and decoding method compatible with heterogeneous chip platforms as described in any one of claims 5 to 8 are implemented.

Images