Audio and video processing chip and electronic device

Abstract

Embodiments of the present application relate to the technical field of computer, and provide an audio and video processing chip and an electronic device. The audio and video processing chip comprises a plurality of codecs, a plurality of memories and a network on chip. The codec is configured to send a target address of audio and video data to the network on chip when reading and writing of the audio and video data is required. The network on chip is configured to determine whether the target address is located in an address range corresponding to the codec, and access a position corresponding to the target address in a neighboring memory of the codec when the target address is located in the address range corresponding to the codec. Each codec is pre-assigned a continuous address range, and addresses in the address range corresponding to each codec are mapped to the neighboring memory of the codec. The chip realizes the nearby access of the codec to the memory, the access mode significantly reduces the access delay, is beneficial to maintaining the stable performance of the codec, and enables the chip power consumption to be maintained at a low level.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an audio / video processing chip and electronic device. Background Technology

[0002] Currently, artificial intelligence (AI) chips are widely used in various audio and video processing tasks. AI chips include codecs and memory. The encoder can read the raw audio and video data in the memory, encode it, and then write the encoded data back to the memory. The decoder can read the encoded audio and video data in the memory, decode it, and then write the decoded data back to the memory.

[0003] Early AI chips were smaller, with codecs and memory located close to each other on-chip. This resulted in lower latency for codecs accessing memory, ensuring good encoding and decoding performance, and also led to lower chip power consumption. However, in recent years, the increasing demands for computing power and storage in AI chips have resulted in significantly larger chip sizes and a greater number of codecs and memory units on the chips. Consequently, some codecs are now located further apart from the memory, leading to higher latency when the codec needs to access all memory locations. This severely impacts encoding and decoding performance and increases chip power consumption. Summary of the Invention

[0004] The purpose of this application is to provide an audio / video processing chip and electronic device to improve the above-mentioned technical problems.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, embodiments of this application provide an audio / video processing chip, including: multiple codecs, multiple memories, and an on-chip network; the codecs are used to send the target address of the audio / video data to the on-chip network when it is necessary to read or write audio / video data; the on-chip network is used to determine whether the target address is within the address range corresponding to the codec, and when the target address is within the address range corresponding to the codec, access the location corresponding to the target address in the adjacent memory of the codec; wherein, each codec is pre-allocated a continuous address range, and the addresses within the address range corresponding to each codec are mapped to the adjacent memory of that codec.

[0007] Since each codec in the aforementioned audio and video processing chip is pre-allocated a continuous address range, and the addresses within the address range corresponding to each codec are mapped to the codec's proximity memory (referring to the on-chip memory that is closer to the codec), when the target address is within the address range corresponding to a certain codec, the on-chip network will access the location in the codec's proximity memory corresponding to the target address. This achieves the codec's proximity access to the memory, which significantly reduces access latency, helps maintain stable codec performance, and keeps the chip power consumption at a low level.

[0008] It should be noted that although the aforementioned audio and video processing chip can improve the technical problems existing in current AI chips, it may or may not be an AI chip.

[0009] In one implementation of the first aspect, the address range corresponding to each codec represents a group of the memory, each group contains at least one memory, and the address within the group is the addressing of the memory units contained in the group; the memory units in the memory contained in each group are divided into at least one channel, and the address within the channel is the addressing of the memory units contained in the channel; the group sequence number is the number of all groups corresponding to the plurality of codecs, the channel sequence number within the group is the number of the channels contained in the group, and the global channel sequence number is the number of the channels contained in all groups corresponding to the plurality of codecs; the on-chip network is used to perform the following operations: according to Based on the group number and address range of each group, determine the group number of the group containing the target address and the intra-group address corresponding to the target address; calculate the intra-group channel number of the channel containing the target address and the intra-channel address corresponding to the target address based on the intra-group address corresponding to the target address and the number of channels in the group containing the target address; calculate the global channel number of the channel containing the target address based on the group number of the group containing the target address, the number of channels in each group, and the intra-group channel number of the channel containing the target address; access the location corresponding to the target address in the proximity memory of the codec based on the global channel number and the intra-channel address corresponding to the target address.

[0010] Dividing channels helps improve the efficiency of memory access (memory cells in different channels can be accessed in parallel). The above implementation method provides a scheme for the codec to access the memory nearest to the local device when dividing channels.

[0011] In one implementation of the first aspect, the on-chip network is further configured to access a location in a global storage space formed by the plurality of memories corresponding to the target address when the target address is outside the address range corresponding to the codec; wherein, addresses outside the address range corresponding to each codec are mapped to the global storage space.

[0012] In the above implementation, if the target address is outside the address range corresponding to the codec, the target address can be regarded as a global address. The location corresponding to the target address in the global storage space can be accessed through the on-chip network. In this way, in addition to enabling the codec to access the memory nearby, it can also support the codec to access the remote memory when necessary.

[0013] In one implementation of the first aspect, the storage units in the global storage space are divided into at least one channel, the channel address is the addressing of the storage units contained in the channel, and the global channel number is the number of the channels contained in the global storage space; the on-chip network is used to perform the following operations: calculate the global channel number of the channel where the target address is located and the channel address corresponding to the target address based on the target address and the total number of channels in the global storage space; access the location in the global storage space corresponding to the target address based on the global channel number and the channel address corresponding to the target address.

[0014] Dividing channels helps improve the efficiency of memory access (memory cells in different channels can be accessed in parallel). The above implementation method provides a scheme for the codec to achieve global access to memory under the condition of dividing channels.

[0015] In one implementation of the first aspect, the codec is further configured to send the identifier of the codec to the on-chip network when it is necessary to read or write audio and video data; the on-chip network is configured to determine the address range corresponding to the codec based on the identifier of the codec, and determine whether the target address is within the address range corresponding to the codec.

[0016] In the above implementation, when the codec sends the target address to the on-chip network, it also sends its own identifier. The on-chip network can then distinguish different codecs based on this identifier, thereby determining the address range corresponding to the codec and making subsequent judgments.

[0017] In one implementation of the first aspect, the on-chip network includes a sub-network corresponding to each codec, wherein the codec is used to send the target address to the sub-network corresponding to the codec in the on-chip network when it is necessary to read or write audio and video data; the sub-network corresponding to the codec is used to determine whether the target address is within the address range corresponding to the codec.

[0018] In the above implementation, the on-chip network is divided into multiple sub-networks, each sub-network corresponding to a codec, and is only responsible for determining whether the target address provided by the codec is within the address range corresponding to the codec. At this time, the codec does not need to provide any identifier to indicate its identity.

[0019] In one implementation of the first aspect, the audio / video processing chip further includes an address translation unit; the codec is used to send the original target address of the audio / video data to the address translation unit when it is necessary to read or write the audio / video data, the original target address being a virtual address; the address translation unit is used to convert the original target address into the target address and send the target address to the on-chip network, the target address being a physical address.

[0020] In the above implementation, the codec can use virtual addresses internally. When addressing is required, the address translation unit is responsible for converting the virtual address into a physical address, thereby achieving the separation of virtual and physical addresses and facilitating the codec's encoding and decoding.

[0021] In one implementation of the first aspect, each codec and its adjacent memory are allocated as a separate virtualization resource.

[0022] In the above implementation, each codec and its adjacent memory are allocated as an independent virtualization resource. Thus, in the virtualization scenario, different users can exclusively enjoy codec and storage resources, achieving hardware-level isolation of user data. This is beneficial for improving the stability and security of user data and avoiding the uncertainty brought about by global data access.

[0023] Secondly, embodiments of this application provide an electronic device, including: a central processing unit, an external memory, and an audio / video processing chip provided in the first aspect or any implementation thereof; the central processing unit is used to run a driver program for the audio / video processing chip stored in the external memory, wherein the driver program allocates a continuous address range for each codec of the audio / video processing chip during operation.

[0024] The aforementioned electronic device, by incorporating an audio / video processing chip provided by the first aspect or any implementation thereof, can achieve efficient audio / video encoding and decoding and has low power consumption.

[0025] In one implementation of the second aspect, the central processing unit is further configured to respond to an address configuration request and modify the address range corresponding to the codec.

[0026] The above implementation method can dynamically modify the address range corresponding to the codec, thereby flexibly binding the codec and its corresponding memory, realizing flexible access of the codec to the memory (not limited to accessing the nearest memory), and meeting different memory access requirements. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram of the global access memory for the codec;

[0029] Figure 2 A schematic diagram illustrating how the codec accesses memory from the nearest memory location;

[0030] Figure 3 A diagram illustrating the principles of local access and global access in codecs;

[0031] Figure 4 This is a schematic diagram showing the channel distribution in the memory.

[0032] Figure 5 This is a schematic diagram of the addressing process of an on-chip network.

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] Figure 1 This is a schematic diagram of the global access memory for the codec. (Refer to...) Figure 1A certain AI chip that can be used to process audio and video data includes an AI core, two codecs (codec 0 and codec 1), two memories (DDR0 and DDR1, where DDR is an abbreviation for Double Data Rate, meaning double data rate synchronous dynamic random access memory), and an on-chip network (NOC is an abbreviation for Network on Chip).

[0035] In this context, "codec" can refer to an encoder, a decoder, or both. An encoder reads raw audio and video data from memory, encodes it, and then writes the encoded data back to memory. A decoder reads encoded audio and video data from memory, decodes it, and then writes the decoded data back to memory, and so on. In short, both reading and writing audio and video data involve accessing specific addresses in memory. Here, "audio and video data" can refer to pure audio data, pure video data, or a combination of both.

[0036] On-chip networks are used to interconnect codecs and memory. When the codec reads or writes audio and video data to the memory, it can use the on-chip network for addressing. Note that... Figure 1 The diagram shows two NOCs, located near codec 0 and codec 1 respectively. The on-chip network can be considered as a whole, but it is drawn as two parts for ease of understanding. Alternatively, the on-chip network can indeed be divided into two interconnected sub-networks, one near codec 0 and the other near codec 1.

[0037] The AI ​​core is used to execute AI algorithms. For example, the audio and video data decoded by the decoder can be sent to the AI ​​core for analysis and processing, and then written to the memory.

[0038] The term "codec global access to memory" simply means that when accessing memory, the codec does not consider the distance between the codec and the memory (which can refer to the distance in circuitry), treating all memory as a global storage space for access. Specifically... Figure 1 This means that when codec 0 reads and writes audio and video data, it can access either DDR0, which is closer to it, or DDR1, which is farther away. Similarly, when codec 1 reads and writes audio and video data, it can access either DDR1, which is closer to it, or DDR0, which is farther away.

[0039] Taking codec 0 as an example, when the AI ​​chip area is small, although the distance between DDR1 and codec 0 is greater than the distance between DDR0 and codec 0, this distance is not significant in absolute terms. Therefore, the latency of codec 0 accessing memory is low, ensuring encoding and decoding performance, and the chip power consumption is also relatively low. However, when the AI ​​chip area is large, the distance between DDR1 and codec 0 becomes relatively large in absolute terms. Therefore, the latency of codec 0 accessing DDR1 is large, which cannot guarantee its encoding and decoding performance and leads to increased chip power consumption.

[0040] Figure 1 This is just the case with two codecs and two memories. If there are more codecs and memories on the AI ​​chip, the latency will be greater when the codecs perform global access, the encoding and decoding performance will be further reduced, and the chip power consumption will be further increased.

[0041] Figure 2 This is a diagram illustrating how the codec accesses memory from the nearest memory location. Figure 1 similar, Figure 2 The AI ​​chip in the chip also includes an AI core, two codecs, two memories, and an on-chip network. The basic functions of each component and... Figure 1 Similarly, I will not repeat the explanation.

[0042] The term "codec accessing memory nearest" simply means that when accessing memory, the codec considers the distance between itself and the memory, accessing only, or usually only, one or more memory locations closest to it. Figure 1 This means that when codec 0 reads and writes audio and video data, it only accesses DDR0, which is closer to it, and does not access DDR1, which is farther away. Similarly, when codec 1 reads and writes audio and video data, it only accesses DDR1, which is closer to it, and does not access DDR0, which is farther away.

[0043] However, it's important to note that, taking codec 0 as an example, considering certain practical needs, it may be permitted to access remote DDR1 under specific circumstances. These needs include: data synchronization between codecs, communication between codecs and the AI ​​core, communication between the codec and the central processing unit of the electronic device to which the AI ​​chip belongs, and so on. Therefore, in Figure 2 In the diagram, the NOCs near codec 0 and near codec 1 are connected by a dashed line, indicating that remote memory access is allowed under certain special circumstances, but these circumstances are less likely to occur than accessing local memory.

[0044] When performing proximity access, the codec and the memory it accesses are close together, resulting in lower latency, ensuring encoding and decoding performance, and also lower chip power consumption. Furthermore, proximity access is not significantly affected by the size of the AI ​​chip, because even if the AI ​​chip area increases, the distance between the codec and its nearest memory does not necessarily increase accordingly.

[0045] Figure 2 The AI ​​chip mentioned can be considered as one possible structure of the audio and video processing chip provided in the embodiments of this application. However, the audio and video processing chip provided in the embodiments of this application is not necessarily an AI chip. For example, it may include an audio and video processing core for executing traditional audio and video processing algorithms (non-AI algorithms), or it may not include an audio and video processing core at all, but only be used to implement the encoding and decoding of audio and video data. Of course, for the sake of simplicity, the following description of the audio and video processing chip provided in the embodiments of this application will still mainly focus on... Figure 2 For example.

[0046] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0047] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0048] This application provides an audio / video processing chip, which includes at least the following components (all hardware components):

[0049] Codec: A codec can be an encoder and / or decoder used to encode and / or decode audio and / or video data, which can be audio and / or video data. The number of codecs can be multiple (at least two), for example, Figure 2 The AI ​​chip in the chip includes two codecs: codec 0 and codec 1.

[0050] Memory: Memory is used to store data, such as raw audio / video data, encoded audio / video data, and non-audio / video data. There can be multiple memories (at least two), for example, Figure 2The AI ​​chip includes two memories in total, DDR0 and DDR1, but the number of memories is not necessarily the same as the number of codecs. It should be understood that the type of memory is not limited to DDR; it can also be other volatile or non-volatile memories.

[0051] Each memory unit can contain multiple memory cells, and each memory cell can store an unlimited amount of data, such as 1 byte, 4 bytes, or 8 bytes. Different memory units may contain the same or different numbers of memory cells. Each memory cell can correspond to one or more addresses: for example, a 4-byte memory cell may have each byte corresponding to an address, with one byte of data stored at each address; or a 4-byte memory cell may correspond to only one address, storing 4 bytes of data at that address. For simplicity, the following text will primarily focus on the case where one memory cell corresponds to one address; the case where one memory cell corresponds to multiple addresses can be analyzed similarly.

[0052] On-chip network: On-chip network is the network inside the chip. It is used to interconnect codecs and memory. For example, any codec can be interconnected with any memory through the on-chip network. In other words, theoretically, any codec can access any memory through the on-chip network (of course, in practice, it may not be possible to access any memory).

[0053] When the codec reads and writes audio and video data to the memory, it can use the on-chip network for addressing. Addressing means locating the read / write address of the audio and video data to the corresponding location in the memory through the on-chip network. The specific addressing process will be explained later.

[0054] On-chip networks may include circuitry for implementing the aforementioned interconnection and addressing functions, as well as on-chip arithmetic units and other components (e.g., Figure 5 The addressing process of the on-chip network (such as the multiplexer MUX, divider DIV, etc.) can be implemented through the hardware structure of the on-chip network.

[0055] During the encoding and decoding process, the codec needs to read and write the audio and video data involved. At this time, the codec can send the read address or write address of the audio and video data to the on-chip network (CNB), collectively referred to as the target address. Of course, the codec doesn't necessarily only send the target address to the CNB; it can also send other information necessary to complete the data read and write operations. For example, when the codec needs to read audio and video data to be encoded or decoded from memory, it can send a data read command to the CNB, which carries the target address. Similarly, when the codec needs to write encoded or decoded audio and video data to memory, it can send a data write command to the CNB, which carries the target address and the data to be written, and so on. However, for simplicity, other data besides the target address will usually be ignored in the following text.

[0056] The target address can be a physical address, that is, an address in the physical storage space formed by the storage cell in the memory.

[0057] In one implementation, the codec can directly send the target address (physical address) to the codec.

[0058] In another implementation, the audio / video processing chip also includes an address translation unit. The input of the address translation unit is connected to the codec, and its output is connected to the on-chip network. Instead of directly sending the destination address of the audio / video data to the on-chip network, the codec first sends the original destination address of the audio / video data to the address translation unit. The address translation unit then converts the original destination address into a destination address (physical address) before sending it to the on-chip network. The original destination address is a virtual address used internally by the codec, corresponding to an address in the virtual memory space. This method separates the virtual address from the physical address, allowing the codec to simplify the encoding and decoding process by not needing to know the actual storage location of the data.

[0059] Address translation units can be one or more. For example, one address translation unit can be set up for each codec, which is only responsible for translating the original target address provided by the corresponding codec. Alternatively, only one address translation unit can be set up for all codecs, and all the original target addresses provided by the codecs can be translated by this address translation unit, and so on.

[0060] After receiving a target address, the on-chip network (CNN) determines whether the target address is within the address range of the codec that sent it. If the target address is within the address range of the codec, it accesses the location corresponding to the target address in the nearest memory of the codec. Conversely, if the target address is outside the address range of the codec, it accesses the location corresponding to the target address in a global memory space formed by multiple memories. The "access" performed by the CNN can be understood as reading audio / video data from or writing audio / video data to that location. Alternatively, the above can be simply summarized as follows: if the target address is within the address range of the codec, the codec (via the CNN) accesses the nearest memory; if the target address is outside the address range of the codec, the codec (via the CNN) accesses the global memory.

[0061] In this audio / video processing chip, each codec is pre-allocated a continuous address range. Addresses within this range are mapped to the nearest neighbor memory for that codec, while addresses outside this range are mapped to a global memory space formed by multiple memories. This address mapping method supports both proximity-based and global access. The address ranges corresponding to different codecs can be continuous or discontinuous; the following analysis primarily focuses on the continuous case, while the discontinuous case can be analyzed similarly.

[0062] For a given codec, its nearest memory can be one or more memories on the audio / video processing chip that are closest to it, for example, in Figure 2 In this context, the neighboring memory of codec 0 can be DDR0, while the neighboring memory of codec 1 can be DDR1. The neighboring memories of different codecs can be either unique or overlapping. The global memory space is the same for all codecs; for example, in... Figure 2 In the codec, the global storage space corresponding to codec 0 and codec 1 is the storage space formed by DDR0 and DDR1.

[0063] Taking the case where each memory cell in the memory corresponds to one address as an example, a continuous address range can be allocated to each codec based on the number of memory cells contained in its neighboring memory. In this way, the number of addresses contained in the address range corresponding to any encoder is the same as the number of memory cells contained in its neighboring memory. Therefore, any address within the address range corresponding to the codec can be mapped to a memory cell in its neighboring memory. The on-chip network can then locate the memory cell in the neighboring memory corresponding to the target address based on this mapping and access that memory cell (nearest access). Any address outside the address range corresponding to the codec cannot be directly mapped to a memory cell in its neighboring memory, so it can be mapped to a memory cell in the global memory space. The on-chip network can then locate the memory cell in the global memory space corresponding to the target address based on this mapping and access that memory cell (global access).

[0064] Figure 3 This diagram illustrates the principles of local and global access in codecs. (Refer to...) Figure 3 Assuming each gray rounded rectangle represents a memory unit, and ignoring the meanings of channels like Ch0 and Ch1 in the diagram, DDR0 includes 16 memory units, and DDR1 also includes 16 memory units. Therefore, the global memory space formed by DDR0 and DDR1 contains a total of 32 memory units, corresponding to addresses 0-31. Thus, address range 0-15 can be allocated to codec 0 (in...). Figure 3 This is denoted as BAR0, corresponding to a memory cell in the adjacent memory DDR0 of codec 0, and an address range of 16 to 31 is allocated to codec 1 (in... Figure 3 This is denoted as BAR1, corresponding to a memory cell in DDR1, the adjacent memory of codec 1. Assuming there is another memory cell, DDR2, closer to codec 0, containing 32 memory cells, then address range 0 to 47 can be allocated to codec 0, corresponding to memory cells in DDR0 and DDR2.

[0065] Furthermore, taking BAR0 as an example, any address within it can be mapped to a memory cell in DDR0. For instance, memory cells in DDR0 are configured according to... Figure 3 Addressing in the order shown by the arrows on the left means that address 8 can be mapped to the memory cell in the 1st row and 3rd column of DDR0, and so on. In other words, if the target address provided by codec 0 is 8, and this address is located inside BAR0, then the on-chip network can access the memory cell in the 1st row and 3rd column of DDR0.

[0066] Addresses outside of BAR0 are mapped to memory cells within the global memory space formed by DDR0 and DDR1. These memory cells may belong to either DDR0 or DDR1, meaning they can access memory DDR1 located far from codec 0. For example, memory cells in the global memory space are arranged according to... Figure 3 Addressing in the order shown by the arrow on the right, address 18 can be mapped to the memory cell in the 3rd row and 3rd column of DDR0, address 20 can be mapped to the memory cell in the 1st row and 3rd column of DDR1, and so on. In other words, if the target address provided by codec 0 is 18, and this address is outside BAR0, then the on-chip network can access the memory cell in the 3rd row and 3rd column of DDR0; if the target address provided by codec 0 is 20, and this address is outside BAR0, then the on-chip network can access the memory cell in the 1st row and 3rd column of DDR1.

[0067] It should be understood that if the storage cells in the memory adopt and Figure 3 Different addressing methods will change the mapping relationship between addresses and storage units accordingly.

[0068] The address range allocated to each codec can be fixed or dynamically modifiable (meaning it can be modified during the use of the audio / video processing chip). The address range allocation can be predetermined during the manufacturing of the audio / video processing chip and embedded in the on-chip network, or it can be allocated by the driver of the audio / video processing chip, written to the on-chip network after allocation, or accessible by the on-chip network. This driver can run in the central processing unit (CPU) of the electronic device with the audio / video processing chip installed.

[0069] Since each codec corresponds to a different address range, after the on-chip network receives the target address, it needs to determine which codec sent the target address so that it can determine which address range the target address should be compared with.

[0070] In one implementation, when sending the target address to the on-chip network, the codec can also send its identifier, such as the codec's name. Upon receiving the target address and the codec identifier, the on-chip network can determine the address range corresponding to the codec based on the codec identifier and the correspondence between pre-configured codec identifiers and their corresponding address ranges. It can then determine whether the target address is within the codec's address range. If the target address is within the codec's address range, it executes the nearest access logic; if the target address is outside the codec's address range, it executes global access logic.

[0071] In another implementation, the on-chip network includes sub-networks corresponding to each codec, for example, Figure 2 The NOC on the left can be considered a sub-network corresponding to codec 0, and the NOC on the right can be considered a sub-network corresponding to codec 1. Of course, these two sub-networks are interconnected and together constitute the on-chip network.

[0072] When any codec needs to read or write audio or video data, it sends the target address to the sub-network corresponding to that codec in the on-chip network. The sub-network corresponding to that codec is only responsible for handling the memory access request of that codec, so it knows the address range corresponding to that codec. Thus, it can determine whether the target address is within the address range corresponding to that codec. If the target address is within the address range corresponding to that codec, it executes the nearest access logic; if the target address is outside the address range corresponding to that codec, it executes the global access logic.

[0073] In the first implementation, the on-chip network distinguishes different codecs by their identifiers, thereby determining the address range corresponding to the codec and making subsequent judgments. This simplifies the on-chip network's structural design, but requires the codec to provide additional information (the codec's identifier) ​​to identify itself. The second implementation, by implementing the on-chip network as multiple dedicated sub-networks corresponding to multiple codecs to determine the relationship between target addresses and address ranges, does not require the codec to provide additional information (the codec's identifier) ​​to identify itself. However, the design of the on-chip network may be slightly more complex.

[0074] It's important to note that global access schemes are optional in audio and video processing chips. If global access is not required—for example, if the target address provided by the codec is always within its corresponding address range—then the on-chip network does not need to implement global access functionality. Figure 2 In the text, the arrow for global access is shown as a dashed line, indicating that it is optional.

[0075] In summary, since each codec in the aforementioned audio and video processing chip is pre-allocated a continuous address range, and the addresses within each codec's address range are mapped to the codec's nearest memory, when a target address is within the address range of a codec, the on-chip network will access the location in the codec's nearest memory corresponding to the target address. This achieves the codec's nearest memory access, which significantly reduces access latency, helps maintain stable codec performance, and keeps chip power consumption at a low level.

[0076] Optionally, if the target address is outside the address range corresponding to the codec, it can be treated as a global address. The on-chip network can then access the location corresponding to the target address in the global memory space. This allows the codec to access nearby memory and, when necessary, access remote memory, enriching the codec's data access methods. It should be noted that the probability of the codec accessing remote memory is low, therefore it does not lead to a significant increase in average access latency or chip power consumption.

[0077] Below, based on the above embodiments, we will continue to introduce a scheme for implementing local and global access to memory by the codec in a channel-divided scenario. Similarly, the global access scheme is also optional. First, the following concepts involved in the scheme need to be explained:

[0078] Packet: The address range corresponding to each codec represents a memory packet, and each packet contains at least one memory location. For example, Figure 3 There are two groups in total, corresponding to BAR0 and BAR1 respectively. For the convenience of the following explanation, they will be referred to as Group0 and Group1. Group0 contains DDR0 and Group1 contains DDR1.

[0079] Group number: The number of all groups corresponding to multiple codecs. This number is global. For example, the group number of Group0 can be 0, and the group number of Group1 can be 1.

[0080] Intra-group address: The address of the memory cell contained within the group, which is independent of the address of the memory cell outside the group.

[0081] Figure 4 This is a schematic diagram of the channel distribution in the memory, which can be considered as corresponding to... Figure 3 DDR0 in the reference Figure 4 Each square represents a storage unit with a size of 4 bytes. The addressing order of these storage units is from top to bottom and from left to right. The group address of the first storage unit is 0, the group address of the second storage unit is 1, the group address of the gray storage unit in the figure is 6, and so on.

[0082] Channel: The memory cells in each group are divided into at least one channel, and each channel contains at least one memory cell. The number of memory cells contained in each channel can be the same. Dividing into channels is beneficial to improving the efficiency of memory access because memory cells in different channels can be accessed in parallel.

[0083] Reference Figure 3DDR0 is divided into 4 channels, namely Ch0, Ch1, Ch2 and Ch3, and DDR1 is also divided into 4 channels, namely Ch4, Ch5, Ch6 and Ch7. The global storage space formed by DDR0 and DDR1 is divided into a total of 8 channels, namely Ch0, Ch1, Ch2, Ch3, Ch4, Ch5, Ch6 and Ch7.

[0084] For DDR0, the distribution of its four channels can be as follows: Figure 4 As shown, the first memory unit belongs to Ch0, the second memory unit belongs to Ch1, the third memory unit belongs to Ch2, the fourth memory unit belongs to Ch3, and the fifth memory unit belongs to Ch0 again, and so on in a cycle. It should be understood that... Figure 4 For example only, the channel distribution in DDR0 may also be different. Figure 4 The differences shown are as follows: for example, the 1st and 2nd memory cells belong to Ch0, the 3rd and 4th memory cells belong to Ch1, the 5th and 6th memory cells belong to Ch2, the 7th and 8th memory cells belong to Ch3, the 9th and 10th memory cells belong to Ch0 again, and so on. The specific memory cell allocation for each channel can be designed based on the characteristics of the memory.

[0085] In-channel address: The addressing of memory cells within the channel, independent of the memory cell's address outside the packet. (See reference...) Figure 4 Taking Ch2 as an example, the channel address of the first memory unit (DDR0, row 3, column 1) is 0, the channel address of the second memory unit (DDR0, row 3, column 2, gray) is 1, the channel address of the third memory unit (DDR0, row 3, column 3) is 2, and the channel address of the fourth memory unit (DDR0, row 3, column 4) is 3.

[0086] Global channel number: A number assigned to the channels contained in all packets corresponding to multiple codecs, or it can be described as a number assigned to the channels contained in the global storage space. For example... Figure 3 The eight channels in the system have global channel numbers ranging from 0 to 7.

[0087] Channel number within a group: The numbering of channels within a group. For example, if four channels in Group0 are numbered 0 to 3, then the channel numbers within Group0, Ch1, Ch2, and Ch3 are 0, 1, 2, and 3, respectively. Similarly, if four channels in Group1 are numbered 0 to 3, then the channel numbers within Group4, Ch5, Ch6, and Ch7 are 0, 1, 2, and 3, respectively.

[0088] After introducing the concept of channels, refer again Figure 3Considering the case where the target address to be accessed is linear, meaning the target address increases continuously according to the arrangement of memory cells, such as 0, 1, 2, 3…, then for codec 0, if the target address is located inside BAR0, the access order of the channel corresponding to the target address is Ch0→Ch1→Ch2→Ch3→Ch0 (meaning first accessing the memory cell corresponding to Ch0 in DDR0, then accessing the memory cell corresponding to Ch1 in DDR0, then accessing the memory cell corresponding to Ch2 in DDR0, then accessing the memory cell corresponding to Ch3 in DDR0, and then accessing the memory cell corresponding to Ch0 in DDR0). For codec 1, if the target address is located outside BAR0, the channel access order corresponding to the target address is Ch0→Ch1→Ch2→Ch3→Ch4→Ch5→Ch6→Ch7→Ch0, and so on. For codec 1, if the target address is located inside BAR1, the channel access order corresponding to the target address is Ch4→Ch5→Ch6→Ch7→Ch4, and so on. If the target address is located outside BAR1, the channel access order corresponding to the target address is Ch0→Ch1→Ch2→Ch3→Ch4→Ch5→Ch6→Ch7→Ch0, and so on.

[0089] It is evident that when the codec performs global access, all channels contained in DDR0 and DDR1 are traversed. However, when performing local access, only channels contained in the nearest memory are traversed; for example, codec 0 will only traverse the channels contained in DDR0. Alternatively, it can be understood that during global access, the target address is repeatedly mapped to various channels in the global memory space, while during local access, the target address is only repeatedly mapped to channels contained in the nearest memory.

[0090] It is important to note that Figure 3 The example given is that the target address to be accessed is linear, just to illustrate the difference between global access and local access. In reality, the target address to be accessed may not be linear; for example, it may be random.

[0091] When the on-chip network determines that the target address is within the address range of the codec that sent the target address, accessing the location corresponding to the target address in the adjacent memory of that codec can be achieved through the following process:

[0092] A1. The on-chip network determines the packet number of the target address and the intra-group address corresponding to the target address based on the packet number and address range of each packet.

[0093] In this example, the sequence number and address range of each group are pre-configured. Continuing with the previous example, we know that the sequence number of Group0 is 0, and its corresponding address range is 0-15. The sequence number of Group1 is 0, and its corresponding address range is 16-31. Assuming the destination address is 6 and it is sent by codec 0, we can determine that the sequence number of the group containing the destination address is 0. Since the address range of Group0 starts from 0, the address within the group corresponding to the destination address is 6-0=6 (the destination address minus the first address of the address range).

[0094] A2. The on-chip network calculates the channel number within the channel where the target address is located and the channel address within the channel corresponding to the target address, based on the address within the group corresponding to the target address and the number of channels in the group to which the target address is located.

[0095] The number of channels in each group is determined when the channels are partitioned. The address within the group corresponding to the target address can be divided by the number of channels in the group containing the target address; the quotient is the address within the channel corresponding to the target address, and the remainder is the channel number within the group containing the target address. However, this calculation method only holds true when one memory unit corresponds to one address. For cases where one memory unit corresponds to multiple addresses, the calculation formula needs to be adjusted accordingly.

[0096] Continuing the previous example, dividing the target address's intra-group address 6 by the channel number 4 of the group containing the target address yields the target address's intra-channel address 1 and the intra-group channel number 2 of the channel containing the target address. Figure 4 It is the second storage unit in Ch2.

[0097] A3. The on-chip network calculates the global channel number of the channel containing the target address based on the group number of the group containing the target address, the number of channels in each group, and the channel number within the group of the channel containing the target address.

[0098] For simplicity, taking the case where each group has the same number of channels and the group number starts from 0 as an example, we can calculate the global channel number of the channel where the target address is located by multiplying the group number of the target address group by the number of channels in each group, and adding the channel number within the target address group.

[0099] Continuing with the previous example, by multiplying the group number 0 of the group containing the target address by the number of channels in each group, 4, and adding the channel number within the group of the channel containing the target address, 2, we can obtain the global channel number 2 of the channel containing the target address, which is Ch2.

[0100] A4. The on-chip network accesses the location in the codec's adjacent memory corresponding to the target address based on the global channel number and the channel address corresponding to the target address.

[0101] Given the global channel number and the channel address corresponding to the target address, the memory cell in which the target address is located is uniquely determined, thus enabling the on-chip network to access it.

[0102] When the on-chip network determines that the target address is outside the address range corresponding to the codec that sent the target address in the case of channel partitioning, accessing the location corresponding to the target address in the global memory space formed by multiple memories can be achieved through the following process:

[0103] B1. The on-chip network calculates the global channel number of the channel containing the target address and the channel address corresponding to the target address based on the target address and the total number of channels in the global storage space.

[0104] The total number of channels in the global storage space is determined when the channels are partitioned. If the global storage space is considered as a group, then step B1 is similar to step A2 and will not be repeated.

[0105] B2. The on-chip network accesses the location in the global memory space corresponding to the target address based on the global channel number and the channel address corresponding to the target address.

[0106] Step B2 is similar to step A4, and will not be repeated here.

[0107] Figure 5 This is a schematic diagram of the addressing process in an on-chip network. The nearest access processes A1-A4 and global access processes B1-B2 are shown in... Figure 5 This is reflected in all of these aspects. The diamond-shaped box can represent a logic judgment unit, MUX stands for multiplexer (which can be understood as a concatenation of input data), and DIV stands for divider. It's easy to see that although the nearest access process and the global access process are described step-by-step above, these two processes can actually be implemented using pure hardware units in the on-chip network.

[0108] Optionally, the audio and video processing chip provided in any of the above embodiments can be applied to a virtualization scenario. In this scenario, multiple users share an audio and video processing chip. Each codec in the chip (representing codec resources) and its adjacent memory (representing storage resources) can be allocated as an independent virtualization resource to a user (of course, a user can also be allocated multiple virtualization resources). In this way, each user believes that they are using the audio and video processing chip's resources exclusively.

[0109] Furthermore, since different users are allocated virtualization resources corresponding to different codecs and memory, the user data of different users are isolated from each other at the hardware level. This helps to improve the stability and security of user data and avoids the uncertainty caused by global data access.

[0110] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. (Refer to...) Figure 6 The electronic device 100 includes a central processing unit 110, an external memory 120, and an audio / video processing chip 130. These components are interconnected and communicate with each other via a communication bus 140 and / or other forms of connection mechanism (not shown).

[0111] The central processing unit 110 is used to perform overall control of the electronic device 100. The audio and video processing chip 130 can be the audio and video processing chip in any of the above embodiments. In the electronic device 100, the audio and video processing chip 130 can be used as a graphics processing unit (GPU). The external memory 120 is the memory of the electronic device 100, such as the disk or memory of the electronic device 100, which can be volatile or non-volatile. It is called "external" only to distinguish it from the memory in the audio and video processing chip 130.

[0112] The central processing unit 110 and other possible components can access the external memory 120, reading and / or writing data therein. Specifically, computer programs can be stored in the external memory 120, which the central processing unit 110 can read and run to implement the functions of the electronic device 100. For example, the external memory 120 can store a driver for the audio / video processing chip 130, which the central processing unit 110 can read and run. This driver, during runtime, allocates a contiguous address range for each codec of the audio / video processing chip 130.

[0113] In one implementation, the central processing unit 110 can also respond to an address configuration request to modify the address range corresponding to one or more codecs in the audio / video processing chip 130. This address configuration request can be triggered by a user, for example, for... Figure 2 In other words, if the address range corresponding to codec 0 is changed to correspond to the global memory space formed by DDR0 and DDR1, then codec 0 will only perform global access to the memory and not local access. That is, the above implementation method can dynamically modify the address range corresponding to the codec, thereby flexibly binding the codec and its corresponding memory, realizing flexible access of the codec to the memory (not limited to local access), and meeting different memory access requirements.

[0114] Understandable. Figure 6 The structure shown is for illustrative purposes only; the electronic device 100 may also include components that are more advanced than those shown. Figure 6 More components shown, or those with the same Figure 6 Different configurations are shown. For example, electronic device 100 may also include a camera for capturing audio and video data for processing by audio and video processing chip 130, and so on.

[0115] Electronic device 100 may be a mobile phone, PC, tablet computer, laptop computer, server, wearable device, robot, etc. Furthermore, electronic device 100 is not limited to a single device, but may also be a combination of multiple devices or a cluster of a large number of devices.

[0116] Because the electronic device 100 includes an audio / video processing chip 130, it can achieve efficient audio / video encoding and decoding and has low power consumption.

[0117] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An audio / video processing chip, characterized in that, include: Multiple codecs, multiple memories, and on-chip networks; The codec is used to send the target address of the audio and video data to the on-chip network when it is necessary to read or write the audio and video data; The on-chip network is used to determine whether the target address is within the address range corresponding to the codec, and when the target address is within the address range corresponding to the codec, access the location in the codec's adjacent memory corresponding to the target address; Each codec is pre-allocated a continuous address range, and the addresses within the address range corresponding to each codec are mapped to the adjacent memory of that codec. The address range corresponding to each codec represents a group of the memory. Each group contains at least one memory. The address within the group is the addressing of the memory cells contained in the group. The memory cells in the memory contained in each group are divided into at least one channel. The address within the channel is the addressing of the memory cells contained in the channel. The group number is the number of all groups corresponding to the multiple codecs. The channel number within the group is the number of the channels contained in the group. The global channel number is the number of the channels contained in all groups corresponding to the multiple codecs. The on-chip network is used to perform the following operations: determining the group number of the group containing the target address and the intra-group address corresponding to the target address based on the group number and address range of each group; calculating the intra-group channel number of the channel containing the target address and the intra-channel address corresponding to the target address based on the intra-group address corresponding to the target address and the number of channels in the group containing the target address; calculating the global channel number of the channel containing the target address based on the group number of the group containing the target address, the number of channels in each group, and the intra-group channel number of the channel containing the target address; and accessing the location corresponding to the target address in the proximity memory of the codec based on the global channel number and the intra-channel address corresponding to the target address.

2. The audio / video processing chip according to claim 1, characterized in that, The on-chip network is also used to access the location corresponding to the target address in the global storage space formed by the plurality of memories when the target address is outside the address range corresponding to the codec; Addresses outside the address range corresponding to each codec are mapped to the global storage space.

3. The audio / video processing chip according to claim 2, characterized in that, The storage units in the global storage space are divided into at least one channel. The address within the channel is the addressing of the storage units contained in the channel, and the global channel number is the number of the channels contained in the global storage space. The on-chip network is used to perform the following operations: calculate the global channel number of the channel where the target address is located and the channel address corresponding to the target address based on the target address and the total number of channels in the global storage space; and access the location in the global storage space corresponding to the target address based on the global channel number and the channel address corresponding to the target address.

4. The audio / video processing chip according to claim 1, characterized in that, The codec is also used to send the codec's identifier to the on-chip network when it is necessary to read or write audio and video data; The on-chip network is used to determine the address range corresponding to the codec based on the identifier of the codec, and to determine whether the target address is within the address range corresponding to the codec.

5. The audio / video processing chip according to claim 1, characterized in that, The on-chip network includes a sub-network corresponding to each codec. The codec is used to send the target address to the sub-network corresponding to the codec in the on-chip network when it is necessary to read or write audio and video data. The sub-network corresponding to the codec is used to determine whether the target address is within the address range corresponding to the codec.

6. The audio / video processing chip according to any one of claims 1-5, characterized in that, The audio / video processing chip also includes an address translation unit; The codec is used to send the original target address of the audio and video data to the address conversion unit when it is necessary to read or write the audio and video data. The original target address is a virtual address. The address translation unit is used to convert the original target address into the target address and send the target address to the on-chip network, wherein the target address is a physical address.

7. The audio / video processing chip according to any one of claims 1-5, characterized in that, Each codec and its adjacent memory are allocated as a separate virtualization resource.

8. An electronic device, characterized in that, include: Central processing unit, external memory, and audio / video processing chip as described in any one of claims 1-7; The central processing unit is used to run the driver program for the audio and video processing chip stored in the external memory. When running, the driver program allocates a continuous address range for each codec of the audio and video processing chip.

9. The electronic device according to claim 8, characterized in that, The central processing unit is also used to respond to address configuration requests and modify the address range corresponding to the codec.

Images