Cache data access scheduling method, processor, electronic device and storage medium

Abstract

Embodiments of the present application provide a cache data access scheduling method, a processor, an electronic device and a storage medium. The method comprises: in response to user access, allocating a cache block in a display memory for the user; generating cache index information corresponding to the user according to the cache block allocated for the user; grouping users belonging to the same batch processing task to instruct the GPU to batch process the cache data of the users in the same group; and generating a cache number list corresponding to each group of users based on the cache index information of the users and storing the cache number list in the display memory. By binding the user information with the cache block occupied by the user through the cache index information, the method of the present application can enable the GPU to perform efficient and reliable access operation and scheduling operation on the cache data in the display memory, greatly reducing the data transmission operation between the display memory and the memory, and thereby improving the computing efficiency of the GPU.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of computer processors, and more particularly, embodiments of the present application relate to a method for scheduling access to cached data, a processor, an electronic device, and a storage medium. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the application recited in the claims. Information in this section can include concepts that can not be previously conceived or previously explored. Accordingly, unless otherwise indicated herein, the content in this section is not, and should not be construed as, prior art to the description and claims of the present application, and is not admitted to be prior art by its inclusion in this section.

[0003] In the application of deep learning, the inference process of many applications adopts stream computing. In stream computing, the input data is not input to the computing device at one time, but is divided into multiple segments, and each segment is input to the computing device for inference computation at intervals. For example, in stream automatic speech recognition, the audio input by the user from the microphone is divided into several segments, and each segment is sent to the server for inference computation, so that the recognition result is given synchronously during the process of the user inputting the audio.

[0004] However, the computing process of the graphics processing unit (GPU) in the computer needs to read the data from the video memory to the computing unit first, and then perform the computation. If only one segment is computed each time, the model data read at this time is only used once, which will result in a large amount of memory access. If a batch of segments is computed each time, that is, a batch processing is performed each time, the model data read each time is used multiple times, and accordingly the memory access amount is relatively small, thereby improving the GPU computing efficiency. The automatic speech recognition model Conformer model widely used in industrial practice generally saves a plurality of computation results of a previous segment in the cache in order to improve the model performance, and the computation results are used together with the next segment for computation, thereby reducing the computation amount.

[0005] If the above computation results in the Conformer model are saved in the memory, a large amount of transmission operation will be generated, and the transmission bandwidth will be seriously insufficient, so that the GPU is in a state of waiting for data transmission most of the time, which greatly restricts the computing efficiency of the GPU. Therefore, in order to improve the computing efficiency, the above computation results need to be saved in the video memory. However, the cached data in the video memory is not convenient to manage, especially when coexisting with the batch processing, the operation of data access and scheduling of the cached data in the video memory will be more complex.

[0006] Therefore, it is urgent to provide an access scheduling scheme for cached data, so as to perform efficient and reliable access and scheduling operations on the cached data in the video memory, greatly reduce the data transmission operations between the video memory and the memory, and further improve the computing efficiency of the GPU. SUMMARY

[0007] For the purpose of improving the computing efficiency, the prior art usually caches the computing results of the GPU in the video memory, so as to facilitate the access and calling of data. However, due to the limited management capability of the GPU for the cached data and the large amount of transmission operations caused by repeated transmission between the video memory and the memory, the management of the cached data in the video memory is often complex.

[0008] Therefore, it is very necessary to provide an improved access scheduling scheme for cached data, so as to efficiently and reliably manage the cached data in the video memory.

[0009] In this context, the embodiments of the present application aim to provide an access scheduling method for cached data, a processor, an electronic device and a storage medium.

[0010] In a first aspect of the embodiments of the present application, an access scheduling method for cached data is provided, comprising: in response to user access, allocating a cached block in the video memory for the user; generating cached index information corresponding to the user according to the cached block allocated for the user; the cached index information comprising: a cached number index of the cached block occupied by the user; grouping users belonging to the same batch processing task to instruct the GPU to perform batch processing on the cached data of the users in the same group; based on the cached index information of the user, generating a cached number list corresponding to each group of users and storing it to the video memory; the cached number list is used to instruct the GPU to access the cached block corresponding to each cached number index in the cached number list.

[0011] In an embodiment of the present application, the generating of the cached index information corresponding to the user according to the cached block allocated for the user comprises: generating the cached index information corresponding to the user; updating a hash table in the memory according to the cached index information corresponding to the user; the hash table is used to register the corresponding relationship between the user and the cached index information.

[0012] In an embodiment of the present application, the access scheduling method for cached data further comprises: in response to user disconnection, clearing the information of the user in the hash table to release the cached block occupied by the user.

[0013] In an embodiment of the present application, the access scheduling method of the cached data further comprises: in response to a GPU scheduling signal, transmitting the cached data in the cache block occupied by the user from the video memory of the first GPU to the memory and then to the video memory of the second GPU; wherein the GPU scheduling signal is a signal indicating that the user is scheduled from the first GPU to the second GPU; and updating the cache index information corresponding to the user in the hash table.

[0014] In an embodiment of the present application, the length of the cache block is a fixed preset value; and accordingly, each user occupies N cache blocks, wherein N is a positive integer.

[0015] In an embodiment of the present application, the grouping of the users under the same batch processing task comprises: if the operator in the GPU is a cache-dependent operator, grouping the users according to the number of cache blocks occupied by the users, so that the users in the same group occupy the same number of cache blocks; wherein the number of rows of the input data matrix of the cache-dependent operator is related to the input length, and the number of columns is related to the cache length.

[0016] In an embodiment of the present application, the grouping of the users under the same batch processing task further comprises: if the operator in the GPU is a cache-independent operator, grouping the users under the same batch processing task into one group; wherein the number of rows of the input data matrix of the cache-independent operator is related to the input length, and the number of columns is fixed.

[0017] In an embodiment of the present application, after the grouping of the users according to the number of cache blocks occupied by the users, the method further comprises: if the operator is a non-sensitive operator, performing a 0 padding operation on the invalid region of the cache block corresponding to each group of users, so that the cache length of each cached data in the same group of users is consistent; wherein the calculation result of the non-sensitive operator before and after the 0 padding operation is the same.

[0018] In an embodiment of the present application, the cache index information further comprises a cache length index; and accordingly, after the grouping of the users according to the number of cache blocks occupied by the users, the method further comprises: if the operator is a sensitive operator, generating the cache length index and adding it to the cache index information of the user; wherein the calculation result of the sensitive operator before and after the 0 padding operation is different.

[0019] In an embodiment of the present application, the cache index information further comprises an additional cache number index and an input length index; accordingly, the cache data access scheduling method further comprises: generating the input length index and calculating the sum of the input length and the cache length in response to the input data of the user; if the sum of the input length and the cache length is greater than the fixed preset value, allocating an additional cache block to the user and generating an additional cache number index according to the additional cache block; and adding the input length index and the additional cache number index to the cache index information of the user.

[0020] In a second aspect of the embodiments of the present application, a cache data access scheduling processor is provided, comprising: a cache allocation unit configured to allocate a cache block in a display memory to a user in response to the user accessing; an index generation unit configured to generate cache index information corresponding to the user according to the cache block allocated to the user and store the cache index information in a memory unit; a user grouping unit configured to group users belonging to the same batch processing task to instruct a graphics processing unit (GPU) to batch process cache data of users in the same group; the index generation unit is further configured to generate a cache number list corresponding to each group of users according to the grouping result of the user grouping unit and the cache index information of the users and store the cache number list in the display memory; the memory unit stores a correspondence table of users and cache index information; wherein the cache index information comprises a cache number index of a cache block occupied by the user; and the cache number list is used to instruct the GPU to access the cache block corresponding to each cache number index in the cache number list.

[0021] In a third aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor; and a memory storing executable program instructions, when the program instructions are executed by the processor, the electronic device implements the method of any one of the first aspect.

[0022] In a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer program instructions, when the computer program instructions are executed by one or more processors, the electronic device implements the method of any one of the first aspect.

[0023] According to the access scheduling method of cache data provided by the embodiment of the present application, the information of the user and the cache block can be bound by allocating the corresponding cache block in the video memory for the user and associating the cache index information of the user and the cache block occupied by the user, and when the demand of batch processing is generated, a cache number list can be generated according to the grouping result of the users belonging to the same batch processing task and the cache index information of the users in the same group, and each cache number index in the cache number list can guide the GPU to access and call the cache data in the corresponding cache block. This process not only saves the time consumed for reading data from the memory in the traditional GPU computing process, but also realizes the overall planning and control of the GPU access scheduling operation, realizes efficient and reliable cache management, significantly reduces the management difficulty of the cache in the video memory, and guarantees the efficiency and stability of the GPU computing process.

[0024] Further, in some embodiments, in order to balance the workload between multiple GPUs, the data of a user can be moved from the video memory of the current GPU to the video memory of another GPU through the memory as a transfer station for the cache data scheduling, and the cache index information corresponding to the user is updated to ensure that the cache data can be normally accessed and scheduled subsequently. Moreover, since the interval time between two segments of the user is sufficient to complete two transmission tasks, the task delay of the user will not be affected by the scheduling operation.

[0025] Further, in some embodiments, the users are grouped according to the number of cache blocks occupied by the users, and the users with the cache data in the same cache length range are grouped into a group and then subjected to the 0 padding operation, so as to avoid that too much invalid data is filled in the cache data with a smaller cache length due to the adaptation to the cache data with a larger cache length, thereby causing unnecessary waste of computing resources.

[0026] Further, in some embodiments, the cache length index can be added to the cache index information, and the operator sensitive to the 0 padding operation is guided to perform corresponding calculation through the cache length index, so as to prevent the data in the invalid area of the cache block from affecting the calculation result of the operator.

[0027] Further, the users can also be grouped according to the number of cache blocks occupied by the users while the operator calculation is guided by the cache length index, so as to solve the problem of increased calculation amount caused by the index-guided operator calculation, and the grouping can compensate for the lack of calculation efficiency caused by the introduction of the index in the operator calculation. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which a number of embodiments of the present application are illustrated by way of example and not limitation. In the drawings:

[0029] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present application is shown schematically;

[0030] Figure 2 A flowchart of a method of scheduling access to cached data according to an embodiment of the present application is shown schematically;

[0031] Figure 3 A schematic diagram of a cache block of a user according to one embodiment of the present application is shown schematically;

[0032] Figure 4 A schematic diagram of a cache block of a user according to another embodiment of the present application is shown schematically;

[0033] Figure 5 A flowchart of a method of grouping users in a batch according to one embodiment of the present application is shown schematically;

[0034] Figure 6 A schematic diagram of a cache block of a user according to yet another embodiment of the present application is shown schematically;

[0035] Figure 7 A block diagram of a structure of a processor of scheduling access to cached data according to an embodiment of the present application is shown schematically;

[0036] Figure 8 A block diagram of a structure of an electronic device according to an embodiment of the present application is shown schematically;

[0037] In the drawings, like or corresponding elements are denoted by like or corresponding reference numerals. DETAILED DESCRIPTION

[0038] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present application, and in no way limit the scope of the present application. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present application is shown schematically. As Figure 1As shown, the computing system 100 can include a central processing unit (CPU) 101, a random access memory (RAM) 102, a read only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial peripheral 112, a parallel peripheral 113, and a display 114. Of these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial peripheral 112 is coupled to the serial interface controller 107, the parallel peripheral 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It will be appreciated that Figure 1 The structural diagram described is for the purpose of example only and is not intended to limit the scope of the present application. In some cases, certain devices can be added or removed depending on the specific circumstances.

[0040] Those skilled in the art will appreciate that embodiments of the present application can be realized in the form of a system, method or computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system." Furthermore, the present application can take the form of a computer program product on one or more computer readable medium(s) having computer readable program code embodied in the medium.

[0041] Any combination of one or more computer readable medium(s) can be utilized. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present document, a computer readable storage medium can be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0042] A computer readable signal medium can include a propagated data signal with computer executable code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that can be involved in

[0043] The code can be transmitted in any form, including, but not limited to, radio frequency, electrical, optical, acoustical, or any suitable combination thereof. A computer readable medium can include any medium that can be read by a computer. Examples of computer readable medium include, but are not limited to, magnetic media, optical media, and electrical media.

[0044] The computer program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, application specific circuitry, or field programmable gate array (FPGA) circuitry can execute the computer program code.

[0045] The computer program code can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus. The computer program code can be stored in any computer readable memory that can direct the computer or other programmable data processing apparatus to function in a

[0046] These computer program instructions can also be stored in a computer readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the computer program instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0047] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0048] In this document, it is to be understood that the number of elements in the figures are for illustration only and that any number of elements can be employed in any given case. Any references to claims, appended figures or any text in this document are not intended to limit the scope of the application.

[0049] The principles and spirit of the application will be explained in detail below with reference to several representative embodiments of the application. SUMMARY

[0051] The inventors have found that for GPU computing process, due to the large amount of cache data and frequent access, if all cache data is transmitted back to the memory, the transmission bandwidth will be severely insufficient, resulting in that the GPU is in a state of waiting for data transmission most of the time, which greatly restricts the computing efficiency of the GPU. But if all cache data is stored in the video memory, the ability of the GPU is difficult to adapt to such complex cache management work.

[0052] And by adopting an index-based cache data access scheduling method, the cache data is saved in the video memory, the CPU pre-computes the moving and storage information of the cache data, generates the cache index information corresponding to the user to indicate the storage address of the cache data of the user, and then transmits the cache index information to the video memory to indirectly utilize the processing capacity of the CPU to perform complex cache management.

[0053] After introducing the basic principles of the application, various non-limiting embodiments of the application will be specifically introduced below.

[0054] Overview of Application Scenarios

[0055] In the application of deep learning, the inference process of many applications adopts stream computing. In stream computing, the input data is not input to the computing device at one time, but is divided into multiple segments, and a segment is input every interval to perform inference calculation. For example, in stream automatic speech recognition, the audio input by the user from the microphone is divided into several segments, and each segment is sent to the server for inference calculation, so that the recognition result is given synchronously during the process of inputting the audio by the user.

[0056] In particular, the Conformer model, which is an automatic speech recognition model with high recognition rate in streaming automatic speech recognition, is widely used in industrial practice. The encoding process of the Conformer model has high computational complexity. In order to improve the performance, the calculation results of a previous segment are generally saved, and when a subsequent segment arrives, they are used for calculation together, thereby reducing the computational complexity.

[0057] In actual application, in order to improve the encoding performance of the Conformer model, the locality of data is used to reduce the memory bottleneck of cached data, so that a plurality of segments of a plurality of users are grouped into a batch of data for calculation in the GPU, and the calculation results are returned to each user after calculation, which makes the access scheduling of cached data in the video memory more complex.

[0058] Exemplary Method

[0059] It should be noted that the above application scenarios are only shown for the purpose of facilitating the understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0060] In the technical solution of the present application, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0061] The access scheduling method of cached data according to the exemplary embodiments of the present application will be described below with reference to the accompanying drawings.

[0062] Figure 2 The flowchart of the access scheduling method of cached data according to the embodiments of the present application is schematically shown. Referring to Figure 2 The access scheduling method of cached data provided by the embodiments of the present application can include:

[0063] In step 201, in response to user access, a cache block in the video memory is allocated to the user.

[0064] Each time a user establishes a connection, the CPU will allocate a cache block to the newly accessed user and record it, which is used for subsequent storage of input data and cached data of the newly accessed user.

[0065] It should be noted that in some embodiments, the user can initiate a simple connection establishment request, which is only used to apply to the CPU for allocating a cache block to the user in the video memory, and does not have substantial data content. In some other embodiments, when the user generates input data, it is regarded as user access, and at this time, the CPU allocates a cache block to the newly accessed user and stores the cached data obtained by calculating the input data of the user into the allocated cache block.

[0066] In step 202, the cache index information corresponding to the user is generated according to the cache block allocated to the user.

[0067] The cache index information includes the cache number index of the cache block occupied by the user, and can also be regarded as the cache number index of the cache block allocated to the user. The cache number index corresponds to the cache block one by one, and can be regarded as the identity ID or cache address number of the cache block. The GPU can identify the location of the cache block and access or schedule the cache data in the cache block under the guidance of the cache number index.

[0068] In some embodiments, the generation process of the cache index information can be regarded as an updating process of the correspondence table of the user and the cache index information in the memory.

[0069] For example, the process of generating the cache index information corresponding to the user according to the cache block allocated to the user includes: generating the cache index information corresponding to the user; and updating the hash table in the memory according to the cache index information corresponding to the user.

[0070] A hash table is stored in the memory of the CPU, and is used to register the correspondence between the user and the cache index information. In some embodiments, before the cache block is allocated to the user, the GPU can divide the video memory into a plurality of cache blocks and establish a hash table based on the cache number index of each cache block. The hash table has the correspondence between the user ID and the cache number index, and the corresponding cache number index can be quickly found by inputting the user ID, that is, the process of allocating the cache block by the CPU is equivalent to the process of updating the correspondence between the user ID and the cache number index in the hash table.

[0071] It should be noted that the memory also has a queue storing the number of idle cache blocks. When the cache block is used, a number is selected from the head of the queue, that is, the allocated cache block is determined, and the cache block number is added to the hash table to facilitate subsequent searching according to the user ID; when the cache block is released, the number to be released is added to the tail, and the mapping relationship between the user ID and the cache block number in the hash table is also deleted.

[0072] Correspondingly, in some embodiments of the application, the access and scheduling method of the cache data can also include: in response to the user disconnecting, clearing the information of the user in the hash table to release the cache block occupied by the user. Correspondingly, when the user disconnects, the number of the cache block released by the user needs to be added to the queue storing the number of idle cache blocks.

[0073] Exemplarily, assuming that the user A is disconnected, the user A is deleted from the hash table in correspondence with the cache number index a, and the cache number index a can be re-added to the queue as an idle cache block. In a subsequent allocation process, the CPU can call the cache number index a from the queue and add the correspondence between the cache number index a and a new user ID to the hash table to allocate the cache block corresponding to the new user ID to a newly accessed user.

[0074] In step 203, the users belonging to the same batch processing task are grouped.

[0075] In the embodiment of the present application, the batch processing refers to placing several similar computing tasks together for computing, which can fully utilize the model data read at one time and fully utilize the computing performance of the GPU.

[0076] Since the cache data of each user is of different lengths, there can be a large difference between the cache lengths of the cache data of the users under the same batch processing task, and therefore, the users belonging to the same batch processing task can be grouped, and the cache data of the users in the same group is processed by the GPU in batches.

[0077] It should be noted that if the cache lengths of the users under the same batch processing task are consistent, the users under the same batch processing task can be directly divided into one group.

[0078] In step 204, based on the cache index information of the users, a cache number list corresponding to each group of users is generated and stored in the video memory.

[0079] If several users are divided into the same group, it indicates that when the GPU executes the batch processing task of the group of users, the cache blocks of all the users in the group need to be accessed, and therefore, the CPU generates a cache number list according to the cache index information of the group of users in advance and stores it in the video memory for the GPU to read. The cache number list contains the cache number indexes of the cache blocks occupied by each user in the group and can be used to instruct the GPU to access the cache blocks of each user in the group.

[0080] Exemplarily, when the GPU calculates to the position where the cache data needs to be read at each layer, the GPU goes to the corresponding cache block position according to the cache number list in the video memory to obtain the relevant cache data for calculation. In this process, the GPU does not need to allocate resources to search for the storage position of the required cache data, but can find the cache block to be accessed according to the cache number index in the cache number list.

[0081] According to the access scheduling method of cached data provided above, the user and the cache block information can be bound by allocating a corresponding cache block in the video memory for the user and associating the user with the cache index information of the cache block occupied by the user, when the demand for batch processing occurs, a cache number list can be generated according to the grouping result of the users belonging to the same batch processing task and the cache index information of the users in the same group, each cache number index in the cache number list can guide the GPU to the corresponding cache block to access and call the cached data, this process not only saves the time consumed for reading data from the memory in the traditional GPU computing process, but also realizes the overall planning and control of the GPU access scheduling operation, realizes efficient and reliable cache management, significantly reduces the management difficulty of the cache in the video memory, and guarantees the efficiency and stability of the GPU computing process.

[0082] In actual application, even if the user has established a connection with a GPU, it cannot be guaranteed that all tasks generated by the user can be calculated in the GPU in time, when the GPU tasks are crowded, the response time of the task may be long, therefore, in the actual application process, the user who has established a connection can be scheduled from one GPU to another GPU.

[0083] In this case, the access scheduling method of cached data provided by some embodiments of the application can respond to the GPU scheduling signal to transfer the cached data in the cache block occupied by the user from the video memory of the first GPU to the memory and then to the video memory of the second GPU.

[0084] That is, in the scheduling process of the user, the memory of the CPU plays the role of a transit station to bridge the transmission of the cached data of the user between two video memories.

[0085] After the user cached data transfer is successful, the cache index information corresponding to the user in the hash table needs to be updated in time.

[0086] It should be noted that, since the interval time between the two segments of the user is sufficient to complete two transmission tasks, the task delay of the user will not be affected by the above scheduling operation.

[0087] In some embodiments of the application, the CPU can divide the video memory into a plurality of cache blocks before allocating the cache block for the user. Unlike the cache block in the memory which can simply use a dynamic length, the cache block in the video memory needs to set a special length mechanism.

[0088] In the process of continuously establishing and closing connections, if a dynamic length mechanism is used, the display memory will be cut into more and more fragments, resulting in that the CPU is difficult to allocate display memory space for subsequent user access, or a long time is needed to search in order to allocate appropriate length space for the user, which is not conducive to the performance release of the GPU.

[0089] The division mode of the cache block provided by some embodiments of the application is exemplarily described below.

[0090] In the cache block division process of the embodiment, the length of the cache block is set to a fixed preset value, and accordingly the display memory space occupied by each user is an integer block cache block.

[0091] Further, as the user's cache data increases over time, the length of the user's cache data exceeds the length of a cache block, at which time the CPU allocates additional cache blocks to the user to store data.

[0092] Specifically, the cache index information of the user further includes: an additional cache number index and an input length index.

[0093] When the user is a newly accessed user, the input length index is generated in response to the input data of the user, and whether an additional cache block needs to be allocated to the user is determined according to the input length. The cache data of the newly accessed user in the display memory is empty, i.e., the cache length is 0, at which time only the input length of the input data and the fixed preset value of the single block cache block need to be compared. If the input length is greater than the fixed preset value, it indicates that the single block cache block is insufficient to store the input data of the user, and an additional cache block is allocated to the user, and an additional cache number index is generated according to the additional cache block. The input length index and the additional cache number index are added to the cache index information of the user.

[0094] When the user is an already accessed user, the input length index is generated in response to the input data of the user, and the sum of the input length and the cache length is calculated. If the sum of the input length and the cache length is greater than the fixed preset value, an additional cache block is allocated to the user, and an additional cache number index is generated according to the additional cache block. The input length index and the additional cache number index are added to the cache index information of the user.

[0095] It should be noted that the above process can also regard the cache length of the newly accessed user as 0. In the process of determining whether an additional cache block needs to be allocated to the user, the sum of the input length and the cache length is calculated. If the sum of the input length and the cache length is greater than the fixed preset value, an additional cache block is allocated to the newly accessed user, and an additional cache number index is generated according to the additional cache block.

[0096] Since the fixed length cache block is used, it is inevitable that Figure 3As shown, for the cache data of some users, the cache blocks occupied by the cache data have invalid regions with no data in the column dimension. The valid regions storing the cache data will affect the cache-related operators when the GPU performs batch processing.

[0097] In the embodiments of the present application, the cache-related operator refers to an operator whose number of rows of an input data matrix is related to an input length and whose number of columns is related to a cache length.

[0098] Since the number of frames of the segments of each user is different, the number of rows occupied by the cache data of the user is different, as shown. Figure 4 Therefore, the CPU needs to calculate the corresponding cache number index in advance and transmit it to the GPU, so that each operator can perform corresponding processing based on the cache number index.

[0099] If the number of columns occupied by the cache data of the user is different, since the number of columns of the input data matrix of the cache-related operator is related to the cache length, additional processing is needed. Usually, the invalid regions are filled with 0, but there are sensitive operators in the cache-related operator which are sensitive to the 0 filling operation, and the 0 filling operation will affect the calculation result, resulting in incorrect calculation result. For example, the softmax operator needs to calculate ∑e x , x = 0, e x = 1, which will affect the summation result.

[0100] For such sensitive operators, some embodiments of the present application introduce a cache length index in the cache index information; accordingly, after grouping the users, if the operator is a sensitive operator, a cache length index is generated and added to the cache index information of the user. The cache length index can indicate the length of the cache data processed by the sensitive operator, avoiding the sensitive operator from processing the invalid regions as part of the cache length of the cache data, which results in incorrect calculation result.

[0101] If the cache-related operator is a non-sensitive operator, that is, the calculation result of the operator before and after the 0 filling operation is the same, for example, the matrix multiplication operator, after grouping the users, the invalid regions of the cache blocks corresponding to each group of users are filled with 0 to make the cache length of each cache data in the same group of users consistent.

[0102] Further, the grouping manner of the operators in the GPU when the operators are cache-related operators is introduced.

[0103] In this embodiment, if the operators in the GPU are cache-related operators, the users are grouped according to the number of cache blocks occupied by the users, so that the number of cache blocks occupied by the users in the same group is the same.

[0104] Because no matter whether it is 0 filling operation or guiding operator calculation through cache length index, it will cause the increase of calculation amount.

[0105] Taking 0 filling operation as an example, if users are not grouped according to the number of cache blocks occupied by the users and 0 filling operation is directly performed, in order to ensure that the cache lengths of users in the same batch are consistent, the user with the minimum cache length needs to adapt to the user with the maximum cache length, and the cache length is filled to the maximum cache length through 0 filling operation, which will cause the proportion of data 0 in the cache data of the user with the minimum cache length to increase sharply, and even the part of the extra cache block occupied by the user may be data 0, which not only causes the waste of cache blocks, but also leads to the decrease of GPU data reading speed and the increase of calculation amount.

[0106] After the users are grouped according to the number of cache blocks occupied by the users, 0 filling operation will only fill the remaining area of the cache block occupied by the current user, that is, the 0 filling length of each user will not exceed the length of a cache block.

[0107] The above is the description of the processing mode of the cache related operator, and the processing mode of the cache independent operator will be introduced below.

[0108] Because the number of rows of the input data matrix of the cache independent operator is related to the input length, and the number of columns of the input data matrix is fixed. Therefore, the difference of the cache block in the column dimension will not affect the number of columns of the input data matrix of the cache independent operator, and therefore, if the operator in the GPU is a cache independent operator, the users under the same batch processing task can be grouped into a group and submitted to the operator in the GPU for calculation, without the need for additional processing.

[0109] Based on the above description, in combination with Figure 5 The grouping method of the same batch of users provided by some embodiments of the application is described.

[0110] Figure 5 A flowchart schematically showing the grouping method of the same batch of users according to an embodiment of the application is shown.

[0111] Referring to Figure 5 The grouping method of the same batch of users provided by the embodiment includes:

[0112] In step 501, it is determined whether the operator corresponding to the batch processing task is a cache related operator.

[0113] If yes, steps 502 to 505 are sequentially executed;

[0114] If no, step 506 is executed.

[0115] In step 502, the users are grouped according to the number of cache blocks occupied by the users.

[0116] After grouping, users in the same group occupy the same number of cache blocks. Since the length of the cache block is set to a fixed preset value during the cache block partitioning process, the corresponding video memory space occupied by each user is an integer number of cache blocks. Therefore, grouping can be achieved based on the number of cache blocks occupied by each user.

[0117] In step 503, it is determined whether the operator is a sensitive operator.

[0118] If so, proceed to step 504;

[0119] If not, proceed to step 505.

[0120] In step 504, a cache length index is generated and added to the user's cache index information.

[0121] The cache length index reflects the actual length of the user's cached data and can indicate the number of columns in the operator's input data matrix to prevent it from being affected by invalid regions.

[0122] In step 505, invalid regions of the cache blocks corresponding to each user group are padded with zeros.

[0123] By padding with zeros, invalid areas in the cached data of users in the same group are filled to ensure that the cached data of each user fills the cache block in which they reside.

[0124] In step 506, users belonging to the same batch processing task are grouped together so that the GPU can perform calculations on the cached data of the same group of users.

[0125] For ease of understanding, the following illustrative description is provided in conjunction with the accompanying drawings.

[0126] by Figure 6 For example, Figure 6 The diagram illustrates the user's cache blocks at three time points. At the first time point, users 1 to 4 are grouped together and their tasks are computed together on the GPU. At the second time point, user 2 is grouped separately, while users 1, 3, and 4 are grouped together for computation. At the third time point, users 2 and 4 are grouped together, while users 1 and 3 are grouped together for computation.

[0127] It should be noted that, Figure 6 The blank blocks in the code represent invalid regions in the user's cached data.

[0128] The specific operation methods of steps 501 to 506 above have been described in detail above, and will not be repeated here.

[0129] It should be noted that the above Figure 5The illustrated same-batch user grouping method is only an exemplary illustration in the present application and should not constitute the only limitation of the present application.

[0130] By the same-batch user grouping method provided above, users can be grouped by the number of cache blocks occupied by the users, and users with cache data in the same cache length range are grouped into a group before the 0 padding operation, so as to avoid too much invalid data padding of cache data with a smaller cache length to adapt to cache data with a larger cache length, thereby causing unnecessary waste of computing resources.

[0131] By adding the cache length index in the cache index information, the operator sensitive to the 0 padding operation can be guided to perform corresponding calculation, so as to effectively prevent the data in the invalid area of the cache block from affecting the calculation result of the operator.

[0132] Furthermore, by the above same-batch user grouping method, the operator calculation can be guided by the cache length index, and the users can be grouped according to the number of cache blocks occupied by the users, so as to solve the problem of increased calculation amount caused by index-guided operator calculation, and the calculation efficiency loss caused by the introduction of the index in the operator calculation is compensated by grouping.

[0133] Exemplary Device

[0134] After introducing the method of the exemplary embodiment of the present application, next, with reference to Figure 7 The access scheduling processor of the cache data of the exemplary embodiment of the present application is described.

[0135] Figure 7 The structure block diagram of the access scheduling processor of the cache data of the embodiment of the present application is schematically shown. As shown in the figure, Figure 7 The processor includes:

[0136] The cache allocation unit 701 is configured to allocate a cache block in the video memory for a user in response to the user access;

[0137] The index generation unit 702 is connected with the cache allocation unit 701 and is configured to generate the cache index information corresponding to the user according to the cache block allocated by the cache allocation unit for the user and store the cache index information in the memory unit;

[0138] The user grouping unit 703 is connected with the index generation unit 702 and is configured to group the users belonging to the same batch processing task to instruct the GPU to perform batch processing on the cache data of the users in the same group;

[0139] Correspondingly, the index generation unit 702 is further configured to generate the cache number list corresponding to each group of users according to the grouping result of the user grouping unit and the cache index information of the users and store the cache number list in the video memory.

[0140] Memory unit 704 stores a table that corresponds to user and cache index information;

[0141] The cache index information includes: the cache number index of the cache block occupied by the user; the cache number list is used to instruct the GPU to access the cache block corresponding to each cache number index in the cache number list.

[0142] It should be noted that this processor is located within the CPU and manages the user data stored in the video memory. It generates a list of cache numbers based on the storage location of the cached data and stores it in the video memory. The cache index information in the cache number list can instruct the GPU to access and schedule the cached data at the corresponding location, thereby realizing the CPU's indirect management of the video memory and making full use of the CPU's data processing capabilities.

[0143] Corresponding to the aforementioned functional embodiments, this invention also provides a method such as... Figure 8 The electronic device shown. Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Figure 8 The illustrated electronic device 800 includes: a processor 810; and a memory 820 storing executable program instructions that, when executed by the processor 810, cause the electronic device to perform any of the methods described above.

[0144] exist Figure 8 In the electronic device 800, only the components relevant to this embodiment are shown. Therefore, it will be apparent to those skilled in the art that the electronic device 800 may also include components related to... Figure 8 The following are common constituent elements with different constituent elements.

[0145] Processor 810 can control the operation of electronic device 800. For example, processor 810 controls the operation of electronic device 800 by executing a program stored in memory 820 on electronic device 800. Processor 810 can be implemented by a central processing unit (CPU), application processor (AP), artificial intelligence processor chip (IPU), etc., provided in electronic device 800. However, this disclosure is not limited thereto. In this embodiment, processor 810 can be implemented in any suitable manner. For example, processor 810 can take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc.

[0146] The memory 820 can be hardware for storing various data processed in the electronic device 800, instructions. For example, the memory 820 can store processed data and data to be processed in the electronic device 800. The memory 820 can store a data set processed or to be processed by the processor 810, for example, input data of a user, cache data, and cache index information, etc. In addition, the memory 820 can store an application, a driver, etc. to be driven by the electronic device 800. For example, the memory 820 can store various programs related to task type identification, operator type identification, etc. to be performed by the processor 810. The memory 820 can be a DRAM, but the present disclosure is not limited thereto. The memory 820 can include at least one of a volatile memory or a non-volatile memory. The non-volatile memory can include a read only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a phase change RAM (PRAM), a magnetic RAM (MRAM), a resistive RAM (RRAM), a ferroelectric RAM (FRAM), etc. The volatile memory can include a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), a PRAM, an MRAM, a RRAM, a ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 820 can include at least one of a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, a cache, or a memory stick.

[0147] In summary, the specific functions of the memory 820 and the processor 810 of the electronic device 800 according to the embodiments of the present disclosure can be explained in contrast to the aforementioned embodiments of the present disclosure, and the technical effects of the aforementioned embodiments can be achieved, and thus a detailed description thereof will not be repeated here.

[0148] Alternatively, the present disclosure can also be implemented as a non-transitory machine readable storage medium (or computer readable storage medium, or machine readable storage medium) having stored thereon computer program instructions (or computer programs, or computer instruction codes) for estimating weight of a pig, which when executed by a processor of an electronic device (or electronic device, server, etc.) causes the processor to perform part or all of the steps of the above method according to the present disclosure.

[0149] It should be noted that, although several means or sub-means of the electronic device are mentioned in the foregoing detailed description, such a division is not mandatory only. Indeed, according to embodiments of the application, the features and functions of two or more means described above can be embodied in one means. Conversely, the features and functions of one means described above can be further divided into means embodied by several means.

[0150] Moreover, although the operations of the method according to the application are described in a particular order in the drawings, this is not meant to imply that the operations must be performed in that particular order, or that all illustrated operations must be performed, in order to realize the desired results. On the contrary, the steps depicted in the flowcharts can be changed in execution order. Additionally or alternatively, certain steps can be omitted, combined into one step, and / or divided into several steps.

[0151] The use of the verb "comprise", "comprising", the verb "include", "including" and their conjugations in the application file do not exclude the presence of elements other than those mentioned in the application file. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.

[0152] While the principles and spirit of the application have been described with reference to several specific embodiments, it is to be understood that the application is not limited to the disclosed specific embodiments, and that the division into aspects does not imply that features from these aspects cannot be combined to benefit, this division being only for ease of presentation. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is the broadest interpretation under the law, to include all such modifications and equivalent structures and functions.

Claims

1. A method of scheduling access to cached data, characterized by, The application is applied to a CPU, comprising: in response to a user accessing, allocating a cache block in a display memory for the user; generating cache index information corresponding to the user according to the cache block allocated for the user; the cache index information comprises a cache number index of the cache block occupied by the user; grouping users belonging to the same batch processing task to instruct a graphics processing unit (GPU) to batch process cache data of the same group of users; based on the cache index information of the user, generating a cache number list corresponding to each group of users and storing the cache number list to the display memory; the cache number list is used to instruct the GPU to access the cache block corresponding to each cache number index in the cache number list.

2. The method of claim 1, wherein, The application further comprises: in response to the user disconnecting, clearing the information of the user in the hash table to release the cache block occupied by the user. The application further comprises:

3. The method of claim 2, wherein, in response to a GPU scheduling signal, transmitting the cache data in the cache block occupied by the user from the display memory of a first GPU to the memory and then to the display memory of a second GPU; wherein the GPU scheduling signal is a signal instructing the user to be scheduled from the first GPU to the second GPU; updating the cache index information corresponding to the user in the hash table.

4. The method of claim 2, wherein, 5. The access scheduling method of the cache data according to claim 1, wherein the length of the cache block is a fixed preset value; correspondingly, each user occupies N cache blocks, wherein N is a positive integer. The application further comprises: if the operator in the GPU is a cache-dependent operator, grouping the users according to the number of cache blocks occupied by the users so that the number of cache blocks occupied by the users in the same group is the same; wherein the number of rows of the input data matrix of the cache-dependent operator is related to the input length, and the number of columns is related to the cache length. The application further comprises:

6. The method of claim 5, wherein, if the operator in the GPU is a cache-independent operator, grouping the users belonging to the same batch processing task into one group; wherein the number of rows of the input data matrix of the cache-independent operator is related to the input length, and the number of columns is fixed. The application further comprises:

7. The method of claim 6, wherein, if the operator is a non-sensitive operator, performing a 0 padding operation on the invalid region of the cache block corresponding to each group of users so that the cache length of each cache data in the same group of users is consistent; wherein the calculation result of the non-sensitive operator before and after the 0 padding operation is the same.

9. The access scheduling method of the cache data according to claim 6, wherein 8. The method of claim 6, wherein, the cache index information further comprises a cache length index; correspondingly, the application further comprises: if the operator is a sensitive operator, generating the cache length index and adding the cache length index to the cache index information of the user; wherein the calculation result of the sensitive operator before and after the 0 padding operation is different. ​ ​ ​ 10. The method of claim 5, wherein the cache index information further comprises an additional cache number index and an input length index. Accordingly, the method of claim 5 further comprises: generating the input length index and the additional cache number index in response to the input data of the user, and calculating a sum of the input length and the cache length, wherein if the sum of the input length and the cache length is greater than the fixed preset value, allocating an additional cache block to the user, and generating the additional cache number index according to the additional cache block; adding the input length index and the additional cache number index to the cache index information of the user. The method comprises:

11. A cache data access scheduling processor, such that the processor implements the method of claim 1, wherein, a cache allocation unit configured to allocate a cache block in a display memory to a user in response to the user accessing the display memory; an index generation unit configured to generate cache index information corresponding to the user according to the cache block allocated to the user and store the cache index information in a memory unit; a user grouping unit configured to group users belonging to the same batch processing task to instruct a graphics processing unit (GPU) to batch process cache data of the users in the same group; the index generation unit is further configured to generate a cache number list corresponding to each group of users according to the grouping result of the user grouping unit and the cache index information of the users and store the cache number list in the display memory; a memory unit storing a correspondence table of users and cache index information; wherein the cache index information comprises a cache number index of a cache block occupied by the user, and the cache number list is used to instruct the GPU to access the cache block corresponding to each cache number index in the cache number list. The method comprises:

12. An electronic device, comprising: a processor; and a memory storing executable program instructions, wherein when the program instructions are executed by the processor, the electronic device implements the method of any one of claims 1-10. A computer program instruction is stored thereon, wherein when the computer program instruction is executed by one or more processors, the processor implements the method of any one of claims 1-10. ​ 13. A computer-readable storage medium, characterized in that, ​

Images