Terminal large model parameter collaborative loading method, scheduling method and system

Abstract

The present invention relates to the field of large model parameter processing and discloses a terminal large model parameter collaborative loading method, scheduling method and system, comprising: determining the file size according to the model file to be loaded, and opening a memory area in the inference device based on the file size, and byte-aligning the starting address of the memory area; obtaining the cache status of the model to be loaded in the memory, and dividing the model to be loaded into a cached area and an uncached area according to the cache status; obtaining the hard disk loading speed of the inference device and the network status of the remote memory device, and dividing the uncached area into a hard disk loading area and a remote memory loading area; loading the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area, loading the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area, and loading the model parameters of the remote memory loading area from the remote memory device to the corresponding position of the memory area.

Description

Technical Field

[0001] The present invention relates to the field of large model parameter processing, and in particular to a terminal large model parameter collaborative loading method, scheduling method and system. Background Art

[0002] Large language models are deep learning models trained using large amounts of text data. They can generate natural language text or understand the meaning of text. After years of development, they now offer powerful application capabilities in natural language processing, text generation, machine translation, intelligent customer service, and other fields, significantly improving automation and intelligence. With the continuous improvement of computer hardware performance and the growing demand for stronger model understanding and expression capabilities, the parameter size of large models is increasing. Currently, with the ever-increasing parameter size, local services are no longer able to meet the needs of intelligent applications. Most Transformer-based intelligent applications rely heavily on cloud services. Edge terminals, as the last stop for large models from cloud services, offer advantages such as bandwidth savings, user privacy protection, improved offline support, lower costs, and good scalability. Therefore, performing large model inference on edge terminals is gradually becoming a mainstream trend. However, edge terminals often have limited hardware performance. When reading data on edge terminals, the slow hard drive read speeds within them result in lengthy model parameter loading times during large model inference, significantly reducing the user experience. It can be seen that the existing edge terminal parameter loading method has problems such as long loading time and poor user experience when performing large model inference. Summary of the Invention

[0003] The present invention provides a terminal large model parameter collaborative loading method, scheduling method and system to solve the problems of long loading time and poor user experience in the existing edge terminal parameter loading method when performing large model inference.

[0004] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0005] In a first aspect, the present invention provides a method for collaboratively loading terminal large model parameters, comprising:

[0006] S110: Determine the file size of the model file to be loaded, allocate a memory area in the inference device based on the file size, and byte-align the starting address of the memory area;

[0007] S120, obtaining a cache status of the model to be loaded in the memory, and dividing the model to be loaded into a cached area and a non-cached area according to the cache status;

[0008] S130, obtaining a hard disk loading speed of the inference device and a network status of a remote memory device, and dividing the uncached area into a hard disk loading area and a remote memory loading area;

[0009] S140. Load the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area, load the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area, and load the model parameters of the far memory loading area from the remote memory device to the corresponding position of the memory area.

[0010] Optionally, in S110, allocating a memory area in the inference device based on the file size includes:

[0011] A memory region size determination model is constructed to determine the size of the memory region to be confirmed. The model satisfies the following relationship:

[0012] ;

[0013] Where, Indicates the size of the memory area to be confirmed. Indicates the file size of the model to be loaded, Indicates the number of aligned bytes used by the model to be loaded. Indicates the number of bytes of memory occupied by untyped pointers in programming languages;

[0014] Use the malloc function to allocate a memory area in the local memory of the inference device with the same size as the memory area to be confirmed;

[0015] The byte alignment of the starting address of the memory area includes:

[0016] Construct a byte alignment address determination model, and use the byte alignment address determination model to determine the address after byte alignment. The model satisfies the following relationship:

[0017] ;

[0018] Where, Indicates the byte-aligned address, Indicates the first address of the memory area, Indicates the number of aligned bytes used by the model to be loaded. Indicates the number of bytes of memory occupied by untyped pointers in programming languages. Represents a bitwise AND operation, Represents bitwise inverse operation;

[0019] Load the model parameters starting from the byte-aligned address to complete the byte alignment.

[0020] Optionally, in S120, obtaining a cache status of the model to be loaded in the memory, and dividing the model to be loaded into a cached area and a non-cached area according to the cache status, includes:

[0021] Get the cache status of each page of the model to be loaded in memory, and mark the pages that are cached as cached, and mark the pages that are not cached as not cached;

[0022] The pages marked as cached are logically grouped into a cached area, and all pages marked as uncached are logically grouped into an uncached area.

[0023] Optionally, in S130, obtaining the hard disk loading speed of the inference device and the network status of the far memory device, and dividing the uncached area into a hard disk loading area and a far memory loading area, includes:

[0024] Use the test function to obtain the local hard disk loading speed of the inference device and the bandwidth of the network connecting the remote memory device and the inference device, and calculate the ratio of the local hard disk loading speed to the network bandwidth;

[0025] The total memory size to be occupied by the uncached area is calculated, and the total memory size to be occupied by the uncached area is divided into a hard disk loading area and a remote memory loading area based on the ratio of the local hard disk loading speed to the network bandwidth.

[0026] Optionally, in S140, loading the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area includes:

[0027] Opening a first thread to load the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area;

[0028] Loading the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area includes:

[0029] Opening a second thread to load the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area;

[0030] Loading the model parameters of the remote memory loading area from the remote memory device to the corresponding position of the memory area includes:

[0031] Opening a third thread to load the model parameters of the remote memory loading area from the remote memory device to the corresponding position of the memory area;

[0032] When the model parameters of the first thread, the second thread, and the third thread are loaded, the parameter loading of the model file to be loaded is completed.

[0033] In a second aspect, an embodiment of the present application provides a terminal large model parameter scheduling method for dynamically allocating parameters of a model file to be loaded based on the terminal large model parameter collaborative loading method as described in the first aspect, including:

[0034] S210. Obtain memory usage, maximum memory, and network bandwidth information of all far memory devices connected to the inference device, and calculate a parameter allocation priority for each far memory device based on the memory usage, maximum memory, and network bandwidth information;

[0035] S220: constructing a priority-based parameter allocation algorithm according to the parameter allocation priority, and using the priority-based parameter allocation algorithm to preliminarily allocate the parameters of the model file to be loaded on the inference device to the memory of the remote memory device at a page granularity;

[0036] S230: Acquire the memory usage of the remote memory device and the time series change information of the connection status with the inference device in real time, and dynamically reallocate the parameters of the model file to be loaded using a priority-based parameter allocation algorithm according to the time series change information.

[0037] Optionally, in S210, calculating the parameter allocation priority of each remote memory device according to the memory usage, the maximum memory, and the network bandwidth information includes:

[0038] The parameter allocation priority calculation formula is used to calculate the parameter allocation priority of each far memory device. The calculation formula satisfies the following relationship:

[0039] ;

[0040] Where, Indicates the parameter assignment priority, Indicates the connection status between the far memory device and the inference device. For the connection to exist, If no connection exists, Indicates the maximum memory of the far memory device. represents the network bandwidth between the far memory device and the inference device, Indicates the memory usage of the far memory device;

[0041] The priority-based parameter allocation algorithm includes:

[0042] Divide the parameters to be allocated into pages, calculate the number of pages for the parameters to be allocated, and obtain the number of pages occupied by the parameters;

[0043] A priority-based parameter allocation algorithm is constructed based on the number of pages occupied by the parameters and the parameter allocation priority, and the algorithm satisfies the following relationship:

[0044] ;

[0045] Where, Indicates the number of parameter pages that need to be allocated on each far memory device. Indicates the number of pages occupied by the parameter. Indicates the sum of the priorities of all connected far memory devices.

[0046] Optionally, in S230, dynamically reallocating the parameters of the model file to be loaded using a priority-based parameter allocation algorithm according to the time series change information includes:

[0047] When the memory usage of a far memory device exceeds the memory usage threshold, some parameters on the device are reallocated to other far memory devices using a priority-based parameter allocation method.

[0048] When it is found that a far memory device loses connection, all parameters on the device are reallocated to other far memory devices using a priority-based parameter allocation method.

[0049] In a third aspect, an embodiment of the present application provides a terminal large model parameter collaborative loading system, which is used to implement the terminal large model parameter collaborative loading method described in the first aspect, including:

[0050] System initialization module: used to initialize the inference device and far memory device, enabling the inference device to load data from the memory of the far memory device and preliminarily allocate the parameters of the large model file to the memory of these devices;

[0051] Far memory device management module: used to manage each far memory device and record in real time the connection status between each far memory device and the inference device, network bandwidth, memory usage, maximum memory, and other information;

[0052] Parameter management module: used to manage parameters and record the distribution of different parts of large model parameters on different remote memory devices in real time;

[0053] Parameter scheduling decision module: This module is used to make parameter scheduling strategy decisions and actual parameter scheduling in real time based on the far memory device information from the far memory device management module and the distribution of different parts of the large model parameters on different far memory devices.

[0054] Parameter loading module: used to provide the program with the function of loading parameters from remote memory, and automatically determine the loading method based on the distribution of large model parameters on different devices.

[0055] In a fourth aspect, an embodiment of the present application provides a terminal large model parameter scheduling system, comprising at least one control processor and a memory for communicating with the control processor;

[0056] The memory stores instructions that can be executed by at least one control processor, and the instructions are executed by at least one control processor to enable the at least one control processor to execute the terminal large model parameter scheduling method as described in the second aspect.

[0057] Beneficial effects:

[0058] The method for collaboratively loading large terminal model parameters provided by this invention can more fully utilize the overall IO capabilities of the terminal device. It adds a loading method from remote memory to the conventional hard disk loading method, providing a fixed acceleration effect for model files of any size (for example, a 2.375x acceleration ratio can be provided when the hard disk loading speed is 80MB / s and the network bandwidth is 110MB / s). The larger the model file, the more significant the acceleration effect. Furthermore, this method does not rely on any specific additional hardware and has good scalability. It can be added to any other software or hardware acceleration method to further accelerate parameter loading.

[0059] At the same time, the terminal large model parameter scheduling method provided by this invention can more fully utilize the idle memory of surrounding terminal devices in edge terminal scenarios. Because it uses the memory usage, maximum memory, and network bandwidth of far-memory devices to determine parameter allocation priority, this method can ensure optimization goals such as minimizing far-memory consumption and minimizing switching between model parameters and far-memory device mappings. This can reduce waste of far-end memory and transmission bandwidth. In addition, it also considers far-memory device load balancing to prevent excessive memory pressure on certain far-memory devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the method for collaboratively loading terminal large model parameters provided by the present invention;

[0061] Figure 2 This is a data flow diagram of the terminal large model parameter collaborative loading method provided by the present invention;

[0062] Figure 3 It is a flow chart of the terminal large model parameter scheduling method provided by the present invention;

[0063] Figure 4 This is a structural diagram of the terminal large model parameter collaborative loading system provided by the present invention;

[0064] Figure 5 It is a structural diagram of the terminal large model parameter scheduling system provided by the present invention. DETAILED DESCRIPTION

[0065] The following is a clear and complete description of the technical solutions of the present invention. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0066] Unless otherwise defined, the technical or scientific terms used in the present invention shall have the usual meanings understood by persons of ordinary skill in the field to which the present invention belongs. The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "a" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0067] See Figure 1-2 ,like Figure 1 As shown, an embodiment of the present application provides a method for collaboratively loading terminal large model parameters based on far memory, which method includes at least the following steps:

[0068] S110, obtaining the file size according to the model file to be loaded by the inference device, and manually opening up a memory area of ​​a certain size based on the file size and byte-aligning the starting address of the area;

[0069] According to the model file path, use the fopen function to open the model file in "rb" mode, use the fseek function to move the file read / write pointer to the end of the file, use the ftell function to get the offset of the file read / write pointer relative to the beginning of the file at this moment, which is equal to the file size, and the file size is recorded as filesize. Finally, use the rewind function to return the file read / write pointer to the beginning of the file;

[0070] Use the getpagesize function to obtain the page size supported by the system, which is recorded as pagesize. The number of alignment bytes used by the large model is recorded as alignment. According to the formula Get the size of the memory area to be opened, memorysize. The sizeof(void *) in the formula refers to the untyped pointer in the programming language, that is, the number of bytes of memory space occupied by the void type pointer in the C++ language. Use the malloc function to open a memory area of ​​memorysize in the memory, and save the first address of the memory area to the raw pointer.

[0071] Using the formula Align raw to the modelbuffer address according to the alignment requirements, and subsequently use modelbuffer as the starting address for loading large model parameters. At the same time, assign the value of the raw pointer to the modelbuffer[-1] pointer to facilitate the subsequent release of the entire memory segment opened by malloc.

[0072] S120, obtaining the cache status of the model file in memory, dividing the model file into a cached area and an uncached area;

[0073] Use the open function to open the file in read-only mode according to the model file path, obtain the file descriptor of the model file, and use the mmap function to map the model file to the memory address space according to the file descriptor;

[0074] Use the mincore function to retrieve the cache status of each page of the model file in memory. Create two vector containers, a cached page vector and an uncached page vector, and iterate through the cache status of all pages in sequence, performing different operations depending on whether the page is cached. If the page is cached, the starting and ending addresses of the parameters of the page are stored in the cached page vector. If the page is not cached, the starting and ending addresses of the parameters of the page are stored in the uncached vector.

[0075] S130, obtaining the hard disk loading speed of the inference device and the network status of the remote memory device, and further dividing the uncached area into a hard disk loading area and a remote memory loading area;

[0076] Use a custom test function to obtain the hard disk loading speed of the inference device and the network speed of the far memory device (that is, the speed of loading parameters from the far memory). Divide the local hard disk loading speed by the far memory loading speed to get the ratio, which is recorded as M.

[0077] Calculate the number of pages in the uncached page vector and divide the pages into two parts according to the ratio M. The first part of the pages is further divided into the hard disk load page vector, and the second part of the pages is divided into the far memory load page vector.

[0078] S140. Load the model parameters of the cached area from the memory of the inference device to the corresponding position of the manually opened memory area, load the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the manually opened memory area, and load the model parameters of the far memory loading area from the remote memory to the corresponding position of the manually opened memory area.

[0079] Create a thread and use the fread function to load the parameters pointed to in the cached page vector into the corresponding location of the manually allocated memory area. At this time, the fread function uses the file descriptor of the model file in the memory of the inference device to read the parameters.

[0080] Create a thread and use the fread function to load the parameters pointed to in the hard disk load page vector into the corresponding position of the manually allocated memory area. At this time, the fread function uses the file descriptor of the model file on the hard disk of the inference device to read the parameters;

[0081] Create a thread and use the fread function to load the parameters pointed to in the cached page vector into the corresponding position of the manually allocated memory area. At this time, the fread function uses the file descriptor of the model file in the far memory to read the parameters.

[0082] Use the pthread_join function for each thread to wait for them to complete execution. After all threads have completed execution, the model parameter loading process is completed, and the first address of the model file in the memory is returned to the program for use in subsequent inference processes.

[0083] It can be understood that this terminal large model parameter collaborative loading method based on far memory can provide a stable parameter loading acceleration effect under a stable network condition. After experimental testing, when the hard disk loading speed of the inference device is 80MB / s and the network bandwidth between the inference device and the far memory device is 110MB / s, the use of this parameter collaborative loading method can increase the original 80MB / s parameter loading speed to 190MB / s, which can provide a stable 2.375 times acceleration ratio. When using a 3.6GB model file, the parameter loading time can be reduced from 44 seconds to 18 seconds, greatly reducing the waiting time for users when performing large model inference.

[0084] See Figure 3 The embodiment of the present application provides a large model parameter scheduling method based on far memory, which is used to support the terminal large model parameter collaborative loading method based on far memory, including:

[0085] S210. Obtain memory usage, maximum memory, and network bandwidth information of all far memory devices connected to the inference device, and calculate the parameter allocation priority of each far memory device based on the memory usage, maximum memory, and network bandwidth information of the far memory devices;

[0086] Use the ping command to obtain the connection status between the inference device and each far memory device. When the connect value is 1, it means that the inference device has a network connection with the device. When it is 0, it means that the inference device has no network connection with the device. Use the free command to calculate the memory usage mem_usage and maximum memory max_mem of each far memory. Start the iperf server on each far memory device, use the iperf client program on the inference device to measure the network bandwidth net_speed of each far memory device, and then use the formula Get the priority P of each far memory device i .

[0087] S220: Preliminarily allocate the parameters of the model file on the inference device to the memory of the far memory device at a page granularity using a priority-based parameter allocation algorithm according to the parameter allocation priority of each far memory device;

[0088] The number of pages occupied by the model file is recorded as N, then according to the formula The calculated parameter amount to be allocated to each far memory is S i Pages, then S i Round up and down so that S i The values ​​of are all integers, and the total number is equal to the total number of model file pages;

[0089] Traverse each far memory device in turn, according to their number of pages to be allocated S i To determine the range of model file parameters that should be stored in their memory, and load the corresponding part of the model parameters into their memory in advance through the network.

[0090] S230. Acquire the memory usage of the remote memory device and the time series change information of the connection status with the inference device in real time, and dynamically reallocate the large model parameters using a priority-based parameter allocation algorithm based on the time series change information.

[0091] Run the free command every second to calculate the memory usage (mem_usage) and maximum memory (max_mem) of each far memory device. Use the ping command to check the connection status of the inference device with other far memory devices.

[0092] If the mem_usage of a far memory device exceeds a specified threshold (e.g. 80%), the model parameters on the device memory will be reallocated to other far memory devices. Calculate the priority P of other far memory devices i , determine whether releasing the memory of some parameters of the device can reduce the memory usage to below a specified threshold (for example, 60%). If so, then the parameters are reallocated to the corresponding far memory device according to the parameter allocation priority of other far memory devices, and the memory occupied by the parameters is released on the far memory device. If not, then all parameters are reallocated to the corresponding far memory device according to the parameter allocation priority of other far memory devices, and the memory occupied by the parameters is released on the far memory device.

[0093] If it is found that a far memory device loses connection, the model parameters on the device are reallocated to the corresponding far memory device according to the parameter allocation priority of other far memory devices.

[0094] Repeat the above process.

[0095] See Figure 4 , an embodiment of the present application provides a terminal large model parameter collaborative loading system, including:

[0096] System initialization module 310: used to initialize the inference device and the far memory device, so that the inference device can load data from the memory of the far memory device, and preliminarily allocate the parameters of the large model file to the memory of these devices;

[0097] The specific initialization process includes:

[0098] Create a memory file system using tmpfs on each remote memory device and mount a portion of the memory space as a virtual memory disk;

[0099] Use NFS technology on each host to share the tmpfs mounted directory through the network;

[0100] The inference device accesses data on remote memory by mounting directories shared by various connectable remote memory devices through NFS technology;

[0101] According to the aforementioned large model parameter scheduling method based on far memory, all model parameters are allocated and loaded onto each far memory device.

[0102] Far memory device management module 320: used to manage each far memory device and record the connection status, network bandwidth, memory usage, maximum memory, and other information of each far memory device and the inference device in real time;

[0103] Parameter management module 330: used to manage parameters and record the distribution of different parts of large model parameters on different remote memory devices in real time;

[0104] Parameter scheduling decision module 340: used to make parameter scheduling strategy decisions and actual parameter scheduling in real time based on the far memory device information of the far memory device management module and the distribution of different parts of the large model parameters on different far memory devices;

[0105] Parameter loading module 350: used to provide the program with the function of loading parameters from remote memory, and automatically determine the loading method based on the distribution of large model parameters on different devices.

[0106] This module accepts requests from programs to load parameters from far memory, and based on information from the parameter management module, determines the distribution of different parameters on different far memory devices, and converts a request to load parameters from far memory into a request to read different parameters from different far memory devices.

[0107] For example, when the program issues a request to load parameters from pages 1 to 10 from far memory, the module will access the parameter distribution information of the parameter management module after receiving the request. Assume that it is found that the parameters of pages 1 to 3 are stored on far memory device 1, the parameters of pages 4 to 8 are stored on far memory device 2, and the parameters of pages 9 to 10 are stored on far memory device 3. At this time, the module will convert the request to load parameters from pages 1 to 10 from far memory into three requests to load pages 1 to 3 from far memory device 1, load pages 4 to 8 from far memory device 2, and load pages 9 to 10 from far memory device 3, and use different threads to load them.

[0108] See Figure 5 , an embodiment of the present application provides a terminal large model parameter scheduling system, including:

[0109] at least one memory;

[0110] at least one control processor;

[0111] at least one program;

[0112] The program is stored in the memory, and the control processor executes at least one program to implement the terminal large model parameter collaborative loading method based on far memory and the large model parameter scheduling method based on far memory described in the embodiment of the present disclosure.

[0113] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0114] The electronic device according to the embodiment of the present application is described in detail below.

[0115] The control processor 410 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0116] Memory 420, one of which is implemented as non-volatile memory (NVMe) and the other as random access memory (RAM). Memory 420 can store an operating system and other applications. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program code is stored in memory 420 and is called by the control processor 410 to execute the terminal large model parameter collaborative loading method based on far memory and the large model parameter scheduling method based on far memory in the embodiments of this disclosure.

[0117] Input / output interface 430, used to implement information input and output;

[0118] Communication interface 440, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0119] bus 450 , which transmits information between various components of the device (e.g., control processor 410 , memory 420 , input / output interface 430 , and communication interface 440 );

[0120] The control processor 410 , the memory 420 , the input / output interface 430 and the communication interface 440 are connected to each other in communication within the device via a bus 450 .

[0121] An embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned terminal large model parameter collaborative loading method based on far memory and large model parameter scheduling method based on far memory.

[0122] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the control processor, and these remote memories may be connected to the control processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for collaboratively loading terminal large model parameters, characterized in that: include: S110: Determine the file size of the model file to be loaded, allocate a memory area in the inference device based on the file size, and byte-align the starting address of the memory area; S120, obtaining a cache status of the model to be loaded in the memory, and dividing the model to be loaded into a cached area and a non-cached area according to the cache status; S130, obtaining a hard disk loading speed of the inference device and a network status of a remote memory device, and dividing the uncached area into a hard disk loading area and a remote memory loading area; S140. Load the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area, load the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area, and load the model parameters of the far memory loading area from the remote memory device to the corresponding position of the memory area.

2. The terminal large model parameter collaborative loading method according to claim 1 is characterized in that: In S110, allocating a memory area in the inference device based on the file size includes: A memory region size determination model is constructed to determine the size of the memory region to be confirmed. The memory region size determination model satisfies the following relationship: ； Where, Indicates the size of the memory area to be confirmed. Indicates the file size of the model to be loaded, Indicates the number of aligned bytes used by the model to be loaded. Indicates the number of bytes of memory occupied by untyped pointers in programming languages; Use the malloc function to allocate a memory area in the local memory of the inference device with the same size as the memory area to be confirmed; The byte alignment of the starting address of the memory area includes: Construct a byte alignment address determination model, and use the byte alignment address determination model to determine the address after byte alignment. The byte alignment address determination model satisfies the following relationship: ； Where, Indicates the byte-aligned address, Indicates the first address of the memory area, Indicates the number of aligned bytes used by the model to be loaded. Indicates the number of bytes of memory occupied by untyped pointers in programming languages. Represents a bitwise AND operation, Represents bitwise inverse operation; Load the model parameters starting from the byte-aligned address to complete the byte alignment.

3. The terminal large model parameter collaborative loading method according to claim 1 is characterized in that: In S120, the cache status of the model to be loaded in the memory is obtained, and the model to be loaded is divided into a cached area and a non-cached area according to the cache status, including: Get the cache status of each page of the model to be loaded in memory, and mark the pages that are cached as cached, and mark the pages that are not cached as not cached; The pages marked as cached are logically grouped into a cached area, and all pages marked as uncached are logically grouped into an uncached area.

4. The terminal large model parameter collaborative loading method according to claim 1 is characterized in that: In S130, obtaining the hard disk loading speed of the inference device and the network status of the far memory device, and dividing the uncached area into a hard disk loading area and a far memory loading area, includes: Use the test function to obtain the local hard disk loading speed of the inference device and the bandwidth of the network connecting the remote memory device and the inference device, and calculate the ratio of the local hard disk loading speed to the network bandwidth; The total memory size to be occupied by the uncached area is calculated, and the total memory size to be occupied by the uncached area is divided into a hard disk loading area and a remote memory loading area based on the ratio of the local hard disk loading speed to the network bandwidth.

5. The terminal large model parameter collaborative loading method according to claim 1 is characterized in that: In S140, loading the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area includes: Opening a first thread to load the model parameters of the cached area from the memory of the inference device to the corresponding position of the memory area; Loading the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area includes: Opening a second thread to load the model parameters of the hard disk loading area from the hard disk of the inference device to the corresponding position of the memory area; Loading the model parameters of the remote memory loading area from the remote memory device to the corresponding position of the memory area includes: Opening a third thread to load the model parameters of the remote memory loading area from the remote memory device to the corresponding position of the memory area; When the model parameters of the first thread, the second thread, and the third thread are loaded, the parameter loading of the model file to be loaded is completed.

6. A terminal large model parameter scheduling method for dynamically allocating parameters of a model file to be loaded based on the terminal large model parameter collaborative loading method according to any one of claims 1 to 5, characterized in that: include: S210. Obtain memory usage, maximum memory, and network bandwidth information of all far memory devices connected to the inference device, and calculate a parameter allocation priority for each far memory device based on the memory usage, maximum memory, and network bandwidth information; S220: constructing a priority-based parameter allocation algorithm according to the parameter allocation priority, and using the priority-based parameter allocation algorithm to preliminarily allocate the parameters of the model file to be loaded on the inference device to the memory of the remote memory device at a page granularity; S230: Acquire the memory usage of the remote memory device and the time series change information of the connection status with the inference device in real time, and dynamically reallocate the parameters of the model file to be loaded using a priority-based parameter allocation algorithm according to the time series change information.

7. The terminal large model parameter scheduling method according to claim 6, characterized in that: In S210, calculating the parameter allocation priority of each remote memory device according to the memory usage, the maximum memory, and the network bandwidth information includes: The parameter allocation priority calculation formula is used to calculate the parameter allocation priority of each far memory device. The calculation formula satisfies the following relationship: ； Where, Indicates the parameter assignment priority, Indicates the connection status between the far memory device and the inference device. For the connection to exist, If no connection exists, Indicates the maximum memory of the far memory device. represents the network bandwidth between the far memory device and the inference device, Indicates the memory usage of the far memory device; The priority-based parameter allocation algorithm includes: Divide the parameters to be allocated into pages, calculate the number of pages for the parameters to be allocated, and obtain the number of pages occupied by the parameters; A priority-based parameter allocation algorithm is constructed based on the number of pages occupied by the parameters and the parameter allocation priority, and the algorithm satisfies the following relationship: ； Where, Indicates the number of parameter pages that need to be allocated on each far memory device. Indicates the number of pages occupied by the parameter. Indicates the sum of the priorities of all connected far memory devices.

8. The terminal large model parameter scheduling method according to claim 6, characterized in that: In S230, dynamically reallocating the parameters of the model file to be loaded using a priority-based parameter allocation algorithm according to the time series change information includes: When the memory usage of a far memory device exceeds the memory usage threshold, some parameters on the device are reallocated to other far memory devices using a priority-based parameter allocation method. When it is found that a far memory device loses connection, all parameters on the device are reallocated to other far memory devices using a priority-based parameter allocation method.

9. A terminal large model parameter collaborative loading system, used to implement the terminal large model parameter collaborative loading method according to any one of claims 1 to 5, characterized in that: include: System initialization module: used to initialize the inference device and far memory device, enabling the inference device to load data from the memory of the far memory device and preliminarily allocate the parameters of the large model file to the memory of these devices; Far memory device management module: used to manage each far memory device and record the connection status, network bandwidth, memory usage, and maximum memory of each far memory device and inference device in real time; Parameter management module: used to manage parameters and record the distribution of different parts of large model parameters on different remote memory devices in real time; Parameter scheduling decision module: This module is used to make parameter scheduling strategy decisions and actual parameter scheduling in real time based on the far memory device information from the far memory device management module and the distribution of different parts of the large model parameters on different far memory devices. Parameter loading module: used to provide the program with the function of loading parameters from remote memory, and automatically determine the loading method based on the distribution of large model parameters on different devices.

10. A terminal large model parameter scheduling system, characterized in that: comprising at least one control processor and a memory for communicatively connecting with the control processor; The memory stores instructions that can be executed by at least one control processor, and the instructions are executed by at least one control processor to enable the at least one control processor to execute the terminal large model parameter scheduling method according to any one of claims 6 to 8.

Images