Language model retrieval reasoning system and method based on high-speed intercommunication connection

Abstract

The invention discloses a language model retrieval reasoning system and method based on high-speed intercommunication connection, and the system comprises a host, a high-speed intercommunication connection switch, a high-speed intercommunication connection memory pool, and an image processor, the high-speed intercommunication connection memory pool comprises a near memory calculation memory and a double data transmission rate memory, and the high-speed intercommunication connection switch comprises an adaptive partition module; wherein the host is respectively connected with the high-speed intercommunication connection memory pool and the image processor through the high-speed intercommunication connection switch; the host is used for receiving the first cue word, generating a second cue word and storing the second cue word to the double data transmission rate memory through the high-speed intercommunication connection switch; and the host is also used for extracting a second cue word from the double data transmission rate memory through the high-speed intercommunication connection switch for reasoning and determining a task allocation object in a reasoning stage by utilizing a self-adaptive partition module connected to the high-speed intercommunication connection switch, and the allocation object is a near-memory computing memory or an image processor.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a language model retrieval and reasoning system and method based on high-speed interconnection. Background Art

[0002] At present, for the inference application of language models, a multi-host distributed system architecture is usually adopted to divide the application work to multiple hosts for simultaneous processing. Each host is equipped with multiple GPUs (Graphic Processing Units) and connected to their respective GPUs through PCIe (Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) switches to accelerate computing tasks. Among them, GPU video memory is usually used to store model parameters and intermediate calculation results. However, due to the limited capacity of GPU video memory, when the data processed by the GPU exceeds the video memory, part of the computing load needs to be allocated to the host's DDR (Double Data Rate) memory or NVMe (Non-Volatile Memory Express) storage device. At this time, the data needs to be transmitted back to the host first, and then migrated to the host's storage unit (such as DDR or NVMe). The host CPU processes the data in the storage unit to ensure the continuation of the computing task.

[0003] Although existing technologies have made certain progress in implementing reasoning systems for distributed language models, there are still some significant shortcomings: memory capacity and bandwidth cannot meet the needs; large-scale models and computing tasks often require frequent memory migration across hosts, which brings large performance overhead; at the same time, related technology memory resource scheduling usually relies on static configuration or simple load balancing algorithms, lacks intelligence and flexibility, and cannot adjust memory allocation objects in real time according to the dynamic needs of tasks. Summary of the invention

[0004] The present application provides a language model retrieval and reasoning system and method based on high-speed interconnection, so as to at least solve the problems in the related technology of implementing distributed language model reasoning, that is, the memory capacity and bandwidth cannot meet the requirements, the performance overhead caused by continuously migrating data across hosts during computing tasks, and the inability to adjust memory allocation objects in real time according to the dynamic requirements of tasks.

[0005] The present application provides a language model retrieval and reasoning system based on high-speed interconnection, the system comprising: a host, a high-speed interconnection connection switch, a high-speed interconnection connection memory pool, and an image processor, wherein the high-speed interconnection connection memory pool comprises a near-memory computing memory and a double data transfer rate memory, and the high-speed interconnection connection switch comprises an adaptive partitioning module; wherein the host is respectively connected to the high-speed interconnection connection memory pool and the image processor through the high-speed interconnection connection switch; A host, for receiving a first prompt word, starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word, generating a second prompt word, and storing the second prompt word in a double data rate memory through a high-speed intercommunication connection switch; The host is further used to extract the second prompt word from the double data transmission rate memory through the high-speed intercommunication connection switch, and to infer the second prompt word, and to determine the task allocation object in the inference stage by using the adaptive partitioning module in the high-speed intercommunication connection switch connected to it; The adaptive partitioning module is used to obtain the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determine the allocation objects of the step tasks based on the computing power, and allocate the step tasks to the corresponding allocation objects for task calculation, wherein the allocation objects are near-storage computing memory or image processors.

[0006] The present application provides a language model retrieval and reasoning method based on high-speed intercommunication technology, which is applied to a language model retrieval and reasoning system based on high-speed intercommunication connection, and the method includes: receiving a first prompt word, and starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word, and generating a second prompt word; Extracting a second prompt word and inferring the second prompt word; The computing power of all step tasks executed by the second prompt word in the reasoning stage is obtained, and the allocation object of the step task is determined based on the computing power, wherein the allocation object is used to perform task calculation on the step task.

[0007] The present application provides a language model retrieval and reasoning device based on high-speed intercommunication technology, the device comprising: A receiving module, used for receiving a first prompt word, and starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word, and generating a second prompt word; An extraction module, used for extracting a second prompt word and performing reasoning on the second prompt word; The acquisition module is used to obtain the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determine the allocation object of the step task based on the computing power, wherein the allocation object is used to perform task calculation on the step task.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing any of the above-mentioned language model retrieval and reasoning methods based on high-speed intercommunication technology when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, any of the above-mentioned language model retrieval and reasoning methods based on high-speed intercommunication technology is implemented.

[0010] The present application provides a computer program product, including computer instructions, which are used to enable a computer to execute any of the above-mentioned language model retrieval and reasoning methods based on high-speed intercommunication technology.

[0011] Through the present application, the language model retrieval and reasoning system based on high-speed interconnection connection is designed, including: a host, a high-speed interconnection connection switch, a high-speed interconnection connection memory pool, and an image processor. The high-speed interconnection connection memory pool includes a near-storage computing memory and a double data transfer rate memory, and the high-speed interconnection connection switch includes an adaptive partitioning module; wherein the host is respectively connected to the high-speed interconnection connection memory pool and the image processor through the high-speed interconnection connection switch; through this system architecture, through the high-speed interconnection connection memory pool, multiple computing nodes (i.e., the host, the image processor) can share a high-bandwidth, low-latency memory pool, thereby providing higher memory bandwidth, improving the processing efficiency of language model retrieval and reasoning tasks, and supporting the efficient operation of large-scale models and data; at the same time, due to the language model retrieval and reasoning system based on high-speed interconnection connection provided in the embodiment of the present application The processing system uses a host to receive a first prompt word, and based on the first prompt word, starts a retrieval phase, performs vectorization processing and database retrieval on the first prompt word, generates a second prompt word, and stores the second prompt word in a double data transmission rate memory through a high-speed interconnection switch; the host can also extract the second prompt word from the double data transmission rate memory through the high-speed interconnection switch, and infer the second prompt word, and use the adaptive partitioning module in the high-speed interconnection switch connected to it to determine the task allocation object in the inference phase; use the adaptive partitioning module in the high-speed interconnection switch to obtain the computing power of all step tasks executed by the second prompt word in the inference phase, and determine the allocation object of the step task based on the computing power, and allocate the step task to the corresponding allocation object for task calculation, wherein the allocation object is a near storage computing memory or an image processor. In this way, since the generated second prompt word is stored in the double data rate memory through the high-speed interconnection switch, subsequent data interaction does not need to go through the host, and only needs to extract data from the double data rate memory, thereby reducing the need for memory migration and optimizing memory sharing. In particular, the high-speed interconnection memory pool is used to reduce cross-host memory migration, thereby reducing the performance loss caused by memory migration. In addition, by performing task calculation on the computing power generated by each step task in the reasoning stage, the allocation object of the step task is determined based on the computing power, thereby realizing dynamic allocation of resources, optimizing the utilization efficiency of computing resources, and avoiding the problem of affecting the execution efficiency of reasoning tasks due to excessive or insufficient resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1A schematic diagram of a language model retrieval and reasoning system based on high-speed interconnection provided in an embodiment of the present application; Figure 2 A schematic diagram of a memory access path provided in an embodiment of the present application; Figure 3 A flowchart of a language model retrieval and reasoning method based on a high-speed interconnection connection provided in an embodiment of the present application; Figure 4 A structural block diagram of a language model retrieval and reasoning device based on high-speed intercommunication technology provided in an embodiment of the present application; Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0016] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0017] With the rapid development of artificial intelligence and deep learning technologies, large-scale language models and retrieval enhancement generation systems have achieved remarkable results in the field of natural language processing. These applications require efficient computing resources and large-scale memory support to achieve efficient reasoning processes. However, traditional memory architectures and computing resource configurations cannot meet these requirements, limiting the efficiency and scale of model reasoning. Especially for complex large-scale language models and retrieval enhancement generation, traditional memory management technologies can no longer meet their requirements for high bandwidth, low latency, and large-scale memory access. Factors such as memory capacity, bandwidth, and access latency often become performance bottlenecks. Therefore, the Compute Express Link (CXL) protocol and distributed computing architecture were proposed to break through these bottlenecks and achieve more efficient reasoning calculations.

[0018] However, although CXL technology and distributed architecture provide new means for language model reasoning, existing technologies have limitations in the following key aspects: a. Bottlenecks of memory capacity and bandwidth Large-scale language model and retrieval enhancement generation applications usually need to process models and data of hundreds of GB or even larger. Traditional memory architectures cannot provide sufficient memory capacity and bandwidth to support the efficient operation of these applications. Even with the use of graphics processing units (GPUs) for acceleration, the capacity and bandwidth of GPU memory often become bottlenecks that limit performance. Existing technologies mainly rely on frequent data transmission between GPU memory and host memory, which increases the pressure on memory bandwidth and thus affects reasoning efficiency.

[0019] b. Memory migration overhead Large-scale models and complex computing tasks require frequent data migration between different memory areas, such as CPU memory, GPU memory, and PNM (Proximity Computing). However, this memory migration brings significant overhead, especially in memory sharing and data exchange across hosts. Memory migration in existing technologies is usually synchronized, which increases the waiting time of computing tasks and reduces the throughput of the overall system. Inflexible memory management also makes it impossible for the system to fully utilize the maximum efficiency of resources under high-concurrency tasks.

[0020] c. Insufficient computation offloading and resource scheduling Existing computing architectures often rely on fixed resource configurations, such as forcibly offloading computing tasks to GPUs or CPUs, without dynamically adjusting computing resources based on task requirements. This fixed resource scheduling method cannot flexibly select the best computing resources based on the arithmetic intensity, memory access pattern, and other characteristics of the task, resulting in low processing efficiency of computing loads, especially when facing dynamically changing workloads, and cannot achieve optimal computing resource allocation.

[0021] d. Complexity of memory pool management and scheduling The introduction of the CXL protocol provides a new direction for memory pool management, but the existing CXL memory pool technology still has certain deficiencies in dynamic management and scheduling. The current memory pool management lacks sufficient intelligence and automation. The allocation and adjustment of memory resources mainly rely on manual configuration or simple static rules, and fail to give full play to the advantages of CXL technology in memory sharing and dynamic management. Especially in multi-host systems, how to reasonably configure and schedule memory resources to avoid memory waste and excessive competition is still a problem that needs to be solved.

[0022] In order to solve the above problems, according to an embodiment of the present application, a language model retrieval and reasoning system embodiment based on high-speed interconnection is provided, such as Figure 1 As shown, Figure 1 The system comprises: a host, a high-speed interconnection connection switch, a high-speed interconnection connection memory pool, and an image processor, wherein the high-speed interconnection connection memory pool includes a near-memory computing memory and a double data transfer rate memory, and the high-speed interconnection connection switch includes an adaptive partitioning module; wherein the host is respectively connected to the high-speed interconnection connection memory pool and the image processor through the high-speed interconnection connection switch; A host, for receiving a first prompt word, starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word, generating a second prompt word, and storing the second prompt word in a double data rate memory through a high-speed intercommunication connection switch; The host is further used to extract the second prompt word from the double data transmission rate memory through the high-speed intercommunication connection switch, and to infer the second prompt word, and to determine the task allocation object in the inference stage by using the adaptive partitioning module in the high-speed intercommunication connection switch connected to it; The adaptive partitioning module is used to obtain the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determine the allocation objects of the step tasks based on the computing power, and allocate the step tasks to the corresponding allocation objects for task calculation, wherein the allocation objects are near-storage computing memory or image processors.

[0023] Optionally, the language model retrieval and reasoning system based on high-speed interconnection connection proposed in the embodiment of the present application, Figure 1In the figure, the system consists of the following components: 4 host servers (host 1, host 2, host 3, host 4), 4 high-speed interconnection switches (i.e., CXL switches in the figure), 6 image processors (i.e., GPUs in the figure), and a high-speed interconnection memory pool (i.e., CXL memory pool in the figure). The high-speed interconnection memory pool is equipped with near-memory computing memory (i.e., PNM memory in the figure) near the high-speed interconnection switch end (short physical link), and double data transfer rate memory (i.e., DDR memory in the figure) far from the high-speed interconnection switch end (long physical link). In addition, the high-speed interconnection switch includes an adaptive partitioning module, through which resource allocation is realized.

[0024] It should be noted that the language model retrieval and reasoning system based on high-speed interconnection connection in the embodiment of the present application is not limited to Figure 1 There are 4 hosts, 4 high-speed interconnection switches, and 6 image processors shown in the figure. The number of hosts, high-speed interconnection switches, and image processors can be flexibly changed.

[0025] Further, from Figure 1 It can be concluded that the host is connected to the high-speed interconnection memory pool and the image processor respectively through the high-speed interconnection switch, and a double data transfer rate memory is provided in the high-speed interconnection memory pool for data storage.

[0026] The language model retrieval and reasoning task is completed by four hosts. The language model retrieval and reasoning task is divided into the RAG (Retrieval-augmented Generation) stage, the Prefill stage, and the Decode stage.

[0027] RAG stage: The user can input the first prompt word from any host, and multiple hosts can input the first prompt word at the same time. The host starts the retrieval stage based on the output first prompt word, performs vectorization processing and database retrieval on the first prompt word, generates the second prompt word, and stores the second prompt word in the double data transmission rate memory through the high-speed interconnection switch.

[0028] The Prefill stage of inference is handled by two hosts (for example, host 1 and host 2). Through a high-speed interconnection switch, the second prompt word is extracted from the double data transmission rate memory, and the second prompt word is inferred: embedding and encoding, layer normalization, QKV mapping, QK matrix calculation, Softmax, SV matrix calculation, output projection, residual connection, layer normalization, fully connected layer calculation, activation function calculation, fully connected layer calculation, residual connection, linear layer, Softmax function, record KV address, first token (i.e., letter word) value. The host collects the second prompt word once every unit time (for example, one second) and uses continuous batch processing to perform model inference.

[0029] The Decode stage of inference is performed by the other two hosts (for example, host 3 and host 4). The Decode task process is similar to that of Prefill. Hosts 3 and 4 obtain the KV value and the address of the first token value from hosts 1 and 2 through RDMA (remote direct memory access) and decode them. After that, they go through the same inference process as the Prefill stage to finally obtain the inference result.

[0030] In the Prefill stage and the Decode stage, the following steps will be performed: embedding and encoding, layer normalization, QKV mapping, QK matrix calculation, Softmax, SV matrix calculation, output projection, residual connection, layer normalization, fully connected layer calculation, activation function calculation, fully connected layer calculation, residual connection, linear layer, Softmax function. The computing power of these step tasks is obtained by using the adaptive partitioning module in the high-speed interconnection switch connected to it, and the current computing tasks are allocated to the image processor or the near-storage computing memory based on the computing power. Operations with lower arithmetic intensity will generate higher data transmission overhead, so the task steps with lower arithmetic intensity are allocated to the near-storage computing memory, and the task steps with higher arithmetic intensity are performed by the image processor.

[0031] Preferably, when a step task needs to be calculated in a near-storage computing memory, the calculation is preferentially placed in a near-storage computing memory that is close to the host server to reduce latency.

[0032] In the embodiments of the present application, by designing a language based on high-speed interconnection, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0034] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] With the rapid development of artificial intelligence and deep learning technologies, large-scale language models and retrieval enhancement generation systems have achieved remarkable results in the field of natural language processing. These applications require efficient computing resources and large-scale memory support to achieve efficient reasoning processes. However, traditional memory architectures and computing resource configurations cannot meet these requirements, limiting the efficiency and scale of model reasoning. Especially for complex large-scale language models and retrieval enhancement generation, traditional memory management technologies can no longer meet their requirements for high bandwidth, low latency, and large-scale memory access. Factors such as memory capacity, bandwidth, and access latency often become performance bottlenecks. Therefore, the Compute Express Link (CXL) protocol and distributed computing architecture were proposed to break through these bottlenecks and achieve more efficient reasoning calculations.

[0036] However, although CXL technology and distributed architecture provide new means for language model reasoning, existing technologies have limitations in the following key aspects: a. Bottlenecks of memory capacity and bandwidth Large-scale language model and retrieval enhancement generation applications usually need to process models and data of hundreds of GB or even larger. Traditional memory architectures cannot provide sufficient memory capacity and bandwidth to support the efficient operation of these applications. Even with the use of graphics processing units (GPUs) for acceleration, the capacity and bandwidth of GPU memory often become bottlenecks that limit performance. Existing technologies mainly rely on frequent data transmission between GPU memory and host memory, which increases the pressure on memory bandwidth and thus affects reasoning efficiency.

[0037] b. Memory migration overhead Large-scale models and complex computing tasks require frequent data migration between different memory areas, such as CPU memory, GPU memory, and PNM (Proximity Computing). However, this memory migration brings significant overhead, especially in memory sharing and data exchange across hosts. Memory migration in existing technologies is usually synchronized, which increases the waiting time of computing tasks and reduces the throughput of the overall system. Inflexible memory management also makes it impossible for the system to fully utilize the maximum efficiency of resources under high-concurrency tasks.

[0038] c. Insufficient computation offloading and resource scheduling Existing computing architectures often rely on fixed resource configurations, such as forcibly offloading computing tasks to GPUs or CPUs, without dynamically adjusting computing resources based on task requirements. This fixed resource scheduling method cannot flexibly select the best computing resources based on the arithmetic intensity, memory access pattern, and other characteristics of the task, resulting in low processing efficiency of computing loads, especially when facing dynamically changing workloads, and cannot achieve optimal computing resource allocation.

[0039] d. Complexity of memory pool management and scheduling The introduction of the CXL protocol provides a new direction for memory pool management, but the existing CXL memory pool technology still has certain deficiencies in dynamic management and scheduling. The current memory pool management lacks sufficient intelligence and automation. The allocation and adjustment of memory resources mainly rely on manual configuration or simple static rules, and fail to give full play to the advantages of CXL technology in memory sharing and dynamic management. Especially in multi-host systems, how to reasonably configure and schedule memory resources to avoid memory waste and excessive competition is still a problem that needs to be solved.

[0040] In order to solve the above problems, according to an embodiment of the present application, a language model retrieval and reasoning system embodiment based on high-speed interconnection is provided, such as Figure 1 As shown, Figure 1 The system comprises: a host, a high-speed interconnection connection switch, a high-speed interconnection connection memory pool, and an image processor. The high-speed interconnection connection memory pool includes a near-memory computing memory and a double data transfer rate memory. The high-speed interconnection connection switch includes an adaptive partitioning module. The host is connected to the high-speed interconnection connection memory pool and the image processor respectively through the high-speed interconnection connection switch. A host, for receiving a first prompt word, starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word, generating a second prompt word, and storing the second prompt word in a double data rate memory through a high-speed intercommunication connection switch; The host is further used to extract the second prompt word from the double data transmission rate memory through the high-speed intercommunication connection switch, and to infer the second prompt word, and to determine the task allocation object in the inference stage by using the adaptive partitioning module in the high-speed intercommunication connection switch connected to it; The adaptive partitioning module is used to obtain the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determine the allocation objects of the step tasks based on the computing power, and allocate the step tasks to the corresponding allocation objects for task calculation, wherein the allocation objects are near-storage computing memory or image processors.

[0041] Optionally, the language model retrieval and reasoning system based on high-speed interconnection connection proposed in the embodiment of the present application, Figure 1 In the figure, the system consists of the following components: 4 host servers (host 1, host 2, host 3, host 4), 4 high-speed interconnection switches (i.e., CXL switches in the figure), 6 image processors (i.e., GPUs in the figure), and a high-speed interconnection memory pool (i.e., CXL memory pool in the figure). The high-speed interconnection memory pool is equipped with near-memory computing memory (i.e., PNM memory in the figure) near the high-speed interconnection switch end (short physical link), and double data transfer rate memory (i.e., DDR memory in the figure) far from the high-speed interconnection switch end (long physical link). In addition, the high-speed interconnection switch includes an adaptive partitioning module, through which resource allocation is realized.

[0042] It should be noted that the language model retrieval and reasoning system based on high-speed interconnection connection in the embodiment of the present application is not limited to Figure 1 There are 4 hosts, 4 high-speed interconnection switches, and 6 image processors shown in the figure. The number of hosts, high-speed interconnection switches, and image processors can be flexibly changed.

[0043] Further, from Figure 1 It can be concluded that the host is connected to the high-speed interconnection memory pool and the image processor respectively through the high-speed interconnection switch, and a double data transfer rate memory is provided in the high-speed interconnection memory pool for data storage.

[0044] The language model retrieval and reasoning task is completed by four hosts. The language model retrieval and reasoning task is divided into the RAG (Retrieval-augmented Generation) stage, the Prefill stage, and the Decode stage.

[0045] RAG stage: The user can input the first prompt word from any host, and multiple hosts can input the first prompt word at the same time. The host starts the retrieval stage based on the output first prompt word, performs vectorization processing and database retrieval on the first prompt word, generates the second prompt word, and stores the second prompt word in the double data transmission rate memory through the high-speed interconnection switch.

[0046] The Prefill stage of inference is handled by two hosts (for example, host 1 and host 2). Through a high-speed interconnection switch, the second prompt word is extracted from the double data transmission rate memory, and the second prompt word is inferred: embedding and encoding, layer normalization, QKV mapping, QK matrix calculation, Softmax, SV matrix calculation, output projection, residual connection, layer normalization, fully connected layer calculation, activation function calculation, fully connected layer calculation, residual connection, linear layer, Softmax function, record KV address, first token (i.e., letter word) value. The host collects the second prompt word once every unit time (for example, one second) and uses continuous batch processing to perform model inference.

[0047] The Decode stage of inference is performed by the other two hosts (for example, host 3 and host 4). The Decode task process is similar to that of Prefill. Hosts 3 and 4 obtain the KV value and the address of the first token value from hosts 1 and 2 through RDMA (remote direct memory access) and decode them. After that, they go through the same inference process as the Prefill stage to finally obtain the inference result.

[0048] In the Prefill stage and the Decode stage, the following steps will be performed: embedding and encoding, layer normalization, QKV mapping, QK matrix calculation, Softmax, SV matrix calculation, output projection, residual connection, layer normalization, fully connected layer calculation, activation function calculation, fully connected layer calculation, residual connection, linear layer, Softmax function. The computing power of these step tasks is obtained by using the adaptive partitioning module in the high-speed interconnection switch connected to it, and the current computing tasks are allocated to the image processor or the near-storage computing memory based on the computing power. Operations with lower arithmetic intensity will generate higher data transmission overhead, so the task steps with lower arithmetic intensity are allocated to the near-storage computing memory, and the task steps with higher arithmetic intensity are performed by the image processor.

[0049] Preferably, when a step task needs to be calculated in a near-storage computing memory, the calculation is preferentially placed in a near-storage computing memory that is close to the host server to reduce latency.

[0050] In an embodiment of the present application, a language model retrieval and reasoning system based on high-speed interconnection is designed. The system includes: a host, a high-speed interconnection switch, a high-speed interconnection memory pool, and an image processor. The high-speed interconnection memory pool includes a near-memory computing memory and a double data transfer rate memory, and the high-speed interconnection switch includes an adaptive partitioning module; wherein the host is connected to the high-speed interconnection memory pool and the image processor respectively through the high-speed interconnection switch; through this system architecture, through the high-speed interconnection memory pool, multiple computing nodes (i.e., the host and the image processor) can share a high-bandwidth, low-latency memory pool, thereby providing higher memory bandwidth, improving the processing efficiency of language model retrieval and reasoning tasks, and supporting the efficient operation of large-scale models and data; at the same time, due to the language model based on high-speed interconnection provided in the embodiment of the present application The retrieval reasoning system uses a host to receive a first prompt word, and based on the first prompt word, starts a retrieval phase, performs vectorization processing and database retrieval on the first prompt word, generates a second prompt word, and stores the second prompt word in a double data transmission rate memory through a high-speed interconnection switch; the host can also extract the second prompt word from the double data transmission rate memory through the high-speed interconnection switch, and perform reasoning on the second prompt word, and use the adaptive partitioning module in the high-speed interconnection switch to determine the task allocation object in the reasoning phase; use the adaptive partitioning module in the high-speed interconnection switch to obtain the computing power of all step tasks executed by the second prompt word in the reasoning phase, and determine the allocation object of the step task based on the computing power, and allocate the step task to the corresponding allocation object for task calculation, wherein the allocation object is a near storage computing memory or an image processor. In this way, since the generated second prompt word is stored in the double data rate memory through the high-speed interconnection switch, subsequent data interaction does not need to go through the host, and only needs to extract data from the double data rate memory, thereby reducing the need for memory migration and optimizing memory sharing. In particular, the high-speed interconnection memory pool is used to reduce cross-host memory migration, thereby reducing the performance loss caused by memory migration. In addition, by performing task calculation on the computing power generated by each step task in the reasoning stage, the allocation object of the step task is determined based on the computing power, thereby realizing dynamic allocation of resources, optimizing the utilization efficiency of computing resources, and avoiding the problem of affecting the execution efficiency of reasoning tasks due to excessive or insufficient resources.

[0051] In some optional embodiments, the double data transfer rate memory includes a non-fixed memory pool and a fixed memory pool, wherein the non-fixed memory pool is used to store intermediate parameters generated in the reasoning stage, and the fixed memory pool is used to store fixed parameters in the retrieval stage and the reasoning stage, as well as storage addresses of the intermediate parameters and the fixed parameters; The high-speed interconnected memory pool also includes a memory migration module; A memory migration module is used to store the first prompt word and the second prompt word in a non-fixed memory pool through a high-speed interconnection switch, migrate the vector value obtained after the allocation object calculates the step task from the local memory to the non-fixed memory pool through the high-speed interconnection switch, store the intermediate parameters in the non-fixed memory pool through the high-speed interconnection switch, and store the fixed parameters and storage addresses in the fixed memory pool through the high-speed interconnection switch.

[0052] Optionally, in Figure 1 In the figure, the double data transfer rate memory includes a non-fixed memory pool and a fixed memory pool. It should be noted that the non-fixed memory pool and the fixed memory pool here do not have a fixed position. They only load fixed parameters such as the weight of the language model, the fixed parameters of the retrieval stage and the inference stage into the fixed memory pool, and the intermediate parameters generated in the inference stage, such as the KV value of the Prefill stage, the first token value, and the inference result obtained at the end of the Decode stage inference, into the non-fixed memory pool, partitioning the storage of the parameters.

[0053] Of course, the non-fixed memory pool also includes variable parameters such as the language model program, the RAG program, the first prompt word, the second prompt word, etc. In addition, the fixed memory pool will also have the storage addresses of all parameters in the non-fixed memory pool and the fixed memory pool.

[0054] After the double data rate memory is set up with a non-fixed memory pool and a fixed memory pool, based on Figure 1 The memory migration module in the high-speed interconnection memory pool stores the first prompt word and the second prompt word in the non-fixed memory pool through the high-speed interconnection switch, migrates the vector value obtained after the allocation object calculates the step task from the PNM memory or the local memory of the GPU to the non-fixed memory pool through the high-speed interconnection switch, stores the intermediate parameters in the non-fixed memory pool through the high-speed interconnection switch, and stores the fixed parameters and storage addresses in the fixed memory pool through the high-speed interconnection switch.

[0055] In an embodiment of the present application, a memory migration module is utilized to directly store data into a double data transfer rate memory through a high-speed interconnection switch without passing through a host, thereby reducing data migration and replication across hosts, greatly reducing the performance loss caused by memory migration, and improving the overall reasoning efficiency of the system.

[0056] In some optional implementations, the high-speed interconnect connection memory pool further includes a high-speed interconnect connection protocol module, and the high-speed interconnect connection memory pool further includes a memory configuration module; A memory configuration module, used for initiating an access request from a host or an image processor to a non-fixed memory pool and / or a fixed memory pool in a double data transfer rate memory through a high-speed intercommunication connection protocol module; The memory configuration module is also used to initiate an access request from the near storage computing memory to the non-fixed memory pool and / or the fixed memory pool in the double data transfer rate memory through the high-speed interconnection protocol module.

[0057] Alternatively, if Figure 1 , the high-speed interconnection memory pool also includes a high-speed interconnection protocol module (i.e. Figure 1 The high-speed interconnection memory pool also includes a memory configuration module. The memory configuration module is used to initiate access requests to both the non-fixed memory pool and the fixed memory pool in the double data transfer rate memory through the high-speed interconnection protocol module, and can also initiate access requests to either the non-fixed memory pool or the fixed memory pool. Figure 2 In the data transmission channel 1, the host or image processor communicates with the high-speed interconnection connection protocol module through the data transmission channel 1, and then initiates an access request to the non-fixed memory pool and / or the fixed memory pool in the double data transfer rate memory through the high-speed interconnection protocol module.

[0058] The memory configuration module is also used to initiate access requests to both the non-fixed memory pool and the fixed memory pool in the double data transfer rate memory through the high-speed interconnection protocol module, and can also initiate access requests to either the non-fixed memory pool or the fixed memory pool. Figure 2 The near memory computing memory first communicates with the high-speed interconnection connection protocol module through the data transmission channel 2, and then the memory configuration module connects the memory access request to the non-fixed memory pool and / or the fixed memory pool in the double data transmission rate memory through the data transmission channel 3.

[0059] Furthermore, the number of data transmission channels 1, data transmission channel 2, and data transmission channel 3 connected to the high-speed interconnection connection memory pool by the high-speed interconnection connection protocol module is limited. For example, there is only one data transmission channel 1, but the memory configuration module based on the memory pool can enable the host and the image processor to access the double data transmission rate memory of any physical address; similarly, there are only one data transmission channel 2 and one data transmission channel 3, but the memory configuration module based on the memory pool can enable the near memory computing memory to access the double data transmission rate memory of any physical address.

[0060] In an embodiment of the present application, the memory configuration module further supports the dynamic connection of the host, image processor and near-memory computing memory with the double data transfer rate memory of any physical address in the high-speed interconnected memory pool, thereby improving the utilization efficiency of memory resources, avoiding resource waste and bottlenecks, and optimizing system performance.

[0061] In this embodiment, a language model retrieval and reasoning method based on high-speed intercommunication technology is provided, which can be used in the above-mentioned language model retrieval and reasoning system based on high-speed intercommunication connection, such as Figure 3 As shown, the process includes the following steps: Step S301: receiving a first prompt word, and starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word, and generating a second prompt word.

[0062] Optionally, the host (which may be multiple hosts, such as host 1, host 2, host 3, host 4) contained in the language model retrieval reasoning system based on high-speed interconnection will receive the first prompt word input by the user through the input interface, and then store the first prompt word in the high-speed interconnection memory pool of the above-mentioned system, and then send a message to the host to instruct the start of the retrieval enhancement generation (ie, RAG) stage.

[0063] When the host receives the "start RAG phase" message, it will start the RAG task, perform vectorization processing and database search on the first prompt word, and then generate the second prompt word.

[0064] The second prompt word can then be put into a non-fixed memory pool of double data transfer rate memory in the language model retrieval and reasoning system based on a high-speed interconnection connection.

[0065] Step S302: extract the second prompt word and perform reasoning on the second prompt word.

[0066] Optionally, the host can then collect the second prompt word and infer the second prompt word. The inference stage includes two parts, the Prefill stage and the Decode stage. Furthermore, in the embodiment of the present application, host 1 and host 2 can be responsible for the Prefill stage, and host 1 and host 2 can collect the second prompt word from the non-fixed memory pool once per unit time (such as one second). The continuous batch processing method is adopted, that is, multiple second prompt words are collected and processed together to improve the inference efficiency.

[0067] Then the reasoning begins: embedding and encoding: embedding the collected second prompt words, converting the text into an encoding form suitable for model processing, and further extracting the semantic features of the text.

[0068] Layer normalization: Normalize the data to make the data distribution more stable, which helps the training and reasoning of the model and improves the convergence speed and performance of the model.

[0069] QKV mapping: Mapping the processed data into Query (query vector), Key (key vector) and Value (value vector), which is an important step in many attention-based models (such as Transformer).

[0070] QK matrix calculation: Calculate the matrix product of Query and Key to measure the similarity between different elements and thus determine the importance of each element in the current context.

[0071] Softmax function: Apply the Softmax function to the result of the QK matrix calculation to convert it into a probability distribution in order to determine the weight of each element.

[0072] SV matrix calculation: Calculate the matrix product of the Softmax result and the Value to obtain the final weighted result.

[0073] Output projection: Perform a projection transformation on the result of the SV matrix calculation to convert it into a suitable dimension and form.

[0074] Residual connection: Introducing residual connection to add the original input to the output after the above processing helps solve the gradient vanishing and gradient exploding problems in deep neural networks and improve the training effect and generalization ability of the model.

[0075] Layer Normalization (again): Perform layer normalization again to further stabilize the data distribution.

[0076] Fully connected layer calculation: The data is linearly transformed through the fully connected layer to extract more advanced feature representations.

[0077] Activation function calculation: Apply the activation function to perform nonlinear transformation on the output of the fully connected layer to increase the expressive power of the model.

[0078] Fully connected layer calculation (again): Perform another fully connected layer calculation to further process the data.

[0079] Residual connections (again): Residual connections are used again to enhance the performance of the model.

[0080] Linear layer: The data is subjected to a final linear transformation through a linear layer.

[0081] Softmax (again): Apply the Softmax function to get the final probability distribution used to generate prediction results.

[0082] Record the KV address and the first token value: After the inference is completed, the KV value (i.e., Key value and Value value) generated during the calculation process and the first token value are stored in the non-fixed memory pool for use in the subsequent Decode stage.

[0083] Host 3 and host 4 are responsible for the Decode stage. Host 3 and host 4 will obtain the KV value and the address of the first token value from host 1 and host 2 through RDMA and perform decoding operations to generate the final inference result. After the inference is completed, the generated inference result is stored in the non-fixed memory pool to complete the entire inference process.

[0084] Step S303, obtaining the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determining the allocation object of the step task based on the computing power, wherein the allocation object is used to perform task calculation on the step task.

[0085] Optionally, the language model retrieval reasoning system based on high-speed interconnection performs computing power calculations on the step tasks (such as embedding and encoding, layer normalization, etc.) involved in the above-mentioned Prefill stage and Decode stage, and determines the allocation object of the step tasks based on the obtained computing power, so that the allocation object performs task calculations on the corresponding step tasks. Among them, the step tasks with higher arithmetic intensity are calculated by the image processor, and the step tasks with higher arithmetic intensity are calculated by the near storage computing memory.

[0086] In addition, when computing step tasks with high arithmetic intensity in the near-storage computing memory, the computation is preferentially placed in the near-storage computing memory that is close to the host server to reduce latency.

[0087] The embodiment of the present application performs task calculation on the computing power generated by each step task in the reasoning stage, determines the allocation object of the step task based on the computing power, optimizes the utilization efficiency of computing resources, and avoids the problem of affecting the execution efficiency of reasoning tasks due to excessive or insufficient resources.

[0088] In some optional implementations, receiving a first prompt word and starting a search phase based on the first prompt word includes: When the first prompt word is received, obtaining a first storage address corresponding to the first prompt word; Determining a collection state of the first prompt word based on the first storage address; When the collection status meets the preset conditions, the retrieval phase starts.

[0089] Optionally, upon receiving the first prompt word, the host starts the RAG phase and obtains a first storage address corresponding to the first prompt word.

[0090] Multiple hosts can jointly perform the RAG task. Each host determines the collection status of the first prompt word according to the first storage address, and then decides whether to start the retrieval stage based on the specific situation of the collection status, such as determining the collection status by setting a flag bit, and setting a specific flag bit in the relevant data structure or memory area storing the first prompt word to indicate the collection status. For example, a binary bit is used to indicate whether the first prompt word has been collected, 0 indicates not collected, and 1 indicates collected. The host determines the collection status of the first prompt word by reading the value of the flag bit.

[0091] The acquisition status can also be determined by comparing the timestamps, and the timestamp information is recorded for the storage address of each first prompt word, including the writing time of the first prompt word and the expected acquisition completion time. The host determines the acquisition status of the first prompt word by comparing the current time with the timestamp. For example, if the current time exceeds the expected acquisition completion time, and the first prompt word is still not marked as collected, it may indicate that an abnormality has occurred in the acquisition; or by comparing the writing time and the current time, it is determined whether the set acquisition interval time has been reached to determine whether the acquisition operation can be performed, and the acquisition status of the first prompt word is obtained.

[0092] The acquisition status can also be determined by a message queue or event mechanism, and the information of the acquisition status of the first prompt word can be transmitted by using the message queue or event mechanism. When the first prompt word is stored in the first storage address, a corresponding message is sent or an event is triggered to inform the host that the status of the first prompt word has been updated. The host obtains the acquisition status of the first prompt word by monitoring the message queue or event.

[0093] In some optional implementations, determining the collection state of the first prompt word based on the first storage address includes: Acquire a first read state of a status register of a first storage address and a second read state of a status register of a storage address next to the first storage address; Based on the first reading state and the second reading state, a collection state of the first prompt word is determined.

[0094] Optionally, when the host performs a RAG task, it will first check whether the status register of the first storage address is to be read. If it is to be read, it will read the first prompt word and change the status register to read. If the status register is read, the host will first check whether the status register of the next storage address is to be read. If the status register is finished, it means that all the first prompt words of the inference have been collected at this time, and the retrieval stage is started. For details, please refer to Table 1.

[0095] As shown in Table 1

[0096] In the embodiment of the present application, the host determines the first prompt word collection result by checking the status of the status register, without the need to perform complex addressing and data retrieval operations like accessing a file system or database. The collection status of the first prompt word can be obtained at an extremely fast speed, thereby improving the operating efficiency of the system.

[0097] In some optional implementations, vectorization processing and database retrieval are performed on the first prompt word to generate the second prompt word, including: Convert the first prompt word into a first vector; Input the first vector into a database for retrieval to obtain a second vector associated with the first vector; The first vector and the second vector are integrated to obtain a second prompt word.

[0098] Optionally, the host can read the previously stored first prompt word from the non-fixed memory pool of the double data rate memory. This is the starting step of the RAG task, obtaining input data for subsequent processing.

[0099] Convert the first prompt word read into a vector form. In natural language processing, a specific algorithm or model (such as a word embedding model) is usually used to convert text into a numerical vector so that the computer can process and analyze the text information and prepare for subsequent retrieval in the vector database.

[0100] The vectorized first prompt word is used to search in the vector database to find relevant information. The vector database can quickly find other vectors similar to the input vector, which may correspond to related text or knowledge, thereby obtaining supplementary information related to the first prompt word.

[0101] The retrieved relevant information is integrated with the first prompt word, and the context is reconstructed to generate the second prompt information.

[0102] In some optional implementations, obtaining the computing power of all step tasks executed by the second prompt word in the reasoning stage includes: Get the computation amount and memory access amount corresponding to the step task; Based on the ratio of computational effort to memory access effort, the computing power of the step task is obtained.

[0103] Optionally, some non-matrix operations (such as embedding and encoding, layer normalization, activation function calculation, and residual connection) exhibit low arithmetic intensity. The embodiment of the present application will directly identify the computing power of embedding and encoding, layer normalization, activation function calculation, and residual connection as low, and allocate them to the PNM memory for calculation. As for other step tasks in the reasoning stage, when calculating the computing power, it is necessary to obtain the corresponding computing amount and memory access amount, and then obtain the computing power of the step task based on the ratio of the computing amount and the memory access amount.

[0104] It should be noted that the computational effort and memory access in a step task are obtained by integrating the batch size, sequence length, and hidden layer size in the step task.

[0105] For example: The amount of calculation for QKV mapping in Prefill = 3×2×B×L×S 2 ; Memory access amount of QKV mapping in Prefill = B×L×S+3×S 2 . Where B is the batch size, L is the input sequence length, and S is the hidden layer size.

[0106] Therefore, the calculation of the computing power of the step task is as follows: QKV mapping in Prefill: I=3×2×B×L×S 2 / (B×L×S+3×S 2 ) QK matrix calculation or SV matrix calculation in Prefill: I=2×B×L 2 ×S / (2×B×L×S) Output projection in Prefill: I=2×B×L×S 2 / (B×L×S+S 2 ) Calculation of the fully connected layer in Prefill: I=4×B×L×S 2 / (B×L×S+4×S 2 ) QKV mapping in Decode: I=3×2×B×L×S 2 / (B×S+3×S 2 ) QK matrix calculation or SV matrix calculation in Decode: I = 2 × B × L 2 ×S / (2×B×S) Output projection in Decode: I=2×B×L×S 2 / (1+B×S+S 2 ) Fully connected layer calculation in Decode: I=4×B×L×S 2 / (B×S+4×S 2 ) In some optional implementations, determining the allocation object of the step task based on the computing power includes: When the computing power is less than or equal to the preset value, the step task is allocated to a first allocation object, wherein the first allocation object is used to process tasks with arithmetic intensity of the first intensity; When the computing power is greater than a preset value, the step task is allocated to a second allocation object, wherein the second allocation object is used to process tasks with arithmetic intensity of the second intensity.

[0107] Optionally, when the arithmetic intensity of the step task satisfies the following formula, the task will be assigned to the PNM memory closer to the host for calculation, that is, the PNM memory is used to process tasks with arithmetic intensity of the first intensity (lower intensity); if the formula is not satisfied, the task will be assigned to the GPU for calculation, that is, the GPU is used to process tasks with arithmetic intensity of the second intensity (higher intensity).

[0108] Formula: I≤d×(1 / BW cxl −1 / BW pnm ) / (1 / TH pnm −1 / TH gpu ) Where, d×(1 / BW cxl −1 / BW pnm ) / (1 / TH pnm −1 / TH gpu ) is the default value, d is the size of the operand data type (in bytes), BW cxl With BW pnm They represent the bandwidth of CXL and the memory bandwidth of PNM (in GB / s), TH pnm With TH gpu Represent the computational throughput of PNM memory and GPU for a given data type, respectively.

[0109] The embodiment of the present application can determine the allocation target (GPU or PNM memory) in real time according to the arithmetic intensity of the step task. This dynamic resource allocation mechanism ensures that tasks with low arithmetic intensity are offloaded to the PNM memory closer to the host, reducing memory read and write latency, while tasks with higher arithmetic intensity are handled by the GPU. By optimizing the task offloading strategy, the advantages of different computing resources are fully utilized, thereby improving computing efficiency and reducing idle resources.

[0110] In some optional embodiments, the method further comprises: Acquire a data log, wherein the data log includes a first storage address of a first prompt word, a second storage address of a second prompt word, a third storage address of an intermediate parameter obtained in a reasoning phase, and a fourth storage address of a fixed parameter in a retrieval phase and a reasoning phase; Acquire an adjustment operation performed on the data stored in the first storage address, the second storage address, the third storage address, and the fourth storage address, and obtain the adjusted data and the adjusted first storage address, the adjusted second storage address, the adjusted third storage address, and the adjusted fourth storage address; Based on the adjusted data and the adjusted first storage address, the adjusted second storage address, the adjusted third storage address, and the adjusted fourth storage address, the data log is updated, and the data is shared using the updated data log.

[0111] Optionally, after enabling the inference application, the fixed memory pool will retain the first storage address of the first prompt word stored in the non-fixed memory pool, the second storage address of the second prompt word, the third storage address of the intermediate parameters obtained in the inference phase, and the fourth storage address of the fixed parameters of the retrieval phase and the inference phase stored in the fixed memory pool. These data are called data logs and are jointly maintained by the four hosts and the high-speed interconnected memory pool.

[0112] When the host and the high-speed interconnection memory pool perform adjustment operations such as adding, deleting, and modifying the data stored in the non-fixed memory pool and the fixed memory pool, after the operation is completed, the modification method, data type, and adjusted data and the adjusted first storage address, adjusted second storage address, adjusted third storage address, and adjusted fourth storage address will be stored in the data log to update the data log. Other hosts can access the task output of other hosts by obtaining the data log, thereby realizing data sharing.

[0113] In addition, since the KV value generated by the GPU or PNM memory of the host responsible for the Prefill stage occupies too much memory, it can be directly transmitted to the non-fixed memory pool of the high-speed interconnection memory pool through the high-speed interconnection switch, and the data log is updated. When the KV value of the data log is added, the host responsible for the Decode stage can obtain the KV value and token position through the data log to start the Decode stage task.

[0114] The embodiment of the present application ensures real-time sharing and access of data by migrating the intermediate parameters (such as KV values) generated during the reasoning process to a non-fixed memory pool in real time, and updating the data log according to the progress of the reasoning task, thereby achieving efficient collaboration between multiple hosts and avoiding frequent data transmission between multiple hosts, thereby effectively reducing the data transmission delay in distributed computing and accelerating the overall reasoning process.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0116] In this embodiment, a language model retrieval and reasoning device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0117] This embodiment provides a language model retrieval and reasoning device based on high-speed intercommunication technology, such as Figure 4 , the device comprises: The receiving module 401 is used to receive a first prompt word, and start a search phase based on the first prompt word, perform vectorization processing and database search on the first prompt word, and generate a second prompt word; An extraction module 402, used to extract a second prompt word and perform reasoning on the second prompt word; The acquisition module 403 is used to acquire the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determine the allocation object of the step task based on the computing power, wherein the allocation object is used to perform task calculation on the step task.

[0118] In some optional implementations, the receiving module 401 is used to obtain a first storage address corresponding to the first prompt word when the first prompt word is received; determine a collection state of the first prompt word based on the first storage address; and start a retrieval phase when the collection state meets a preset condition.

[0119] In some optional implementations, the receiving module 401 is further used to obtain a first read state of a status register of a first storage address and a second read state of a status register of a storage address next to the first storage address; and determine a collection state of the first prompt word based on the first read state and the second read state.

[0120] In some optional implementations, the receiving module 401 is further configured to determine that the collection state is that the first prompt word collection is completed and start the retrieval phase when the first reading state is read and the second reading state is read.

[0121] In some optional implementations, the receiving module 401 is further used to convert the first prompt word into a first vector; input the first vector into a database for retrieval to obtain a second vector associated with the first vector; and integrate the first vector and the second vector to obtain a second prompt word.

[0122] In some optional implementations, the acquisition module 403 is used to obtain the computational load and memory access load corresponding to the step task; based on the ratio of the computational load and the memory access load, the computing power of the step task is obtained.

[0123] In some optional implementations, the acquisition module 403 is also used to obtain the batch size, sequence length and hidden layer size in the step task; based on the batch size, sequence length and hidden layer size, the calculation amount and memory access amount are obtained.

[0124] In some optional embodiments, the acquisition module 403 is also used to allocate the step task to a first allocation object when the computing power is less than or equal to a preset value, wherein the first allocation object is used to process tasks with arithmetic intensity of the first intensity; when the computing power is greater than the preset value, allocate the step task to a second allocation object, wherein the second allocation object is used to process tasks with arithmetic intensity of the second intensity.

[0125] In some optional embodiments, the device also includes: obtaining a data log, wherein the data log contains a first storage address of a first prompt word, a second storage address of a second prompt word, a third storage address of intermediate parameters obtained in the reasoning stage, and a fourth storage address of fixed parameters in the retrieval stage and the reasoning stage; obtaining an adjustment operation performed on the data stored in the first storage address, the second storage address, the third storage address, and the fourth storage address, and obtaining adjusted data and an adjusted first storage address, an adjusted second storage address, an adjusted third storage address, and an adjusted fourth storage address; updating the data log with the adjusted data and the adjusted first storage address, the adjusted second storage address, the adjusted third storage address, and the adjusted fourth storage address, and sharing data using the updated data log.

[0126] For the description of the features in the embodiment corresponding to the language model retrieval and reasoning device based on high-speed intercommunication technology, please refer to the relevant description of the embodiment corresponding to the language model retrieval and reasoning method based on high-speed intercommunication technology, which will not be repeated here.

[0127] The embodiment of the present application also provides an electronic device, such as Figure 5 As shown, it includes a memory 10 and a processor 20, wherein the memory 10 stores a computer program, and the processor 20 is configured to run the computer program to execute the steps in any of the above-mentioned language model retrieval and reasoning method embodiments based on high-speed intercommunication technology.

[0128] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned language model retrieval and reasoning method embodiments based on high-speed intercommunication technology when running.

[0129] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0130] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned language model retrieval and reasoning method embodiments based on high-speed intercommunication technology are implemented.

[0131] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned language model retrieval and reasoning method embodiments based on high-speed intercommunication technology.

[0132] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] The above is a detailed introduction to a language model retrieval and reasoning system and method based on high-speed intercommunication technology provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A language model retrieval and reasoning system based on high-speed interconnection, characterized in that: The system comprises: a host, a high-speed interconnection connection switch, a high-speed interconnection connection memory pool, and an image processor, wherein the high-speed interconnection connection memory pool comprises a near-memory computing memory and a double data transfer rate memory, and the high-speed interconnection connection switch comprises an adaptive partitioning module; wherein the host is connected to the high-speed interconnection connection memory pool and the image processor respectively through the high-speed interconnection connection switch; The host is used to receive a first prompt word, start a search phase based on the first prompt word, perform vectorization processing and database search on the first prompt word, generate a second prompt word, and store the second prompt word in the double data rate memory through the high-speed intercommunication connection switch; The host is further used to extract the second prompt word from the double data rate memory through the high-speed intercommunication connection switch, and to infer the second prompt word, and to determine the task allocation object in the inference phase by using the adaptive partitioning module in the high-speed intercommunication connection switch connected thereto; The adaptive partitioning module is used to obtain the computing power of all step tasks executed by the second prompt word in the reasoning stage, and determine the allocation objects of the step tasks based on the computing power, and allocate the step tasks to the corresponding allocation objects for task calculation, wherein the allocation objects are the near storage computing memory or the image processor.

2. The system according to claim 1, characterized in that The double data transfer rate memory includes a non-fixed memory pool and a fixed memory pool, wherein the non-fixed memory pool is used to store intermediate parameters generated in the reasoning stage, and the fixed memory pool is used to store fixed parameters in the retrieval stage and the reasoning stage and storage addresses of the intermediate parameters and the fixed parameters; The high-speed intercommunication connection memory pool also includes a memory migration module; The memory migration module is used to store the first prompt word and the second prompt word in the non-fixed memory pool through the high-speed interconnection switch, migrate the vector value obtained after the allocation object calculates the step task from the local memory to the non-fixed memory pool through the high-speed interconnection switch, store the intermediate parameter in the non-fixed memory pool through the high-speed interconnection switch, and store the fixed parameter and the storage address in the fixed memory pool through the high-speed interconnection switch.

3. The system according to claim 1, characterized in that The high-speed intercommunication connection memory pool further includes a high-speed intercommunication connection protocol module, and the high-speed intercommunication connection memory pool further includes a memory configuration module; The memory configuration module is used for initiating an access request from the host or the image processor to the non-fixed memory pool and / or the fixed memory pool in the double data rate memory through the high-speed intercommunication connection protocol module; The memory configuration module is also used to initiate an access request from the near storage computing memory to the non-fixed memory pool and / or the fixed memory pool in the double data transfer rate memory through the high-speed intercommunication connection protocol module.

4. A language model retrieval and reasoning method based on high-speed intercommunication technology, characterized in that: The method is applied to a language model retrieval and reasoning system based on a high-speed intercommunication connection, and the method comprises: receiving a first prompt word, and starting a search phase based on the first prompt word, performing vectorization processing and database search on the first prompt word to generate a second prompt word; Extracting the second prompt word and performing reasoning on the second prompt word; The computing power of all step tasks executed by the second prompt word in the reasoning stage is obtained, and the allocation object of the step task is determined based on the computing power, wherein the allocation object is used to perform task calculation on the step task.

5. The method according to claim 4, characterized in that The receiving of the first prompt word and starting the search phase based on the first prompt word includes: When the first prompt word is received, obtaining a first storage address corresponding to the first prompt word; determining a collection state of the first prompt word based on the first storage address; When the acquisition status meets the preset conditions, the retrieval phase is started.

6. The method according to claim 5, characterized in that The determining the collection status of the first prompt word based on the first storage address includes: Acquire a first read state of a status register of the first storage address and a second read state of a status register of a storage address next to the first storage address; Based on the first reading status and the second reading status, a collection status of the first prompt word is determined.

7. The method according to claim 6, characterized in that When the acquisition state meets the preset condition, starting the retrieval phase includes: When the first reading status is read and the second reading status is read, it is determined that the collection status is that the first prompt word collection is completed, and the search phase is started.

8. The method according to claim 4, characterized in that The step of performing vectorization processing and database retrieval on the first prompt word to generate a second prompt word includes: Converting the first prompt word into a first vector; Inputting the first vector into the database for retrieval to obtain a second vector associated with the first vector; The first vector and the second vector are integrated to obtain the second prompt word.

9. The method according to claim 4, characterized in that The step of obtaining the computing power of all step tasks executed by the second prompt word in the reasoning stage includes: Obtain the computational amount and memory access amount corresponding to the step task; Based on the ratio of the computation amount to the memory access amount, the computing power of the step task is obtained.

10. The method according to claim 9, characterized in that The obtaining of the computation amount and memory access amount corresponding to the step task includes: Obtain the batch size, sequence length, and hidden layer size in the step task; The computation amount and the memory access amount are obtained based on the batch size, the sequence length, and the hidden layer size.

11. The method according to claim 4, characterized in that The determining the allocation object of the step task based on the computing power includes: When the computing power is less than or equal to a preset value, the step task is allocated to a first allocation object, wherein the first allocation object is used to process tasks with a first arithmetic intensity; When the computing power is greater than the preset value, the step task is allocated to a second allocation object, wherein the second allocation object is used to process tasks with arithmetic intensity of the second intensity.

12. The method according to claim 4, characterized in that The method further comprises: Acquire a data log, wherein the data log includes a first storage address of a first prompt word, a second storage address of a second prompt word, a third storage address of an intermediate parameter obtained in the inference phase, and a fourth storage address of a fixed parameter in the retrieval phase and the inference phase; Acquire an adjustment operation performed on the data stored in the first storage address, the second storage address, the third storage address, and the fourth storage address, and obtain adjusted data and an adjusted first storage address, an adjusted second storage address, an adjusted third storage address, and an adjusted fourth storage address; Based on the adjusted data and the adjusted first storage address, the adjusted second storage address, the adjusted third storage address, and the adjusted fourth storage address, the data log is updated, and the data is shared using the updated data log.

13. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the language model retrieval and reasoning method based on high-speed intercommunication technology as described in any one of claims 4 to 12 by executing the computer instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the language model retrieval and reasoning method based on high-speed intercommunication technology according to any one of claims 4 to 12.

15. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the language model retrieval and reasoning method based on high-speed intercommunication technology according to any one of claims 4 to 12.

Images