Inference method and apparatus based on pre-stored context information in large model

Abstract

The present application provides an inference method and apparatus based on pre-stored context information in a large model. The method comprises: a first node determining a first feature vector on the basis of first information and second information, wherein the first information comprises one or more of the following: role information, rule information or output format information of an agent, and the second information comprises tool description information and / or computing node description information; on the basis of a second feature vector and the first feature vector, the first node determining context information related to a user request, wherein the second feature vector is a feature vector generated on the basis of the user request; and on the basis of the context information, the first node determining a feature vector corresponding to the context information, wherein the feature vector corresponding to the context information is used for model inference. Using the present application can solve the problem of limitations of the length of a context window of a large model, thereby reducing the amount of computation, effectively increasing the speed of inference computation, and improving the real-time performance of inference.

Description

A method and device for reasoning with pre-stored context information in a large model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 26, 2024, with application number 202410116016.2 and application name “A method and device for reasoning with pre-stored context information of a large model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to a method and device for reasoning about pre-stored context information of a large model. Background Art

[0003] With the development of artificial intelligence technology, the services in future networks will include a large number of artificial intelligence (AI) models. The computing capabilities of the terminal side and the edge side will be increasingly enhanced, the services will be diversified, and they will have the ability of collaborative computing. When a user (for example: the network platform itself / terminal side / third party) submits a service request to the network side, the network side needs to reasonably orchestrate and schedule the service to ensure that the service runs normally on one or more computing nodes. Among them, the orchestration function entity is mainly used to orchestrate the service workflow based on the submitted service request, splitting the service into N tasks (N>=1), and there are certain dependencies between the tasks. The scheduling function entity is mainly used to schedule one or more tasks after the service orchestration to the appropriate computing nodes for operation based on information such as the orchestrated tasks, the current network / resource status, and service needs.

[0004] In future edge-cloud collaboration scenarios, with the continuous emergence of new services and the increasing number of applications, tool libraries, and application programming interfaces (APIs) in the network, deploying large models for intelligent business management, orchestration, and scheduling in the network, such as large language models (LLMs) based on the Transformer architecture, is a future development trend. However, the actual execution of specific tasks by this LLM faces many problems, such as its context window, that is, the number of text tokens it can process in a forward pass. The current LLM architecture limits the size of the context window, which can typically reach a maximum of several thousand tokens. When large models handle more complex tasks, these input requests often contain longer contexts and require specific domain knowledge. Traditional large language models are limited by the length of the context window, and processing long input contexts is a challenge for large models. Secondly, the long inference latency of large models is also a prominent problem in current large model applications. In addition to the large number of model parameters, the auto-regressive working mode in the Transformer architecture calculates the dependencies before and after each token, resulting in high computing requirements and frequent repeated calculations, which affects the real-time performance of inference. Summary of the Invention

[0005] This application proposes an inference method and device for pre-storing context information of a large model, which can solve the problem of limited length of the context window of a large model, save the amount of calculation, effectively improve the inference calculation speed, and improve the real-time performance of the inference.

[0006] In a first aspect, an embodiment of the present application provides an inference method for pre-stored context information of a large model, the method comprising: a first node determines a first feature vector based on first information and second information, the first information comprising one or more of the following: role information, rule information or output format information of the agent, the second information comprising tool description information and / or computing node description information; the first node determines context information related to the user request based on the second feature vector and the first feature vector, the second feature vector being a feature vector generated based on the user request; the first node determines a feature vector corresponding to the context information based on the context information, the feature vector corresponding to the context information being used for model inference.

[0007] The method can be applied to the first node, including being executed by the first node, or by a component in the first node (for example, a processor, a chip, or a chip system, etc.), or being executed by a logic module or software that can implement all or part of the functions of the first node.

[0008] In the above method, by determining the first feature vector by the first node based on the first and second information, it is possible to pre-process the information related to frequently used tools and computing nodes to obtain the first feature vector, eliminating the need to repeatedly calculate the relevant available tool information or computing node information to obtain the feature quantity during each inference, thus reducing the computational complexity of real-time inference. Furthermore, by determining the context information related to the user request by the first node based on the second and first feature vectors, when a user request arrives, context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the tool text segment or computing node text segment with the highest relevance can be selected based on the user request, thereby achieving a filtering effect. This eliminates the need to input all of the large amount of tool or node information in the network into the large model during application, and only requires selecting the most relevant portion. The first node then determines the feature vector corresponding to the context information based on the context information. By performing model inference using the feature vector corresponding to the context information, i.e., selecting the feature vector corresponding to the most relevant portion for model inference, this overcomes the problem of limited context window length for large models, thereby reducing computational complexity, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0009] In one possible implementation, the first node determines context information related to the user request based on the second feature vector and the first feature vector, including: the first node performs a correlation calculation on the second feature vector and the first feature vector to determine the feature vectors corresponding to the N text segments with the greatest correlation, where the N text segments with the greatest correlation are the context information, and N is a positive integer.

[0010] Through the above method, it is possible to select the tool segment or calculation node segment with the greatest relevance according to the user's request, thereby achieving the effect of selection filtering. In this way, a large amount of tool information or node information in the network does not need to be input into the large model when applied. Only the part with relatively large relevance needs to be selected, which can reduce the input segment length of the large model and effectively solve the problem of the limitation of the context length window of the large model.

[0011] In another possible implementation, the method further includes: the first node determining a third eigenvector, where the third eigenvector is determined based on third information, and the third information is information generated based on a combination of the first information and / or the second information.

[0012] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, including: the first node determines the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0013] Optionally, the correspondence between the third information and the third feature vector can be stored in a database, and the first node can search for the feature vector corresponding to the context information from the correspondence between the third information and the third feature vector in the database based on the context information. There is no need to repeatedly calculate the relevant available tool information or computing node information during each reasoning, thereby saving the computational complexity of real-time reasoning.

[0014] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, including: the first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0015] In the above method, in this way, it is possible to avoid repeated calculation of relevant available tool information or computing node information during each reasoning, thereby saving the computational complexity of real-time reasoning.

[0016] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, the first feature vector and the third feature vector, including: the first node determines a first model, the first model is obtained by training with a first training set, and the first training set includes the first feature vector and the third feature vector; the first node determines the feature vector corresponding to the context information based on the context information and the first model.

[0017] In this method, the connections between context paragraphs are supplemented, preserving the input order of the context paragraphs. This allows for the generation of feature vectors corresponding to the complete context information, avoiding the loss of inference accuracy associated with segmented calculations. Furthermore, the first model has the ability to merge text segments. When a new selected text segment is input, the second model can be directly invoked to generate the merged context segment feature vectors. This not only saves database storage space, but also significantly reduces the computational complexity of inference using the second model compared to inference using the larger model.

[0018] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, the first feature vector and the third feature vector, including: the first node determines a second model, the second model is obtained by training with a second training set, the second training set includes a part of the third feature vector and the first feature vector; the first node determines the feature vector corresponding to the context information based on the context information and the second model.

[0019] In this method, the connections between context paragraphs are supplemented, preserving the input order of the context paragraphs. This allows for the generation of feature vectors corresponding to the complete context information, avoiding the loss of inference accuracy associated with segmented computation. This second model has a fitting function, allowing it to use fitted smaller paragraphs to represent the information contained in the complete paragraph. This further reduces context length and computational complexity, while ensuring that the final inference performance meets the agent's accuracy requirements.

[0020] In yet another possible implementation, the method further includes: the first node sending a feature vector corresponding to the context information.

[0021] In another possible implementation, each of the first feature vector, the second feature vector, the third feature vector, and the feature vector corresponding to the context information includes one or more of the following: a key-value vector of each token in a plurality of word-unit tokens, an attention value of each token, and embedding information, wherein the key-value vector is used to represent the features of the token, the attention value is used to represent the association relationship between the token and other tokens in the plurality of tokens, and the embedding information is used to represent the features of the paragraph.

[0022] In another possible implementation, the tool description information includes one or more of the following: the function completed by the tool, the input and output content required by the tool to complete the function, the restriction conditions for using the tool, or the location information of the node with the function.

[0023] In another possible implementation, the computing node information includes one or more of the following: the location of the computing node, computing power capability information of the computing node, available resource capability information, current busy / idle status, or transmission interface rate.

[0024] On the second aspect, an embodiment of the present application provides an inference device for pre-storing context information in a large model. The device can be a first node, or a component in the first node (for example, a processor, chip, or chip system, etc.), or a logic module or software that can realize all or part of the functions of the first node.

[0025] In one possible implementation, the device may include a module or unit corresponding to each of the methods / operations / steps / actions described in the first aspect. The module or unit may be a hardware circuit, software, or a combination of hardware circuit and software.

[0026] In one possible implementation, the device includes: a preprocessing module, a selection module and a merging module, the preprocessing module is used to determine a first feature vector based on first information and second information, the first information includes one or more of the following: agent role information, rule information or output format information, the second information includes tool description information and / or computing node description information; the selection module is used to determine context information related to the user request based on the second feature vector and the first feature vector, the second feature vector is a feature vector generated based on the user request; the merging module is used to determine a feature vector corresponding to the context information based on the context information, and the feature vector corresponding to the context information is used for model inference.

[0027] In a possible implementation, the selection module is configured to perform correlation calculation on the second feature vector and the first feature vector to determine feature vectors corresponding to N text segments with the greatest correlation, where the N text segments with the greatest correlation are the context information, and N is a positive integer.

[0028] In yet another possible implementation, the merging module is further configured to determine a third eigenvector, where the third eigenvector is determined based on third information, and the third information is information generated based on a combination of the first information and / or the second information.

[0029] In yet another possible implementation, the merging module is configured to determine the feature vector corresponding to the context information based on the context information and a correspondence between the third information and the third feature vector.

[0030] In yet another possible implementation, the merging module is configured to determine a feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0031] In another possible implementation, the merging module is used to determine a first model, where the first model is obtained by training with a first training set, and the first training set includes the first feature vector and the third feature vector; the merging module is used to determine the feature vector corresponding to the context information based on the context information and the first model.

[0032] In another possible implementation, the merging module is used to determine a second model, where the second model is obtained by training with a second training set, and the second training set includes a part of the third feature vector and the first feature vector; the merging module is used to determine the feature vector corresponding to the context information based on the context information and the second model.

[0033] In yet another possible implementation, the apparatus further includes a sending module, configured to send a feature vector corresponding to the context information.

[0034] In another possible implementation, each of the first feature vector, the second feature vector, the third feature vector or the feature vector corresponding to the context information includes one or more of the following: a key-value vector of each word in multiple word tokens, an attention value of each token, and embedding information, wherein the key-value vector is used to represent the characteristics of the token, the attention value is used to represent the association relationship between the token and other tokens in the multiple tokens, and the embedding information is used to represent the characteristics of the text paragraph.

[0035] In another possible implementation, the tool description information includes one or more of the following: the function completed by the tool, the input and output content required by the tool to complete the function, the restriction conditions for using the tool, or the location information of the node with the function.

[0036] In another possible implementation, the computing node information includes one or more of the following: the location of the computing node, computing power capability information of the computing node, available resource capability information, current busy / idle status, or transmission interface rate.

[0037] Regarding the technical effects brought about by the second aspect or possible implementation methods, reference may be made to the introduction to the technical effects of the first aspect or corresponding implementation methods.

[0038] In a third aspect, an embodiment of the present application provides an inference device for pre-stored context information in a large model, the device comprising at least one processor and a communication interface, the at least one processor calling a computer program or instruction stored in a memory to execute the method described in the first aspect or a possible implementation method of the first aspect.

[0039] In a possible implementation, the communication device further includes the memory. Optionally, the memory and the processor are integrated together.

[0040] In a possible implementation, the memory is located outside the communication device.

[0041] In a fourth aspect, an embodiment of the present application provides a chip device, comprising at least one processor, wherein the at least one processor is configured to execute computer programs or instructions to implement the method described in any one of the above aspects or any possible implementation method of any one of the aspects.

[0042] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction runs on a processor, the method described in any one of the above aspects or any possible implementation method of any one of the aspects is implemented.

[0043] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the method described in any one of the above aspects or any possible implementation method of any one of the aspects is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of a model reasoning solution in the prior art;

[0045] FIG2 is a schematic diagram of the architecture of an inference system for pre-storing contextual information in a large model provided by an embodiment of the present application;

[0046] FIG3 is a flow chart of a method for inferring context information stored in a large model according to an embodiment of the present application;

[0047] FIG4 is a schematic diagram of determining a feature vector corresponding to context information provided by an embodiment of the present application;

[0048] FIG5 is a flow chart of another inference method for pre-storing context information in a large model provided by an embodiment of the present application;

[0049] FIG6 is a flow chart of another inference method for pre-storing context information in a large model provided by an embodiment of the present application;

[0050] FIG7 is a flowchart of another inference method for pre-storing context information in a large model provided by an embodiment of the present application;

[0051] FIG8 is a schematic diagram of the structure of an inference device for pre-storing context information in a large model provided by an embodiment of the present application;

[0052] FIG9 is a schematic structural diagram of another inference device for pre-storing context information in a large model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of this application.

[0054] References to "one embodiment" or "some embodiments" in this application mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0055] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c; a and b; a and c; b and c; or a, b, and c. Among them, a, b, and c can be single or multiple.

[0056] It is understood that in this application, "indication" can include direct indication, indirect indication, explicit indication, and implicit indication. When describing that a certain indication information is used to indicate A, it can be understood that the indication information carries A, directly indicates A, or indirectly indicates A.

[0057] In this application, the information indicated by the indication information is referred to as the information to be indicated. In the specific implementation process, there are many ways to indicate the information to be indicated, such as but not limited to, the information to be indicated can be directly indicated, such as the information to be indicated itself or the index of the information to be indicated, etc., or the information to be indicated can be indirectly indicated by indicating other information, wherein there is an association between the other information and the information to be indicated. It is also possible to indicate only a part of the information to be indicated, while the other parts of the information to be indicated are known or agreed in advance. For example, the indication of specific information can also be achieved with the help of the arrangement order of each information agreed in advance (such as specified in the protocol), thereby reducing the indication overhead to a certain extent.

[0058] The information to be indicated can be sent as a whole or divided into multiple sub-information and sent separately. The transmission period and / or transmission timing of these sub-information can be the same or different. The specific transmission method is not limited in this application. The transmission period and / or transmission timing of these sub-information can be predefined, for example, according to a protocol, or can be configured by the transmitting device through sending configuration information to the receiving device.

[0059] It can be understood that "sending" and "receiving" in this application indicate the direction of signal transmission. For example, "sending information to XX" can be understood as the destination of the information is XX, which can include direct sending through the air interface or wired medium, and also includes indirect sending by other units or modules through the air interface or wired medium. "Receiving information from YY" can be understood as the source of the information is YY, which can include direct receiving from YY through the air interface or wired medium, and also includes indirect receiving from YY from other units or modules through the air interface or wired medium. "Sending" can also be understood as the "output" of the chip interface, and "receiving" can also be understood as the "input" of the chip interface.

[0060] In other words, sending and receiving can be performed between devices, for example, between a network device and a terminal device, or can be performed within a device, for example, sending or receiving between components, modules, chips, software modules or hardware modules within the device through a bus, wiring or interface.

[0061] It is understandable that information may be processed between the source and destination of information transmission, such as coding, modulation, etc., but the destination can understand the valid information from the source. Similar expressions in this application can be understood similarly and will not be repeated.

[0062] The communication method provided in the embodiment of the present application can be applied to cellular communication systems related to the third generation partnership project (3GPP), for example, fourth generation (4G) communication systems, such as long term evolution (LTE) communication systems, and can also be applied to fifth generation (5G) communication systems, such as 5G new radio (NR) communication systems, or to various communication systems evolved after 5G, such as sixth generation (6G) communication systems. The method provided in the embodiment of the present application can also be applied to Bluetooth systems, wireless fidelity (WiFi) systems, LoRa systems or Internet of Vehicles systems, communication systems that support the integration of multiple wireless technologies, and device-to-device (D2D) systems. The method provided in the embodiment of the present application can also be applied to satellite communication systems, wherein the satellite communication system can be integrated with the above-mentioned communication system.

[0063] Please refer to Figure 1, which is a schematic diagram of a model inference solution in the prior art. Input information is obtained, including role information, the function to be completed, an output format example, available tool information, and the user request. The input information is then input into a large language model (LLM). The LLM performs inference based on the input information to obtain the result corresponding to the user request. However, as the tool set in the network increases or the complexity of the task increases, the input information will become very long. Some of the input information requires specific domain knowledge or historical records. The large model needs to wait until all the context is fully input before processing. However, the input information in this way may exceed the input context window limit of the large model. The current main solutions are to retrain or fine-tune to support longer context windows, or select highly similar text segments for processing through vector plug-ins, or split the context into paragraphs for separate processing, or crop the context content to adapt to the length limit of the large model context window. While vector plug-in methods or splitting long contexts can address the model's context window limitations, they still require repeated calculation of key and attention values ​​for each inference after identifying segments with similar features. The large model cannot find connections between the split context segments or identify the input order of the segments. The pruned context also loses information, all of which affect the accuracy of large model inference. Furthermore, large model applications often contain numerous prompts indicating network status or available tools. These prompts contain information inherent to the network and rarely change relative to user input requests. Recalculating key and attention values ​​for each request would be redundant, impacting the real-time performance of large model inference.

[0064] Please refer to Figure 2, which is a schematic diagram of the architecture of an inference system 200 for pre-storing context information in a large model, as proposed in an embodiment of the present application. The inference system 200 for pre-storing context information in a large model may include three modules: a pre-processing module 201, a selection module 202, and a merging module 203. These three modules have the following functions:

[0065] Preprocessing module 201: configured to determine a first feature vector based on the first information and the second information.

[0066] The first information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent role may be orchestration or scheduling. Rule information includes limitations on the agent, such as the required functions, required inputs, and restrictions. Output format information may include, for example, the workflow of a tool invocation or the computing nodes allocated to the execution unit for the corresponding task.

[0067] The second information includes tool description information and / or computing node description information. The tool description information may include one or more of the following: the function performed by the tool, the input and output content required by the tool to complete the function, the restrictions on the use of the tool, or the location information of the node that supports the function. The computing node description information may include one or more of the following: the location of the computing node, the computing power capacity information of the computing node, the available resource capacity information, the current busy / idle status, or the transmission interface rate.

[0068] The first feature vector includes one or more feature vectors, each of which may include one or more of the following: a key-value vector of each token in a plurality of tokens, an attention value of each token, and embedding information; wherein the key-value vector of each token is used to represent the characteristics of the token, the attention value is used to represent the relationship between the token and other tokens, and the embedding information is used to represent the characteristics of the text segment. The embedding information can be understood as information corresponding to all tokens. A text segment may include one or more tokens.

[0069] The selection module 202 is configured to determine context information related to the user request based on the second feature vector and the first feature vector, wherein the second feature vector is a feature vector generated based on the user request and includes one or more feature vectors.

[0070] Merging module 203 is configured to determine a feature vector corresponding to the context information based on the context information, wherein the feature vector corresponding to the context information is used for model inference, wherein the feature vector corresponding to the context information includes one or more feature vectors.

[0071] Optionally, the preprocessing module 201 can also be called a registration center preprocessor, the selection module 202 can also be called a selector, and the merging module 203 can also be called a merger. Of course, there can be other names, which are not limited in the embodiments of the present application.

[0072] It should be noted that the above description is based on the example of the reasoning system 200 for pre-stored context information of the large model including the three modules. In another possible implementation, the reasoning system 200 for pre-stored context information of the large model may also include two modules, a processing module and a transceiver module. For example, the processing module is used to implement the processing operations in the pre-processing module 201, the selection module 202 and the merging module 203, and the transceiver module is used to implement the transceiver operations between the processing module and other devices outside the reasoning system 200 for pre-stored context information of the large model. This embodiment of the present application does not limit this.

[0073] It should be noted that the embodiments of this application do not limit whether the three modules are deployed separately or together. In other words, the three modules can be three independent modules or combined into one module, which is not limited in the embodiments of this application. The module can be a hardware circuit, software, or a combination of hardware circuit and software, which is not limited in the embodiments of this application.

[0074] Exemplarily, the inference system 200 for pre-storing context information in a large model can be deployed on a first node, which can be a core network element, an access network element, a control plane element, a management plane element, or other network elements added in the future, and is not limited in the embodiments of the present application. Optionally, the first node can be deployed in the core network, access network, or other locations, and can also be deployed in the control plane or management plane, or other planes added in the future, and is not limited in the embodiments of the present application.

[0075] For example, the first node may be a device deployed in a radio access network to provide wireless communication functions for a terminal device. The first node may also be referred to as an access network (radio access network, RAN) entity, an access node, a network node, or a communication device, etc. Exemplarily, the first node may be an access network device of a cellular system related to the third generation partnership project (3GPP). For example, a fourth-generation (4G) mobile communication system, or a 5G mobile communication system. The first node may also be an access network device in an open access network (open RAN, O-RAN or ORAN) or a cloud radio access network (cloud radio access network, CRAN). Alternatively, the first node may also be an access network device in a communication system obtained by integrating two or more of the above communication systems.

[0076] The first node includes, but is not limited to, an evolved Node B (eNB), a radio network controller (RNC), a Node B (NB), a base station controller (BSC), a base transceiver station (BTS), a home base station (e.g., home evolved NodeB, or home Node B, HNB), a baseband unit (BBU), an access point (AP) in a wireless fidelity (WIFI) system, a macro base station, a micro base station, a wireless relay node, a donor node, a wireless controller in a CRAN scenario, a wireless backhaul node, a transmission point (TP) or a transmission and receiving point (TRP). The network device can also be an access network device in a 5G mobile communication system. For example, a next generation NodeB (gNB) in a new radio (NR) system, a TRP, a TP, or one or a group of antenna panels (including multiple antenna panels) of a base station in a 5G mobile communication system. Alternatively, the network device may also be a network node constituting a gNB or a transmission point. For example, a centralized unit (CU), a distributed unit (DU), a CU-control plane (CP), a CU-user plane (UP), or a radio unit (RU). The CU and DU may be configured separately or included in the same network element. For example, a BBU. The RU may be included in a radio frequency device or radio frequency unit. For example, in a remote radio unit (RRU), an active antenna unit (AAU), or a remote radio head (RRH). Alternatively, the network device may also be a server, a wearable device, a vehicle, or an on-board device. For example, in V2X technology, the network device may be a road side unit (RSU).

[0077] It should be noted that in different systems, CU (or CU-CP and CU-UP), DU or RU may have different names, but those skilled in the art can understand their meanings. For example, in the ORAN system, CU may also be called an open centralized unit (O-CU) or an open CU, DU may also be called an open distributed unit (O-DU), a centralized unit control plane (CU-CP) may also be called an open centralized unit control plane (O-CU-CP) or an open CU-CP, a centralized unit user plane (CU-UP) may also be called an open centralized unit user plane (O-CU-UP) or an open CU-UP, and RU may also be called an open radio unit (O-RU). This application does not limit this. Any of the CU, CU-CP, CU-UP, DU and RU in this application may be implemented by a software module, a hardware module, or a combination of a software module and a hardware module.

[0078] In some deployments, the CU implements some gNB functions, while the DU implements some gNB functions. For example, the CU implements the functions of the radio resource control (RRC) and packet data convergence protocol (PDCP) layers, while the DU implements the functions of the radio link control (RLC), media access control (MAC), and physical (PHY) layers. Because RRC layer information ultimately becomes PHY layer information, or is converted from PHY layer information, in this architecture, higher-layer signaling, such as RRC layer signaling or PDCP layer signaling, can also be considered to be sent by the DU, or by the DU+RU. It is understood that a network device can be a CU node, a DU node, or a device that includes both a CU node and a DU node. Furthermore, the CU can be classified as a network device in the access network (RAN) or a network device in the core network (CN), without limitation here.

[0079] Optionally, the first node may also be a core network device. The core network device is responsible for access control, registration management, service management, mobility management, etc. for terminal devices accessing the network. For example, the core network device is an AMF.

[0080] It should be noted that the first node can be the device or apparatus shown above, or it can be a component (for example, a chip), module, or unit in the device or apparatus shown above, and this application does not limit it.

[0081] The following describes in detail a method for reasoning about pre-stored context information of a large model provided by an embodiment of the present application in conjunction with the system shown in FIG2 .

[0082] Please refer to FIG3 , which is a flowchart of a method for inferring context information stored in a large model according to an embodiment of the present application. The method includes but is not limited to the following steps:

[0083] Step S301: The first node determines a first eigenvector based on the first information and the second information.

[0084] Among them, the first information may include one or more of the following: role information, rule information or output format information of the agent. For example, the role of the agent can be orchestration, scheduling, etc. The rule information includes limitations on the agent, such as what functions need to be completed, what inputs are required, what restrictions are there, etc. The output format information can be, for example, the workflow of the output calling tool, or it can also output the computing power nodes allocated to the execution unit required for the corresponding work task. Optionally, for example, when the role of the agent is orchestration, the output format information can be the workflow of the output calling tool; for example, when the role of the agent is scheduling, the output format information can be the computing power nodes allocated to the execution unit required for the output corresponding work task.

[0085] Among them, the second information includes tool description information and / or computing node description information. The tool description information may include one or more of the following: the function completed by the tool, the input and output content required for the tool to complete the function, the restriction conditions for using the tool, or the location information of the node with the function. Optionally, the tool description information can be in the form of a document or other form, which is not limited in the embodiment of the present application. The computing node information includes one or more of the following: the location of the computing node, the computing power capability information of the computing node, the available resource capability information, the current busy or idle status, or the transmission interface rate.

[0086] Optionally, a computing node may refer to a device with computing capabilities, such as a physical machine, a virtual machine, or other types of devices, including nodes in a core network or cloud computing resource pool, nodes in a RAN or edge computing resource pool, and terminals in a terminal computing resource pool, which is not limited in the embodiments of the present application.

[0087] Among them, the first feature vector includes one or more feature vectors. Each feature vector in the first feature vector includes one or more of the following: the key-value vector of each token in multiple tokens, the attention and embedding information of each token, the key-value vector of each token is used to represent the characteristics of the token, the attention value is used to represent the association relationship between the token and other tokens, and the embedding information is used to represent the characteristics of the text segment. The embedding information can be understood as information corresponding to all tokens. Among them, a text segment can include one or more word elements. Optionally, the key-value vector of each token can be called a KV vector. Optionally, the first node can also store the first feature vector, for example, storing the first feature vector in a database.

[0088] In one possible implementation, the first node can call LLM, take the first information and the second information as inputs of LLM, and thus obtain the first feature vector; in another possible implementation, the first node can call a small model after function trimming (for example, a small model obtained by trimming some functions from LLM), take the first information and the second information as inputs of the small model, and thus obtain the first feature vector.

[0089] In a possible implementation manner, before the first node determines the first eigenvector based on the first information and the second information, the method further includes: the first node acquires the first information and the second information.

[0090] The first node receives the first information from the agent. For example, the role of the agent can be orchestration, for example, it can be called an orchestration agent; the role of the agent can be scheduling, for example, it can be called a scheduling agent, which is not limited in the embodiments of the present application. The agent can be deployed in the first node or not, which is not limited in the embodiments of the present application. When the agent is deployed in the first node, the first node receiving the first information from the agent can be understood as the first node determining the first information.

[0091] The first node may receive the second information from an input text provider, which may be a tool provider or a computing node provider.

[0092] In one possible implementation, the input text provider is a tool provider, and the second information includes tool description information. The tool provider can be a node in the network that has specific functions or tools, such as a computing node on the end, edge, or cloud. This specific function can be used to execute workflow tasks in the network, meaning that the corresponding function or tool can be registered with the network and provided to the orchestrator for organizing the functions or tools required to complete the user's requested task into a workflow. When a new tool is added to the network, the description of the newly added tool can be sent to the first node. Accordingly, the first node can add the description of the newly added tool to the second information.

[0093] In another possible implementation, the input text provider is a computing node provider, and the second information includes computing node description information. The computing node can be a device with computing capabilities in the network, which can be a physical machine, a virtual machine or other types of devices, including nodes in the core network or cloud computing resource pool, nodes in the RAN or edge computing resource pool, and nodes in the terminal computing resource pool. Optionally, a node agent (hostAgent) can be deployed in the computing node. The node agent is responsible for managing the status of the computing node and has functions such as reporting the status of the computing node and creating / deleting executable bodies. Accordingly, the node agent of the computing node registers the status information of the computing node into the network and provides it to the scheduler for allocating work tasks to appropriate computing nodes for execution. When a new computing node is added to the network or the status of the computing node changes, the description information of the newly added computing node can be sent to the first node, or the description information of the computing node whose status has changed can be added or modified.

[0094] Step S302: The first node determines context information related to the user request based on the second feature vector and the first feature vector.

[0095] The second feature vector is a feature vector generated based on the user request. The second feature vector includes one or more feature vectors, each of which includes one or more of the following: a key-value vector of each token in a plurality of tokens, and attention and embedding information of each token.

[0096] Wherein, the first node determines the context information related to the user request based on the second eigenvector and the first eigenvector, which may include, for example: the first node performs a correlation calculation on the second eigenvector and the first eigenvector to determine the eigenvectors corresponding to the N most correlated text segments, where the N most correlated text segments are the context information, and N is a positive integer. For example, each eigenvector in the second eigenvector performs a correlation calculation on each eigenvector in the first eigenvector to determine the eigenvectors corresponding to the N most correlated text segments. The N text segments may be a part of the first information, a part of the second information, or a combination of a part of the first information and a part of the second information. The N text segments are obtained based on the second eigenvector and the first eigenvector. For example, when calculating the correlation, the correlation can be judged by indicators such as cosine similarity or dot product similarity. For example, the second eigenvector includes two eigenvectors, namely vector 1 and vector 2, and the first eigenvector includes three eigenvectors, namely vector 3, vector 4 and vector 5. The first node can calculate the cosine similarity of vector 1 and vector 3 to obtain similarity 1, the cosine similarity of vector 1 and vector 4 to obtain similarity 2, the cosine similarity of vector 1 and vector 5 to obtain similarity 3, the cosine similarity of vector 2 and vector 3 to obtain similarity 4, the cosine similarity of vector 2 and vector 4 to obtain similarity 5, and the cosine similarity of vector 2 and vector 5 to obtain similarity 6, and then determine the eigenvectors corresponding to the N text segments with the greatest correlation based on similarity 1, similarity 2, similarity 3, similarity 4, similarity 5 and similarity 6.

[0097] In a possible implementation, before the first node determines the context information related to the user request based on the second feature vector and the first feature vector, the first node obtains the second feature vector.

[0098] Exemplarily, the first node can receive a second feature vector from the agent. The agent can receive a user request from a user. The user can be a deployment / maintenance / user personnel within the network or the end-edge cloud distribution platform, etc., and the embodiments of the present application do not limit this. The user request includes the business that the user needs to complete on the network side, and the network side needs to execute the corresponding functional entity to complete the business request and give the execution result. Optionally, the agent can call the LLM to take the user request as input to generate a second feature vector. Optionally, the agent can call a small model after functional tailoring (for example, a small model obtained by tailoring some functions from the LLM), take the user request as input, and generate a second feature vector.

[0099] In the above method, by determining the context information related to the user request based on the second eigenvector and the first eigenvector, it is possible to select the tool segment or calculation node segment with the greatest relevance according to the user request, thereby achieving the effect of selection filtering. This makes it unnecessary for a large amount of tool information or node information in the network to be input into the large model when applied. Instead, only the part with greater relevance needs to be selected, which can reduce the input segment length of the large model and effectively solve the problem of the limitation of the context length window of the large model.

[0100] Step S303: The first node determines a feature vector corresponding to the context information based on the context information.

[0101] The feature vector corresponding to the context information includes one or more feature vectors, and each feature vector in the feature vector corresponding to the context information includes one or more of the following: a key-value vector of each token in multiple tokens, attention of each token, and embedding information.

[0102] Optionally, the method further includes: the first node determining a third eigenvector.

[0103] The third feature vector is used to determine the feature vector corresponding to the context information. Each feature vector in the third feature vector includes one or more of the following: a key-value vector of each token in a plurality of tokens, and attention and embedding information of each token.

[0104] Among them, the third feature vector is determined based on the third information, and the third information is information generated based on the first information and the second information. Optionally, the third information can also be generated based on the first information, or it can also be generated based on the second information. In one example, the second information includes tool description information, and the tool description information includes description information of tool 1, description information of tool 2... and description information of tool N. The third information can be information generated by combining the description information of the first information and the description information of tool 1, or it can be information generated by combining the description information of tool 1 and the description information of tool 2. Since the first information and the second information include a large number of text segments and a large number of possibilities for mutual combination, the text segments can be classified and merged, and possible combinations can be found and merged to generate the third information.

[0105] The first node may call the LLM and take the third information as input to generate a third eigenvector. Optionally, the first node may also call a small model after function pruned (e.g., a small model obtained by pruned some functions from the LLM) and take the third information as input to generate a third eigenvector.

[0106] Optionally, the first node stores the correspondence between the third information and the third feature vector, for example, in a database. Optionally, the first node stores the third feature vector, for example, by storing the third feature vector in a database. For example, compressed storage can be used, specifically: utilizing the distribution property of the key-value vector in the third feature vector, that is, the closer the tokens are in the context, the closer their features are. The tokens in the third feature vector can be grouped, and the tokens in each group are only compared with the first token in the group to determine the difference, and then the difference is stored as the feature of the token. In this way, storage overhead can be saved.

[0107] When the first node determines the third feature vector, there are multiple ways for the first node to determine the feature vector corresponding to the context information based on the context information. For example, there are the following ways 1 and 2:

[0108] Mode 1: The first node determines the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0109] The first node determining the feature vector corresponding to the context information based on the context information and the correspondence relationship may mean that the first node searches for the feature vector corresponding to the context information from the correspondence relationship between the third information and the third feature vector based on the context information.

[0110] In one example, the context information includes tool 1 and tool 2, and the correspondence between the third information and the third feature vector includes: tool 1 and tool 3, and feature vector 1 corresponding to tool 1 and tool 3, tool 2 and feature vector 2 corresponding to tool 2, tool 2 and tool 4 and feature vector 3 corresponding to tool 2 and tool 4, tool 1 and tool 2 and feature vector 4 corresponding to tool 1 and tool 2, and so on. The first node determines the feature vector 4 corresponding to tool 1 and tool 2 from the correspondence between the third information and the third feature vector based on the context information including tool 1 and tool 2. The feature vector 4 is the feature vector corresponding to the context information.

[0111] In the above method, through the above manner, the first node can search for the feature vector corresponding to the context information from the correspondence between the third information and the third feature vector in the database based on the context information. For example, please refer to option 1 in Figure 4. The white small square represents a feature vector in the first feature vector, and all the white small squares constitute the first feature vector. The gray small squares represent a feature vector in the second feature vector, and all the gray small squares constitute the second feature vector. The small squares in the black squares represent a feature vector in the feature vector corresponding to the context information, and the black squares represent the feature vector corresponding to the context information. When used, the feature vector corresponding to the context information can be directly retrieved, so that there is no need to repeatedly calculate the relevant available tool information or computing node information during each reasoning, saving the computational complexity of real-time reasoning.

[0112] Mode 2: The first node determines a feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0113] Method 2 includes two methods: Method A and Method B:

[0114] Method A: The first node determines the feature vector corresponding to the context information based on the context information, the first feature vector and the third feature vector, which may include: the first node determines a first model, the first model is obtained by training with a first training set, and the first training set includes a first feature vector and a third feature vector; based on the context information and the first model, determines the feature vector corresponding to the context information.

[0115] It can be understood that the first node is trained with the first training set to obtain a first model, and then the context information is used as the input of the first model to output a feature vector corresponding to the context information. Optionally, the third feature vector can be used as a label in the first training set.

[0116] In one example, the first training set includes: a feature vector of paragraph 1, a feature vector of paragraph 2, ..., a feature vector of paragraph M, and a third feature vector, wherein the feature vector of paragraph 1, the feature vector of paragraph 2, ..., the feature vector of paragraph M can be a first feature vector. The M paragraphs are the content included in the first information and the second information. The selection of the M paragraphs can be based on the classification of tools in the paragraphs, and different combinations of paragraphs of tools that may be used in combination are extracted. The first node can call LLM to generate a first training set, and then input the first training set into the small model to obtain a first model. The first model is a trained dedicated small model. Optionally, the small model can be a model obtained by cutting out some functions of LLM, and then the context information is used as the input of the first model to obtain a feature vector corresponding to the context information.

[0117] In the above method, the connection between context paragraphs can be supplemented through the above method, and the input order between context paragraphs is retained, so as to obtain the feature vector corresponding to the complete context information, avoiding the problem of loss of reasoning accuracy caused by segmented calculation. In addition, the first model has the function of merging paragraphs. Please refer to option 2 in Figure 4. The small white square represents a feature vector in the first feature vector, and all the small white squares constitute the first feature vector. The small gray square represents a feature vector in the second feature vector, and all the small gray squares constitute the second feature vector. The small square in the black square represents a feature vector in the feature vector corresponding to the context information. The black square represents the feature vector corresponding to the context information obtained by calling the first model based on the context information. In this way, database storage space can be saved, and on the other hand, the amount of calculation for reasoning using the first model is much less than that for reasoning using the large model.

[0118] Mode B: The first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector, which may include:

[0119] The first node determines a second model, where the second model is obtained by training with a second training set, and the second training set includes a portion of the third eigenvector and the first eigenvector; and determines a eigenvector corresponding to the context information based on the context information and the second model.

[0120] It can be understood that the first node is trained with the second training set to obtain the second model, and then the context information is used as the input of the second model to output the feature vector corresponding to the context information. Optionally, a portion of the third feature vector can be used as the label in the second training set.

[0121] The portion of the third eigenvector may be selected and determined according to a certain rule. For example, a eigenvector whose token's attention is greater than a first threshold may be selected from the third eigenvector as the selected eigenvector, i.e., the portion of the third eigenvector. Optionally, the first threshold may be predefined and determined by the first node, which is not limited in this embodiment of the present application.

[0122] In one example, the second training set includes: the feature vector of paragraph 1, the feature vector of paragraph 2, ..., the feature vector of paragraph M, and a part of the third feature vector, wherein the feature vector of paragraph 1, the feature vector of paragraph 2, ..., the feature vector of paragraph M can be the first feature vector. The selection of M paragraphs can refer to the above description and will not be repeated here. The first node can call LLM to generate a second training set, input the second training set into the small model, and thus obtain a second model. The second model is a trained special small model. Optionally, the small model can be a model obtained after LLM trimming some functions, and then the context information is used as the input of the second model to obtain the feature vector corresponding to the context information.

[0123] In the above method, the connection between context paragraphs can be supplemented through the above manner, and the input order between context paragraphs can be retained, so as to obtain the feature vector corresponding to the complete context information and avoid the problem of loss of reasoning accuracy caused by segmented calculation. This second model has a fitting function. That is, the second model is trained using the second training set, which includes a portion of the third eigenvector. This allows the partial third eigenvector to replace the entire third eigenvector, meaning that the fitted short paragraph can express the information contained in the complete paragraph. See option 3 in Figure 4. A small white square represents an eigenvector in the first eigenvector, and all the white squares make up the first eigenvector. A small black square represents an eigenvector in a portion of the third eigenvector, and all the black squares make up a portion of the third eigenvector. A small striped square represents an eigenvector in the eigenvector corresponding to the context information, and all the striped squares make up the eigenvector corresponding to the context information. A small gray square represents an eigenvector in the second eigenvector, and all the gray squares make up the second eigenvector. A portion of the third eigenvector can be selected from the third eigenvector according to a certain rule. The second model is trained based on a portion of the third eigenvector and the first eigenvector. Then, based on the context information, the second model is called to obtain the eigenvector corresponding to the context information. This approach not only saves database storage space, but also reduces the computational complexity of inference using the first model to much less than inference using the large model. Moreover, it can further reduce the context length and computation while ensuring that the final reasoning performance meets the agent's accuracy requirements.

[0124] Optionally, the method further includes: the first node sending a feature vector corresponding to the context information.

[0125] The first node may send a feature vector corresponding to the context information to the agent. Accordingly, the agent may obtain an output result based on the feature vector and the second feature vector corresponding to the context information. For example, the agent may call a dedicated model and use the feature vector and the second feature vector corresponding to the context information as inputs to the dedicated model to obtain an output result. If the agent is an orchestration agent, the output result may be a workflow plan for the orchestration result task. If the agent is a scheduling agent, the output result may be a plan for the execution unit computing node assigned to the scheduling result task. The agent may send the output result to a scheduler or executor for execution to obtain an execution result. After receiving the execution result, the agent sends the execution result to the user.

[0126] In the method described in FIG3 , by determining the first feature vector by the first node based on the first and second information, it is possible to pre-process the information related to frequently used tools and computing nodes to obtain the first feature vector, eliminating the need to repeatedly calculate the relevant available tool information or computing node information to obtain the feature quantity during each inference, thus reducing the computational complexity of real-time inference. Furthermore, by determining the context information related to the user request by the first node based on the second and first feature vectors, when a user request arrives, context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the tool text segment or computing node text segment with the highest relevance can be selected based on the user request, thereby achieving a filtering effect. This eliminates the need to input all tool or node information in the network into the large model when applying it, and only the highly relevant portion can be selected. The first node then determines the feature vector corresponding to the context information based on the context information. Performing model inference using the feature vector corresponding to the context information, i.e., selecting the feature vector corresponding to the highly relevant portion for model inference, can address the problem of limited context window length for large models, thereby reducing computational complexity, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0127] The following takes the system shown in Figure 2, where the first node includes three modules, namely a preprocessing module, a selection module and a merging module, and the orchestration agent is not deployed on the first node, as an example to describe in detail a method for reasoning about pre-stored context information of a large model provided in an embodiment of the present application.

[0128] Please refer to FIG5 , which is a flowchart of another inference method for pre-storing context information in a large model provided by an embodiment of the present application. The method includes but is not limited to the following steps:

[0129] Step S501: The orchestration agent sends first information to a pre-processing module in a first node.

[0130] The first information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent role may be orchestration, and the rule information may include agent limitations, such as the required functions, required inputs, and restrictions. The output format information may include, for example, the output workflow of a calling tool. For details, please refer to the relevant description in step S301 and will not be repeated here.

[0131] The orchestration agent may be deployed in the first node or not, which is not limited in the present embodiment. When the orchestration agent is deployed in the first node, the orchestration agent sending the first information to the first node can be understood as the first node determining the first information.

[0132] Step S502: The input text provider sends second information to the pre-processing module in the first node.

[0133] The input text provider may be a tool provider. The second information includes tool description information. For details, please refer to the relevant description in step S301 and will not be repeated here.

[0134] Step S503: The preprocessing module in the first node determines a first feature vector based on the first information and the second information.

[0135] The first feature vector includes one or more of the following: a key-value vector of each token in the multiple tokens, attention information of each token, and embedding information. For details, please refer to the description in step S301 and will not be repeated here.

[0136] Step S504: the merging module in the first node determines a third eigenvector.

[0137] The third feature vector is determined based on the third information, which is information generated based on the combination of the first information and / or the second information. Detailed descriptions can be made in step S303 and will not be repeated here.

[0138] Step S505: The user sends a user request to the orchestration agent.

[0139] Step S506: The orchestration agent generates a second feature vector based on the user request.

[0140] For details, please refer to the relevant description in step S302, which will not be repeated here.

[0141] Step S507: The orchestration agent sends the second feature vector to the selection module in the first node.

[0142] Step S508: The selection module in the first node determines context information related to the user request based on the second feature vector and the first feature vector.

[0143] For details, please refer to the description in step S302, which will not be repeated here.

[0144] Step S509: the selection module in the first node sends the context information to the merging module in the first node.

[0145] Step S510: The merging module in the first node determines a feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0146] The feature vector corresponding to the context information is used for model inference. For details, please refer to the relevant description in Method 1 in step S303, which will not be repeated here.

[0147] Step S511: the merging module in the first node sends the feature vector corresponding to the context information to the orchestration agent.

[0148] Step S512: The orchestration agent obtains an output result based on the feature vector corresponding to the context information and the second feature vector.

[0149] The output result may include the task workflow plan of the orchestration result. For details of steps S512 to S516, reference may be made to the description of step S303, which will not be repeated here.

[0150] Step S513: The orchestration agent sends the output result to the scheduler or executor.

[0151] Step S514: the scheduler or executor executes the output result to obtain the execution result.

[0152] Step S515: The scheduler or executor sends the execution result to the orchestration agent.

[0153] Step S516: The orchestration agent sends the execution result to the user.

[0154] In the method described in FIG5 , by determining the first feature vector by the first node based on the first and second information, it is possible to pre-process the information related to frequently used tools and computing nodes to obtain the first feature vector, eliminating the need to repeatedly calculate the relevant available tool information or computing node information to obtain the feature quantity during each inference, thus reducing the computational complexity of real-time inference. Furthermore, by determining the context information related to the user request by the first node based on the second and first feature vectors, when a user request arrives, context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the tool text segment or computing node text segment with the highest relevance can be selected based on the user request, thereby achieving a filtering effect. This eliminates the need to input all tool or node information in the network into the large model during application; only the highly relevant portion needs to be selected. The first node then determines the feature vector corresponding to the context information based on the context information. Performing model inference using the feature vector corresponding to the context information, i.e., selecting the feature vector corresponding to the highly relevant portion for model inference, can address the problem of limited context window length for large models, thereby reducing computational complexity, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0155] The following takes the system shown in Figure 2, where the first node includes three modules, namely a preprocessing module, a selection module and a merging module, and the orchestration agent is not deployed on the first node, as an example to describe in detail a method for reasoning about pre-stored context information of a large model provided in an embodiment of the present application.

[0156] Please refer to FIG6 , which is a flowchart of another inference method for pre-storing context information in a large model provided by an embodiment of the present application. The method includes but is not limited to the following steps:

[0157] For steps S601 to S603 , reference may be made to the relevant descriptions in steps S501 to S503 , which will not be repeated here.

[0158] Step S604: the merging module in the first node determines the first model or the second model.

[0159] For steps S605 to S609, reference may be made to the relevant descriptions in steps S505 to S509, which will not be repeated here.

[0160] Step S610: The merging module in the first node determines a feature vector corresponding to the context information based on the context information, the first model or the second model.

[0161] The feature vector corresponding to the context information is used for model inference. For details, please refer to the relevant description in Method 2 in step S303, which will not be repeated here.

[0162] For steps S611 to S616, reference may be made to the relevant descriptions in steps S511 to S516, which will not be repeated here.

[0163] In the method described in FIG6 , by determining the first feature vector by the first node based on the first and second information, it is possible to pre-process the information related to frequently used tools and computing nodes to obtain the first feature vector, eliminating the need to repeatedly calculate the relevant available tool information or computing node information to obtain the feature quantity during each inference, thus reducing the computational complexity of real-time inference. Furthermore, by determining the context information related to the user request by the first node based on the second and first feature vectors, when a user request arrives, context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the tool text segment or computing node text segment with the highest relevance can be selected based on the user request, thereby achieving a filtering effect. This eliminates the need to input all of the tool or node information in the network into the large model during application; only the highly relevant portion needs to be selected. The first node then determines the feature vector corresponding to the context information based on the context information. Performing model inference using the feature vector corresponding to the context information, i.e., selecting the feature vector corresponding to the highly relevant portion for model inference, can address the problem of limited context window length for large models, thereby reducing computational complexity, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0164] The following takes the system shown in Figure 2, where the first node includes three modules, namely a preprocessing module, a selection module and a merging module, and the orchestration agent is not deployed on the first node, as an example to describe in detail a method for reasoning about pre-stored context information of a large model provided in an embodiment of the present application.

[0165] Please refer to FIG7 , which is a flowchart of another inference method for pre-storing context information in a large model provided by an embodiment of the present application. The method includes but is not limited to the following steps:

[0166] Step S701: The scheduling agent sends first information to the pre-processing module in the first node.

[0167] The first information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent role may be scheduling, and the rule information may include limitations on the agent, such as the required functions, required inputs, and restrictions. The output format information may include, for example, the computing nodes allocated to the execution units required to output the corresponding work tasks. For details, please refer to the relevant description in step S301 and will not be repeated here.

[0168] Step S702: The input text provider sends second information to the pre-processing module in the first node.

[0169] The input text provider may be a computing node provider. The second information includes computing node description information. For details, please refer to the relevant description in step S301 and will not be repeated here.

[0170] For steps S703 and S704, reference may be made to the relevant descriptions in steps S503 and S504, which will not be repeated here.

[0171] Step S705: The user sends a user request to the scheduling agent.

[0172] Step S706: The scheduling agent generates a second feature vector based on the user request.

[0173] For details, please refer to the relevant description in step S302, which will not be repeated here.

[0174] Step S707: the scheduling agent sends the second feature vector to the selection module in the first node.

[0175] For steps S708 to S710, reference may be made to the relevant descriptions in steps S508 to S510, which will not be repeated here.

[0176] Step S711: the merging module in the first node sends the feature vector corresponding to the context information to the scheduling agent.

[0177] Step S712: The scheduling agent obtains an output result based on the feature vector corresponding to the context information and the second feature vector.

[0178] The output result may be a plan of execution unit computing nodes assigned to the scheduling result task. For details of steps S712 to S516, reference may be made to the description of step S303, which will not be repeated here.

[0179] Step S713: The scheduling agent sends the output result to the scheduler or executor.

[0180] Step S714: The scheduler or executor executes the output result to obtain the execution result.

[0181] Step S715: The scheduler or executor sends the execution result to the scheduling agent.

[0182] Step S716: The scheduling agent sends the execution result to the user.

[0183] In the method described in FIG. 7 , by determining the first feature vector by the first node based on the first and second information, it is possible to pre-process the information related to frequently used tools and computing nodes to obtain the first feature vector, eliminating the need to repeatedly calculate the relevant available tool information or computing node information to obtain the feature quantity during each inference, thus reducing the computational complexity of real-time inference. Furthermore, by determining the context information related to the user request by the first node based on the second and first feature vectors, when a user request arrives, context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the tool text segment or computing node text segment with the highest relevance can be selected based on the user request, thereby achieving a filtering effect. This eliminates the need to input all tool or node information in the network into the large model during application; only the highly relevant portion needs to be selected. The first node then determines the feature vector corresponding to the context information based on the context information. By performing model inference using the feature vector corresponding to the context information, i.e., selecting the feature vector corresponding to the highly relevant portion for model inference, this overcomes the problem of limited context window length for large models, thereby reducing computational complexity, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0184] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0185] Please refer to Figure 8, which is a structural diagram of an inference device 800 for pre-storing context information in a large model provided by an embodiment of the present application. The device can be a first node, or a component in the first node (for example, a processor, a chip, or a chip system, etc.), or it can be a logic module or software that can realize all or part of the functions of the first node.

[0186] In one possible implementation, the device may include a module or unit that corresponds one-to-one to the method / operation / step / action performed by the first node in the above method embodiment. The module or unit may be a hardware circuit, software, or a combination of hardware circuit and software.

[0187] In one possible implementation, the device includes: a preprocessing module 801, a selection module 802 and a merging module 803, the preprocessing module 801 is used to determine a first feature vector based on first information and second information, the first information includes one or more of the following: role information, rule information or output format information of the agent, and the second information includes tool description information and / or computing node description information; the selection module 802 is used to determine context information related to the user request based on the second feature vector and the first feature vector, the second feature vector is a feature vector generated based on the user request; the merging module 803 is used to determine a feature vector corresponding to the context information based on the context information, and the feature vector corresponding to the context information is used for model inference.

[0188] In a possible implementation, the selection module 802 is configured to perform correlation calculation on the second feature vector and the first feature vector to determine feature vectors corresponding to the N most correlated text segments, where the N most correlated text segments are the context information, and N is a positive integer.

[0189] In another possible implementation, the merging module 803 is further configured to determine a third eigenvector, where the third eigenvector is determined based on third information, and the third information is information generated based on a combination of the first information and / or the second information.

[0190] In yet another possible implementation, the merging module 803 is configured to determine a feature vector corresponding to the context information based on the context information and a correspondence between the third information and the third feature vector.

[0191] In yet another possible implementation, the merging module 803 is configured to determine a feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0192] In another possible implementation, the merging module 803 is used to determine a first model, where the first model is obtained by training with a first training set, and the first training set includes the first feature vector and the third feature vector; the merging module 803 is used to determine the feature vector corresponding to the context information based on the context information and the first model.

[0193] In another possible implementation, the merging module 803 is used to determine a second model, where the second model is obtained by training with a second training set, and the second training set includes a part of the third feature vector and the first feature vector; the merging module 803 is used to determine the feature vector corresponding to the context information based on the context information and the second model.

[0194] In yet another possible implementation, the apparatus further includes a sending module, configured to send a feature vector corresponding to the context information.

[0195] In another possible implementation, each of the first feature vector, the second feature vector, the third feature vector or the feature vector corresponding to the context information includes one or more of the following: a key-value vector of each word token in multiple tokens, an attention value of the token, and embedding information, wherein the key-value vector is used to represent the characteristics of the token, the attention value is used to represent the association relationship between the token and other tokens, and the embedding information is used to represent the characteristics of the text segment.

[0196] In another possible implementation, the tool description information includes one or more of the following: the function completed by the tool, the input and output content required by the tool to complete the function, the restriction conditions for using the tool, or the location information of the node with the function.

[0197] In another possible implementation, the computing node information includes one or more of the following: the location of the computing node, computing power capability information of the computing node, available resource capability information, current busy / idle status, or transmission interface rate.

[0198] It should be noted that the implementation and beneficial effects of each module can also correspond to the corresponding description of the method embodiments shown in Figures 3, 5, 6 and 7. The division of modules in the embodiments of the present application is schematic and is only a logical function division. In actual implementation, other division methods may be used.

[0199] Please refer to Figure 9, which is a schematic diagram of the structure of another large-model pre-stored context information inference device 900 provided in an embodiment of the present application. The large-model pre-stored context information inference device 900 includes at least one processor 901 and a communication interface 903, and optionally also includes a memory 902. The processor 901, memory 902, and communication interface 903 are interconnected via a bus 904. Optionally, the processor 901 and memory 902 can be integrated together.

[0200] The memory 902 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). The memory 902 is used for storing computer programs and data. The communication interface 903 is used to receive and send data.

[0201] The processor 901 may be one or more central processing units (CPUs). In the case where the processor 901 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0202] The processor 901 in the inference device 900 for pre-storing context information of the large model is used to read the computer program or instructions stored in the memory 902 to implement the functions of the above-mentioned preprocessing module, selection module and merging module, and the communication interface 903 in the inference device 900 for pre-storing context information of the large model is used to implement the functions of the above-mentioned transceiver module.

[0203] An embodiment of the present application further provides a chip device, which includes at least one processor, and the at least one processor is used to call a computer program or instruction stored in a memory so that the processor executes the method provided by the above embodiment.

[0204] In a possible implementation, the input of the chip device corresponds to the receiving operation in any one of the foregoing embodiments, and the output of the chip device corresponds to the sending operation in any one of the foregoing embodiments.

[0205] Optionally, the processor is coupled to the memory via an interface.

[0206] Optionally, the chip device further includes a memory, in which computer program instructions are stored.

[0207] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program or instructions. When the computer program or instructions are executed on a processor, the method executed by the first node in the above method embodiment is implemented.

[0208] An embodiment of the present application further provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed on a processor, the method performed by the first node in the above method embodiment is implemented.

[0209] It is understood that the processor in the embodiments of the present application may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0210] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a base station, a terminal or a core network device. Of course, the processor and the storage medium can also exist as discrete components in a base station, a terminal or a core network device.

[0211] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or nonvolatile storage medium, or may include both volatile and nonvolatile types of storage media.

[0212] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0213] In the description of this application, words such as "first", "second", "S301", or "S302" are only used to distinguish the description and facilitate the context. Different sequence numbers themselves do not have specific technical meanings and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying the order of execution of operations. The execution order of each process should be determined by its function and internal logic.

[0214] In this application, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. A and B can be singular or plural. Additionally, the character " / " in this document indicates that the related objects are in an "or" relationship.

[0215] In this application, "transmission" may include the following three situations: sending of data, receiving of data, or sending of data and receiving of data. In this application, "data" may include business data and / or signaling data.

[0216] In this application, the terms "comprise" or "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process / method comprising a series of steps, or a system / product / apparatus comprising a series of units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes / methods / products / apparatus.

[0217] In the description of this application, unless otherwise specified, the number of nouns refers to "singular or plural," that is, "one or more." "At least one" means one or more. "Including at least one of the following: A, B, C" means that it may include A, or include B, or include C, or include A and B, or include A and C, or include B and C, or include A, B, and C. A, B, and C can be single or plural.

Claims

1. An inference method for pre-stored context information of a large model, characterized in that, Applied to the first node, including: The first node determines a first feature vector based on first information and second information, where the first information includes one or more of the following: role information of an agent, rule information, or output format information, and the second information includes tool description information and / or computing node description information; The first node determines context information related to a user request based on a second feature vector and the first feature vector, where the second feature vector is a feature vector generated based on the user request; The first node determines a feature vector corresponding to the context information based on the context information, and the feature vector corresponding to the context information is used for model inference.

2. The method according to claim 1, wherein The first node determines context information related to a user request based on a second feature vector and the first feature vector, including: The first node performs a correlation calculation on the second feature vector and the first feature vector to determine feature vectors corresponding to the N paragraphs with the greatest correlation, and the N paragraphs with the greatest correlation are the context information, where N is a positive integer.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The first node determines a third feature vector, where the third feature vector is determined based on third information, and the third information is information generated based on the first information and the second information.

4. The method according to claim 3, wherein The first node determines a feature vector corresponding to the context information based on the context information, including: The first node determines the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

5. The method according to any one of claims 1 to 3, characterized in that The first node determines a feature vector corresponding to the context information based on the context information, including: The first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

6. The method according to claim 5, characterized in that, The first node determines a feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector, including: The first node determines a first model, where the first model is trained through a first training set, and the first training set includes the first feature vector and the third feature vector; The first node determines the feature vector corresponding to the context information based on the context information and the first model.

7. The method according to claim 5, wherein The first node determines a feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector, including: The first node determines a second model, where the second model is trained through a second training set, and the second training set includes a part of the third feature vector and the first feature vector; The first node determines the feature vector corresponding to the context information based on the context information and the second model.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The first node sends the feature vector corresponding to the context information.

9. The method according to any one of claims 1-8, characterized in that, Each of the first feature vector, the second feature vector, the third feature vector, and the feature vector corresponding to the context information includes one or more of the following: key-value vectors of each token among a plurality of tokens, attention values of each token, and embedding information. The key-value vectors are used to represent the features of the tokens, the attention values are used to represent the association relationships between the token and other tokens among the plurality of tokens, and the embedding information is used to represent the features of the text passage.

10. The method according to any one of claims 1-9, characterized in that, The tool description information includes one or more of the following: functions completed by the tool, input and output content required for the tool to complete the functions, limiting conditions for using the tool, or location information of nodes with this function.

11. The method according to any one of claims 1-10, characterized in that, The computing node information includes one or more of the following: the location of the computing node, computing power capability information of the computing node, available resource capability information, current busy or idle state, or transmission interface rate.

12. An inference device for pre-storing context information of a large model, characterized in that, The device includes a preprocessing module, a selection module, and a merging module. The preprocessing module is configured to determine a first feature vector based on first information and second information. The first information includes one or more of the following: role information of the agent, rule information, or output format information. The second information includes tool description information and / or computing node description information. The selection module is configured to determine context information related to the user request based on the second feature vector and the first feature vector. The second feature vector is a feature vector generated based on the user request. The merging module is configured to determine a feature vector corresponding to the context information based on the context information. The feature vector corresponding to the context information is used for model inference.

13. The device according to claim 12, wherein The selection module is configured to perform a correlation calculation on the second feature vector and the first feature vector to determine feature vectors corresponding to the N paragraphs with the largest correlation. The N paragraphs with the largest correlation are the context information, and N is a positive integer.

14. The device according to claim 12 or 13, wherein The merging module is further configured to determine a third feature vector, where the third feature vector is determined based on third information, and the third information is information generated by combining the first information and / or the second information.

15. The device according to claim 14, wherein The merging module is configured to determine a feature vector corresponding to the context information based on the corresponding relationship between the context information, the third information, and the third feature vector.

16. The device according to any one of claims 12-14, wherein The merging module is configured to determine a feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

17. The device according to claim 16, wherein The merging module is configured to determine a first model, which is obtained by training with a first training set, and the first training set includes the first feature vector and the third feature vector; The merging module is configured to determine a feature vector corresponding to the context information based on the context information and the first model.

18. The apparatus according to claim 17, wherein The merging module is configured to determine a second model, which is obtained by training with a second training set, and the second training set includes a part of the third feature vectors and the first feature vector; The merging module is configured to determine a feature vector corresponding to the context information based on the context information and the second model.

19. The device according to any one of claims 12-18, characterized in that, The apparatus further includes a sending module, The sending module is configured to send the feature vector corresponding to the context information.

20. The device according to any one of claims 12-19, characterized in that, Each of the first feature vector, the second feature vector, the third feature vector, or the feature vector corresponding to the context information includes one or more of the following: a key-value vector of each token in a plurality of tokens, an attention value of each token, and embedding information. The key-value vector is used to represent the feature of the token, the attention value is used to represent the association relationship between the token and other tokens in the plurality of tokens, and the embedding information is used to represent the feature of the text segment.

21. The device according to any one of claims 12-20, characterized in that, The tool description information includes one or more of the following: the function completed by the tool, the input and output content required for the tool to complete the function, the limiting conditions for using the tool, or the location information of the node with this function.

22. The device according to any one of claims 12-21, characterized in that, The computing node information includes one or more of the following: the location of the computing node, the computing power capacity information of the computing node, the available resource capacity information, the current busy or idle state, or the transmission interface rate.

23. An inference device for pre-storing context information of a large model, characterized in that, The apparatus includes at least one processor and a communication interface, and the at least one processor invokes a computer program or instruction stored in a memory to execute the method according to any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, A computer program or instruction is stored in the computer-readable storage medium, and when it runs on a processor, it implements the method according to any one of claims 1-11.

25. A computer program product, characterized in that, The computer program product includes a computer program or instruction, and when the computer program or instruction runs on a computer, it implements the method according to any one of claims 1-11.

Images