Inference method and apparatus based on pre-stored context information in large model

Abstract

The present application provides an inference method and apparatus based on pre-stored context information in a large model. The method comprises: a first node determining a first feature vector on the basis of first information and second information, wherein the first information comprises one or more of the following: role information, rule information or output format information of an agent, and the second information comprises tool description information and / or computing node description information; on the basis of a second feature vector and the first feature vector, the first node determining context information related to a user request, wherein the second feature vector is a feature vector generated on the basis of the user request; and on the basis of the context information, the first node determining a feature vector corresponding to the context information, wherein the feature vector corresponding to the context information is used for model inference. Using the present application can solve the problem of limitations of the length of a context window of a large model, thereby reducing the amount of computation, effectively increasing the speed of inference computation, and improving the real-time performance of inference.

Description

A reasoning method and apparatus for pre-stored context information in a large model

[0001] This application claims priority to Chinese Patent Application No. 202410116016.2, filed on January 26, 2024, entitled "A Reasoning Method and Apparatus for Pre-stored Context Information in a Large Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communication technology, and in particular to a reasoning method and apparatus for pre-stored context information in a large model. Background Technology

[0003] With the development of artificial intelligence (AI) technology, future network services will incorporate numerous AI models. Terminal and edge computing capabilities will be increasingly enhanced, services will be diversified, and collaborative computing capabilities will be available. When a user (e.g., the network platform itself / terminal / third party) submits a service request to the network, the network needs to orchestrate and schedule the service appropriately to ensure its normal operation on one or more computing nodes. The orchestration function entity is primarily used to orchestrate the workflow of the submitted service request, breaking it down into N tasks (N>=1), with certain dependencies between these tasks. The scheduling function entity is primarily used to schedule one or more orchestrated tasks to appropriate computing nodes for execution, based on information such as the orchestrated tasks, the current network / resource status, and service requirements.

[0004] In future edge-cloud collaboration scenarios, with the emergence of new services and the increasing number of applications, tool libraries, and application programming interfaces (APIs) in the network, deploying large models for managing, orchestrating, and scheduling intelligent services—such as large language models (LLMs) based on the Transformer architecture—is a future trend. However, the actual execution of specific tasks by these LLMs faces many challenges, such as their context window, which is the number of text tokens they can process in a forward pass. Current LLM architectures limit the size of the context window, typically allowing a maximum of a few thousand tokens. When large models handle more complex tasks, these input requests often contain longer contexts and require domain-specific knowledge. Traditional large language models, limited by their context window length, face a challenge in handling long input contexts. Secondly, the large inference latency of large models is also a prominent problem in current large model applications. In addition to the large number of parameters in the model itself, the auto-regressive working mode in the Transformer architecture leads to large computational requirements and a lot of repetitive computations due to the dependency between each token, which affects the real-time performance of inference. Summary of the Invention

[0005] This application proposes a reasoning method and apparatus for pre-stored context information in large models, which can solve the problem of the limitation of the context window length in large models, save computation, effectively improve the reasoning calculation speed, and improve the real-time performance of reasoning.

[0006] In a first aspect, embodiments of this application provide a reasoning method for pre-stored context information in a large model. The method includes: a first node determining a first feature vector based on first information and second information, wherein the first information includes one or more of the following: agent role information, rule information, or output format information; and the second information includes tool description information and / or computation node description information; the first node determining context information related to a user request based on a second feature vector and the first feature vector, wherein the second feature vector is a feature vector generated based on the user request; and the first node determining a feature vector corresponding to the context information based on the context information, wherein the feature vector corresponding to the context information is used for model reasoning.

[0007] This method can be applied to the first node, including execution by the first node, execution by components in the first node (e.g., processor, chip, or chip system), or execution by a logic module or software that can implement all or part of the functions of the first node.

[0008] In the above method, by having the first node determine the first feature vector based on the first and second information, the relevant information of frequently used tools and computing nodes can be preprocessed to obtain the first feature vector in advance. This avoids repeatedly calculating the relevant available tool information or computing node information to obtain feature quantities during each inference, saving the computational load of real-time inference. Furthermore, by having the first node determine the context information related to the user request based on the second and first feature vectors, when a user request arrives, the context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the most relevant tool or computing node segment can be selected based on the user request, achieving a filtering effect. This means that a large amount of tool or node information in the network does not need to be input into the large model during application; only the highly relevant parts need to be selected. Then, the first node determines the feature vector corresponding to the context information. Model inference is performed using the feature vector corresponding to the context information, that is, model inference is performed using the feature vector corresponding to the highly relevant parts. This solves the problem of the context window length limitation of large models, thereby saving computational load, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0009] In one possible implementation, the first node determines context information related to the user request based on the second feature vector and the first feature vector, including: the first node performs a correlation calculation between the second feature vector and the first feature vector to determine the feature vectors corresponding to the N text segments with the highest correlation, wherein the N text segments with the highest correlation are the context information, and N is a positive integer.

[0010] The above method enables the selection of the most relevant tool text or computation node text based on user requests, thereby achieving the effect of selection filtering. This means that a large amount of tool information or node information in the network does not need to be input into the large model during application; only the parts with higher relevance need to be selected. This reduces the length of the input text in the large model and effectively solves the problem of the limitation of the context length window in the large model.

[0011] In another possible implementation, the method further includes: the first node determining a third feature vector, the third feature vector being determined based on third information, the third information being information generated based on a combination of the first information and / or the second information.

[0012] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, including: the first node determines the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0013] Optionally, the correspondence between the third information and the third feature vector can be stored in a database. The first node can search for the feature vector corresponding to the context information from the correspondence between the third information and the third feature vector in the database based on the context information. This eliminates the need to repeatedly calculate the relevant available tool information or calculation node information during each inference, thus saving the computational load of real-time inference.

[0014] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, including: the first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0015] In the above method, it is possible to avoid repeatedly calculating relevant available tool information or computation node information during each inference, thus saving the computational load of real-time inference.

[0016] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector, including: the first node determines a first model, the first model being trained using a first training set, the first training set including the first feature vector and the third feature vector; the first node determines the feature vector corresponding to the context information based on the context information and the first model.

[0017] The above method, by employing the aforementioned approach, supplements the connections between contextual paragraphs, preserves the input order of these paragraphs, and thus obtains feature vectors corresponding to complete contextual information, avoiding the inference accuracy loss caused by segmented calculations. Furthermore, this first model possesses a text merging function; when a new selected textual paragraph is input, the second model can be directly invoked to generate merged contextual segment feature values. This saves database storage space, and the computational cost of inference using the second model is significantly less than that of inference using the larger model.

[0018] In another possible implementation, the first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector, including: the first node determines a second model, the second model being trained using a second training set, the second training set including a portion of the third feature vector and the first feature vector; the first node determines the feature vector corresponding to the context information based on the context information and the second model.

[0019] The above method, by supplementing the connections between contextual paragraphs and preserving the input order of contextual paragraphs, obtains the feature vector corresponding to the complete contextual information, avoiding the reasoning accuracy loss caused by segmented calculation. This second model has a fitting function, using fitted short segments to represent the information of the complete text segment. While ensuring that the final reasoning performance meets the accuracy requirements of the proxy, it further reduces the context length and computational cost.

[0020] In another possible implementation, the method further includes: the first node sending the feature vector corresponding to the context information.

[0021] In another possible implementation, each feature vector among the first feature vector, the second feature vector, the third feature vector, and the feature vector corresponding to the context information includes one or more of the following: a key-value vector of each token among multiple tokens, an attention value of each token, and embedding information, wherein the key-value vector is used to represent the features of the token, the attention value is used to represent the association between the token and other tokens among the multiple tokens, and the embedding information is used to represent the features of the text segment.

[0022] In another possible implementation, the tool description information includes one or more of the following: the function performed by the tool, the input and output content required for the tool to perform the function, the limitations of using the tool, or the location information of the node with the function.

[0023] In another possible implementation, the computing node information includes one or more of the following: the location of the computing node, the computing power information of the computing node, the available resource capacity information, the current busy / idle status, or the transmission interface rate.

[0024] Secondly, embodiments of this application provide an inference device for pre-stored context information in a large model. This device can be a first node, a component in the first node (e.g., a processor, a chip, or a chip system), or a logic module or software that can implement all or part of the functions of the first node.

[0025] In one possible implementation, the device may include modules or units that perform the methods / operations / steps / actions described in the first aspect. These modules or units may be hardware circuits, software, or a combination of hardware circuits and software.

[0026] In one possible implementation, the device includes: a preprocessing module, a selection module, and a merging module. The preprocessing module is used to determine a first feature vector based on first information and second information. The first information includes one or more of the following: agent role information, rule information, or output format information. The second information includes tool description information and / or computation node description information. The selection module is used to determine context information related to the user request based on the second feature vector and the first feature vector. The second feature vector is a feature vector generated based on the user request. The merging module is used to determine a feature vector corresponding to the context information based on the context information. The feature vector corresponding to the context information is used for model inference.

[0027] In one possible implementation, the selection module is used to perform a correlation calculation between the second feature vector and the first feature vector to determine the feature vectors corresponding to the N text segments with the highest correlation, wherein the N text segments with the highest correlation are the context information, and N is a positive integer.

[0028] In another possible implementation, the merging module is further configured to determine a third feature vector, which is determined based on third information, which is information generated based on a combination of the first information and / or the second information.

[0029] In another possible implementation, the merging module is used to determine the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0030] In another possible implementation, the merging module is used to determine the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0031] In another possible implementation, the merging module is used to determine a first model, which is trained on a first training set, the first training set including the first feature vector and the third feature vector; the merging module is used to determine the feature vector corresponding to the context information based on the context information and the first model.

[0032] In another possible implementation, the merging module is used to determine a second model, which is trained using a second training set, the second training set including a portion of the third feature vector and the first feature vector; the merging module is used to determine the feature vector corresponding to the context information based on the context information and the second model.

[0033] In another possible implementation, the device further includes a sending module for sending the feature vector corresponding to the context information.

[0034] In another possible implementation, each feature vector among the first feature vector, the second feature vector, the third feature vector, or the feature vector corresponding to the context information includes one or more of the following: a key-value vector for each word in the plurality of tokens, an attention value for each token, and embedding information, wherein the key-value vector is used to represent the features of the token, the attention value is used to represent the association between the token and other tokens in the plurality of tokens, and the embedding information is used to represent the features of the text segment.

[0035] In another possible implementation, the tool description information includes one or more of the following: the function performed by the tool, the input and output content required for the tool to perform the function, the limitations of using the tool, or the location information of the node with the function.

[0036] In another possible implementation, the computing node information includes one or more of the following: the location of the computing node, the computing power information of the computing node, the available resource capacity information, the current busy / idle status, or the transmission interface rate.

[0037] For the technical effects of the second aspect or possible implementation, please refer to the introduction of the technical effects of the first aspect or corresponding implementation.

[0038] Thirdly, embodiments of this application provide an inference apparatus for pre-stored context information of a large model. The apparatus includes at least one processor and a communication interface. The at least one processor calls a computer program or instruction stored in a memory to execute the method described in the first aspect or a possible implementation thereof.

[0039] In one possible implementation, the communication device further includes the memory. Optionally, the memory and processor are integrated together.

[0040] In one possible implementation, the memory is located outside the communication device.

[0041] Fourthly, embodiments of this application provide a chip device, the chip device including at least one processor, the at least one processor being configured to execute computer programs or instructions to implement the method described in any of the above aspects or possible implementations of any of the above aspects.

[0042] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a processor, implement the method described in any of the above aspects or possible implementations of any of the above aspects.

[0043] Sixthly, embodiments of this application provide a computer program product, which includes a computer program or instructions that, when executed on a computer, implement the method described in any of the above aspects or possible implementations of any of the above aspects. Attached Figure Description

[0044] Figure 1 is a schematic diagram of a model inference scheme in the prior art;

[0045] Figure 2 is a schematic diagram of the architecture of a large model pre-stored context information inference system provided in an embodiment of this application;

[0046] Figure 3 is a flowchart illustrating a reasoning method for pre-stored context information in a large model, as provided in an embodiment of this application.

[0047] Figure 4 is a schematic diagram of determining the feature vector corresponding to context information according to an embodiment of this application;

[0048] Figure 5 is a flowchart illustrating another reasoning method for pre-stored context information in a large model provided in an embodiment of this application;

[0049] Figure 6 is a flowchart illustrating another reasoning method for pre-stored context information in a large model provided in an embodiment of this application;

[0050] Figure 7 is a flowchart illustrating another reasoning method for pre-stored context information in a large model provided in an embodiment of this application;

[0051] Figure 8 is a schematic diagram of the structure of a large model pre-stored context information inference device provided in an embodiment of this application;

[0052] Figure 9 is a schematic diagram of the structure of another inference device for pre-stored context information of a large model provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0054] References to "one embodiment" or "some embodiments" as described in this application mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0055] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c; a and b; a and c; b and c; or a and b and c. Where a, b, and c can be single or multiple.

[0056] It is understood that in this application, "instruction" can include direct instruction, indirect instruction, explicit instruction, and implicit instruction. When describing a certain instruction information to indicate A, it can be understood that the instruction information carries A, directly indicates A, or indirectly indicates A.

[0057] In this application, the information indicated by the instruction information is called the information to be instructed. In specific implementations, there are many ways to instruct the information to be instructed, such as, but not limited to, directly instructing the information to be instructed, such as the information to be instructed itself or its index; indirectly instructing the information to be instructed by instructing other information, where there is a relationship between the other information and the information to be instructed; or instructing only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent.

[0058] The information to be instructed can be sent as a whole or divided into multiple sub-information messages, and the sending period and / or timing of these sub-information messages can be the same or different. This application does not limit the specific sending method. The sending period and / or timing of these sub-information messages can be predefined, for example, according to a protocol, or configured by the transmitting device by sending configuration information to the receiving device.

[0059] It is understood that "send" and "receive" in this application refer to the direction of signal transmission. For example, "send information to XX" can be understood as the destination of the information being XX, which can include direct transmission via air interface or wired medium, or indirect transmission by other units or modules via air interface or wired medium. "Receive information from YY" can be understood as the source of the information being YY, which can include direct reception from YY via air interface or wired medium, or indirect reception from YY by other units or modules via air interface or wired medium. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface.

[0060] In other words, sending and receiving can occur between devices, such as between network devices and terminal devices, or within a device, such as between components, modules, chips, software modules, or hardware modules within the device via buses, wiring, or interfaces.

[0061] It is understandable that information may undergo necessary processing, such as encoding and modulation, between the source and destination, but the destination can understand the valid information from the source. Similar statements in this application can be interpreted in a similar way and will not be elaborated further.

[0062] The communication method provided in this application can be applied to cellular communication systems related to the 3rd Generation Partnership Project (3GPP), such as 4th generation (4G) communication systems, including Long Term Evolution (LTE) systems. It can also be applied to 5th generation (5G) communication systems, such as 5G New Radio (NR) systems, or to various communication systems evolving after 5G, such as 6th generation (6G) systems. The method provided in this application can also be applied to Bluetooth systems, Wireless Fidelity (WiFi) systems, LoRa systems, or vehicle-to-everything (V2X) systems, communication systems supporting the integration of multiple wireless technologies, and device-to-device (D2D) systems. The method provided in this application can also be applied to satellite communication systems, wherein the satellite communication system can be integrated with the aforementioned communication systems.

[0063] Please refer to Figure 1, which is a schematic diagram of a model inference scheme in the prior art. Input information is acquired, including role information, the function to be performed, an example of the output format, available tool information, and the user request. This input information is then fed into a large language model (LLM). The LLM performs inference based on this input information to obtain the result corresponding to the user request. However, as the toolset in the network increases, or the task complexity increases, the input information becomes very long. Some of this requires domain-specific knowledge or historical information. The large model needs to wait for all context information to be fully input before processing. However, the input information in this way may exceed the input context window limit of the large model. Currently, the main solutions are to retrain or fine-tune to support longer context windows, or to select highly similar text segments for processing through vector extensions, or to split the context into paragraphs for separate processing, or to prune the context content to adapt to the limitation of the large model's context window length. While vector-based approaches or splitting long contexts can address the limitation of the model's context window, vector-based approaches still require repeated calculation of key-direction and attention values ​​during each inference iteration after identifying text segments with similar features. Furthermore, large models struggle to identify connections between split context segments and their input order. Pruned contexts also lose information, all of which negatively impact the accuracy of large model inference. Moreover, large model applications often employ numerous cue words representing network status or available tools. These cue words contain information inherent to the network and do not change frequently relative to user input requests. Recalculating key-direction and attention values ​​for each request is redundant and impacts the real-time performance of large model inference.

[0064] Please refer to Figure 2, which is a schematic diagram of the architecture of an inference system 200 with pre-stored context information for a large model according to an embodiment of this application. The inference system 200 with pre-stored context information for a large model may include three modules: a preprocessing module 201, a selection module 202, and a merging module 203. These three modules have the following functions:

[0065] Preprocessing module 201: used to determine the first feature vector based on the first information and the second information.

[0066] The first piece of information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent's role may be orchestration, scheduling, etc.; the rule information includes limitations on the agent, such as what functions it needs to perform, what inputs it needs, what restrictions it has, etc.; and the output format information may be, for example, the workflow for calling the tool, or the computing power nodes allocated to the execution units required for the corresponding task.

[0067] The second information includes tool description information and / or compute node description information. Tool description information may include one or more of the following: the function performed by the tool, the input and output required for the tool to perform its function, the limitations of using the tool, or the location information of the node possessing that function. Compute node description information includes one or more of the following: the location of the compute node, the computing power information of the compute node, the available resource capacity information, the current busy / idle status, or the transmission interface rate.

[0068] The first feature vector contains one or more feature vectors. Each feature vector in the first feature vector may include one or more of the following: the key-value vector of each token in the multiple tokens, the attention value of each token, and embedding information. The key-value vector of each token represents the features of the token, the attention value represents the association between the token and other tokens, and the embedding information represents the features of the text segment. This embedding information can be understood as information corresponding to all tokens. A text segment may include one or more tokens.

[0069] Selection module 202: used to determine context information related to the user request based on a second feature vector and a first feature vector, wherein the second feature vector is a feature vector generated based on the user request. The second feature vector contains one or more feature vectors.

[0070] Merging module 203: Used to determine the feature vector corresponding to the context information based on the context information. This feature vector is used for model inference. The feature vector corresponding to the context information contains one or more feature vectors.

[0071] Optionally, the preprocessing module 201 can also be called the registry center preprocessor, the selection module 202 can also be called the selector, and the merging module 203 can also be called the merger. Of course, there can be other names as well, and this application embodiment does not limit them.

[0072] It should be noted that the above description uses the inference system 200 with pre-stored context information of the large model as an example, which includes the three modules. In another possible implementation, the inference system 200 with pre-stored context information of the large model may also include two modules: a processing module and a transceiver module. For example, the processing module is used to implement the processing operations in the preprocessing module 201, the selection module 202, and the merging module 203. The transceiver module is used to implement the transceiver operations between the processing module and other devices besides the inference system 200 with pre-stored context information of the large model. This application embodiment does not limit this.

[0073] It should be noted that this application does not limit whether the three modules are deployed separately or together. In other words, the three modules can be three independent modules or combined into one module; this application does not limit this either. The module can be hardware circuitry, software, or a combination of hardware circuitry and software implementation; this application does not limit this either.

[0074] For example, the inference system 200 with pre-stored context information for a large model can be deployed on a first node. This first node can be a core network element, an access network element, a control plane network element, a management plane network element, or a network element from other planes added in the future. This application embodiment does not impose any limitations on this. Optionally, the first node can be deployed in the core network, access network, or other locations, and can also be deployed in the control plane or management plane, or other planes added in the future. This application embodiment does not impose any limitations on this.

[0075] For example, a first node can be a device deployed in a radio access network to provide wireless communication functions for terminal devices. A first node can also be referred to as an access network (RAN) entity, access node, network node, or communication device, etc. For instance, a first node can be an access network device for a cellular system related to the 3rd Generation Partnership Project (3GPP), such as a fourth-generation (4G) mobile communication system or a 5G mobile communication system. A first node can also be an access network device in an open RAN (O-RAN or ORAN) or a cloud radio access network (CRAN). Alternatively, a first node can also be an access network device in a communication system resulting from the integration of two or more of the above communication systems.

[0076] The first node includes, but is not limited to: evolved Node B (eNB), radio network controller (RNC), Node B (NB), base station controller (BSC), base transceiver station (BTS), home base station (e.g., home evolved NodeB, or home Node B, HNB), baseband unit (BBU), access point (AP) in a wireless fidelity (WIFI) system, macro base station, micro base station, wireless relay node, donor node, radio controller in a CRAN scenario, wireless backhaul node, transmission point (TP), or transmission and receiving point (TRP). Network equipment can also be access network equipment in a 5G mobile communication system. For example, next-generation Node B (gNB), TRP, TP in a new radio (NR) system, or one or a group of antenna panels (including multiple antenna panels) of a base station in a 5G mobile communication system. Alternatively, network devices can also be network nodes constituting a gNB or transmission point. Examples include centralized units (CUs), distributed units (DUs), CU-control plane (CPs), CU-user plane (UPs), or radio units (RUs). CUs and DUs can be separate entities or included in the same network element, such as a BBU. RUs can be included in radio equipment or radio units, such as remote radio units (RRUs), active antenna units (AAUs), or remote radio heads (RRHs). Alternatively, network devices can also be servers, wearable devices, vehicles, or in-vehicle equipment. For example, in V2X technology, network devices can be roadside units (RSUs).

[0077] It should be noted that CU (or CU-CP and CU-UP), DU, or RU may have different names in different systems, but those skilled in the art will understand their meaning. For example, in an ORAN system, CU can also be called an open centralized unit (O-CU) or an open CU, DU can also be called an open distributed unit (O-DU), centralized unit control plane (CU-CP) can also be called an open centralized unit control plane (O-CU-CP) or an open CU-CP, centralized unit user plane (CU-UP) can also be called an open centralized unit user plane (O-CU-UP) or an open CU-UP, and RU can also be called an open radio unit (O-RU). This application does not impose any specific limitations. Any of the units CU, CU-CP, CU-UP, DU, and RU in this application can be implemented through software modules, hardware modules, or a combination of software and hardware modules.

[0078] In some deployments, the CU and DU implement some of the functions of the gNB. For example, the CU implements the functions of the radio resource control (RRC) and packet data convergence protocol (PDCP) layers, while the DU implements the functions of the radio link control (RLC), media access control (MAC), and physical (PHY) layers. Since RRC layer information ultimately becomes PHY layer information, or is derived from PHY layer information, in this architecture, higher-layer signaling, such as RRC or PDCP signaling, can also be considered to be sent by the DU, or by the DU+RU. It is understood that network devices can be CU nodes, DU nodes, or devices including both CU and DU nodes. Furthermore, the CU can be classified as a network device in the access network (RAN) or a network device in the core network (CN); no restrictions are placed here.

[0079] Optionally, the first node can also be a core network device. The core network device is responsible for access control, registration management, service management, and mobility management of terminal devices accessing the network. For example, the core network device is an AMF (Active Network Provider).

[0080] It should be noted that the first node can be the device or apparatus shown above, or it can be a component (e.g., a chip), module, or unit in the device or apparatus shown above. This application does not limit the specifics.

[0081] The following describes in detail a reasoning method for pre-stored context information in a large model, based on the system shown in Figure 2.

[0082] Please refer to Figure 3, which is a flowchart illustrating a reasoning method for pre-stored context information in a large model according to an embodiment of this application. The method includes, but is not limited to, the following steps:

[0083] Step S301: The first node determines the first feature vector based on the first information and the second information.

[0084] The first piece of information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent's role may be orchestration, scheduling, etc.; the rule information includes limitations on the agent, such as what functions it needs to perform, what inputs it needs, and what restrictions it has; the output format information may be, for example, the workflow for calling the tool, or the computing power nodes allocated to the execution units required for the corresponding task. Optionally, for example, when the agent's role is orchestration, the output format information may be the workflow for calling the tool; for example, when the agent's role is scheduling, the output format information may be the computing power nodes allocated to the execution units required for the corresponding task.

[0085] The second information includes tool description information and / or computing node description information. Tool description information may include one or more of the following: the functions performed by the tool, the input and output content required for the tool to perform its functions, the limitations of using the tool, or the location information of the node possessing the function. Optionally, the tool description information may be a document or other form, which is not limited in this embodiment. Computing node information includes one or more of the following: the location of the computing node, the computing power information of the computing node, the available resource capacity information, the current busy / idle status, or the transmission interface rate.

[0086] Optionally, a computing node can refer to a device with computing capabilities, such as a physical machine, a virtual machine, or other types of devices, including nodes in the core network or cloud computing resource pool, nodes in the RAN or edge computing resource pool, and terminals in the terminal computing resource pool. This application embodiment does not limit the scope of the node.

[0087] The first feature vector includes one or more feature vectors. Each feature vector in the first feature vector includes one or more of the following: the key-value vector of each token among multiple tokens, the attention of each token, and embedding information. The key-value vector of each token is used to represent the features of the token, the attention value is used to represent the association between the token and other tokens, and the embedding information is used to represent the features of the text segment. The embedding information can be understood as information corresponding to all tokens. A text segment may include one or more tokens. Optionally, the key-value vector of each token may be called a key-value vector. Optionally, the first node may also store the first feature vector, for example, by storing the first feature vector in a database.

[0088] In one possible implementation, the first node can call the LLM, using the first and second information as inputs to the LLM to obtain the first feature vector; in another possible implementation, the first node can call a smaller model with trimmed features (e.g., a smaller model obtained by trimming some features from the LLM), using the first and second information as inputs to the smaller model to obtain the first feature vector.

[0089] In one possible implementation, before the first node determines the first feature vector based on the first information and the second information, the method further includes: the first node acquiring the first information and the second information.

[0090] In this embodiment, the first node receives first information from the agent. For example, the agent's role can be orchestration, such as an orchestration agent; or it can be scheduling, such as a scheduling agent. This embodiment does not impose any limitations. The agent may or may not be deployed in the first node; this embodiment does not impose any limitations. When the agent is deployed in the first node, the first node receiving the first information from the agent can be understood as the first node determining the first information.

[0091] The first node can receive second information from the input segment provider. This input segment provider can be a tool provider or a computing node provider.

[0092] In one possible implementation, the input text provider is a tool provider, and the second information includes tool description information. The tool provider can be a node in the network possessing specific functions or tools, such as edge, cloud, or terminal computing nodes. This specific function can be used for workflow task execution within the network; that is, the corresponding function or tool can be registered in the network and provided to the orchestrator to organize the functions or tools needed to complete the user's requested task into a workflow. When a new tool joins the network, it can send its description information to the first node, which can then add this description information to the second information.

[0093] In another possible implementation, the input segment provider is a computing node provider, and the second information includes computing node description information. The computing node can be a device with computing capabilities in the network, such as a physical machine, virtual machine, or other type of device, including nodes in the core network or cloud computing resource pool, nodes in the RAN or edge computing resource pool, and nodes in the terminal computing resource pool. Optionally, a node agent (hostAgent) can be deployed in the computing node. This node agent is responsible for managing the computing node's status, and has functions such as reporting the computing node's status and creating / deleting execution entities. Accordingly, the node agent registers the computing node's status information with the network and provides it to the scheduler for allocating tasks to appropriate computing nodes for execution. When a new computing node joins the network, or the status of a computing node changes, the description information of the newly joined computing node can be sent to the first node, or the description information of the computing node whose status has changed can be added or modified.

[0094] Step S302: The first node determines the context information related to the user request based on the second feature vector and the first feature vector.

[0095] The second feature vector is a feature vector generated based on the user request. The second feature vector includes one or more feature vectors, and each feature vector includes one or more of the following: the key-value vector of each token among multiple tokens, the attention of each token, and the embedding information.

[0096] The first node determines context information related to the user request based on the second feature vector and the first feature vector. For example, this may include: the first node performing a correlation calculation between the second and first feature vectors to determine the feature vectors corresponding to the N most relevant text segments, where the N most relevant text segments are the context information and N is a positive integer. For example, each feature vector in the second feature vector is correlated with each feature vector in the first feature vector to determine the feature vectors corresponding to the N most relevant text segments. The N text segments can be a part of the first information, a part of the second information, or a combination of a part of the first information and a part of the second information. These N text segments are obtained based on the second and first feature vectors. For example, when calculating relevance, indicators such as cosine similarity or dot product similarity can be used to judge relevance. For instance, the second feature vector includes two feature vectors, namely vector 1 and vector 2, and the first feature vector includes three feature vectors, namely vector 3, vector 4 and vector 5. The first node can calculate the cosine similarity between vector 1 and vector 3 to obtain similarity 1, the cosine similarity between vector 1 and vector 4 to obtain similarity 2, the cosine similarity between vector 1 and vector 5 to obtain similarity 3, the cosine similarity between vector 2 and vector 3 to obtain similarity 4, the cosine similarity between vector 2 and vector 4 to obtain similarity 5, and the cosine similarity between vector 2 and vector 5 to obtain similarity 6. Then, based on similarity 1, similarity 2, similarity 3, similarity 4, similarity 5 and similarity 6, the feature vectors corresponding to the N text segments with the highest relevance are determined.

[0097] In one possible implementation, the first node obtains the second feature vector before determining the context information related to the user request based on the second feature vector and the first feature vector.

[0098] For example, the first node can receive a second feature vector from the proxy. The proxy can receive user requests from users. These users can be personnel within the network, or deployment / maintenance / users on an edge-cloud distributed platform, etc., and this embodiment is not limited to these specific users. The user request includes services that the user needs to complete on the network side, requiring the network side to execute corresponding functional entities to complete the service request and provide an execution result. Optionally, the proxy can call an LLM (Elastic Compute Service) to take the user request as input and generate the second feature vector. Optionally, the proxy can call a functionally trimmed mini-model (e.g., a mini-model obtained by trimming some functions from an LLM) to take the user request as input and generate the second feature vector.

[0099] In the above method, by determining the context information related to the user request based on the second feature vector and the first feature vector, it is possible to select the most relevant tool text or computation node text according to the user request, thereby achieving the effect of selection and filtering. This means that a large amount of tool information or node information in the network does not need to be input into the large model when applied. Only the part with relatively high relevance needs to be selected, which can reduce the length of the input text of the large model and effectively solve the problem of the limitation of the context length window of the large model.

[0100] Step S303: The first node determines the feature vector corresponding to the context information based on the context information.

[0101] The feature vector corresponding to the context information includes one or more feature vectors. Each feature vector in the feature vector corresponding to the context information includes one or more of the following: the key-value vector of each token among multiple tokens, the attention of each token, and the embedding information.

[0102] Optionally, the method further includes: the first node determining the third feature vector.

[0103] The third feature vector is used to determine the feature vector corresponding to the context information. Each feature vector in the third feature vector includes one or more of the following: the key-value vector of each token among multiple tokens, the attention of each token, and the embedding information.

[0104] The third feature vector is determined based on third information, which is generated based on the first and second information. Optionally, the third information can also be generated based on the first information, or it can also be generated based on the second information. In one example, the second information includes tool description information, which includes the description information of tool 1, the description information of tool 2, ... and the description information of tool N. The third information can be generated by combining the first information and the description information of tool 1, or it can be generated by combining the description information of tool 1 and the description information of tool 2. Since the first and second information contain a large number of text segments and have many possible combinations, the text segments can be classified and merged to find possible combinations and merge them to generate the third information.

[0105] The first node can call the LLM (Limited Model) to take the third information as input and generate a third feature vector. Optionally, the first node can also call a smaller model with trimmed features (e.g., a smaller model obtained by trimming some features from the LLM) to take the third information as input and generate a third feature vector.

[0106] Optionally, the first node stores the correspondence between the third information and the third feature vector, for example, by storing it in a database. Alternatively, the first node stores the third feature vector, for example, by storing the third feature vector in a database. For example, this can be achieved through compressed storage: utilizing the distribution properties of the key-value vectors in the third feature vector—that is, tokens that are closer in context have more similar features—the tokens in the third feature vector can be grouped. Each token in a group is only compared with the first token in the group to determine the difference, and then this difference is stored as the feature of that token. This method can save storage overhead.

[0107] When the first node determines the third feature vector, there are several ways for the first node to determine the feature vector corresponding to the context information based on the context information. For example, there are methods 1 and 2:

[0108] Method 1: The first node determines the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0109] In this context, the first node determining the feature vector corresponding to the context information based on the context information and the correspondence can mean that the first node searches for the feature vector corresponding to the context information from the correspondence between the third information and the third feature vector based on the context information.

[0110] In one example, the context information includes tool 1 and tool 2, and the correspondence between the third information and the third feature vector includes: tool 1 and tool 3, and feature vector 1 corresponding to tool 1 and tool 3, tool 2 and feature vector 2 corresponding to tool 2, tool 2 and tool 4 and feature vector 3 corresponding to tool 2 and tool 4, tool 1 and tool 2 and feature vector 4 corresponding to tool 1 and tool 2, etc. The first node determines the feature vector 4 corresponding to tool 1 and tool 2 from the correspondence between the third information and the third feature vector based on the context information including tool 1 and tool 2. This feature vector 4 is the feature vector corresponding to the context information.

[0111] In the above method, the first node can find the feature vector corresponding to the context information from the correspondence between the third information and the third feature vector in the database based on the context information. For example, please refer to option 1 in Figure 4. The small white squares represent a feature vector in the first feature vector, and all the small white squares form the first feature vector. The small gray squares represent a feature vector in the second feature vector, and all the small gray squares form the second feature vector. The small squares in the black squares represent a feature vector in the feature vector corresponding to the context information. The black squares represent the feature vector corresponding to the context information. When using it, the feature vector corresponding to the context information can be directly retrieved, so that it is not necessary to repeatedly calculate the relevant available tool information or calculation node information in each inference, thus saving the computational load of real-time inference.

[0112] Method 2: The first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0113] Method 2 includes two sub-methods: Method A and Method B.

[0114] Method A: The first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector. This may include: the first node determining the first model, which is trained on the first training set, which includes the first feature vector and the third feature vector; and determining the feature vector corresponding to the context information based on the context information and the first model.

[0115] This can be understood as follows: the first node is trained using the first training set to obtain the first model, and then the context information is used as the input to the first model to output the feature vector corresponding to the context information. Optionally, this third feature vector can be used as the label in the first training set.

[0116] In one example, the first training set includes: feature vectors of text segment 1, text segment 2, ..., text segment M, and a third feature vector. The feature vectors of text segment 1, text segment 2, ..., text segment M can be the first feature vectors. The M text segments represent the content included in the first and second information. The selection of these M text segments can be based on the tool classification within the text segments. For text segments with potentially combined tools, different combinations are extracted. The first node can call the LLM to generate the first training set, and then input the first training set into a small model to obtain the first model. This first model is a pre-trained dedicated small model. Optionally, this small model can be a model obtained by trimming some functions of the LLM. Then, the context information is used as the input to the first model to obtain the feature vectors corresponding to the context information.

[0117] The above method, by supplementing the connections between contextual paragraphs and preserving the input order of contextual paragraphs, obtains the feature vector corresponding to the complete contextual information, avoiding the inference accuracy loss caused by segmented calculation. Furthermore, this first model has a text merging function; please refer to option 2 in Figure 4. The small white squares represent a feature vector in the first feature vector, and all the white squares form the first feature vector. The small gray squares represent a feature vector in the second feature vector, and all the gray squares form the second feature vector. The small black squares represent a feature vector in the feature vector corresponding to the contextual information, and the black squares represent the feature vector corresponding to the contextual information obtained by calling the first model based on the contextual information. This approach saves database storage space, and the computational cost of inference using the first model is far less than that of inference using a large model.

[0118] Method B: The first node determines the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector. This can include:

[0119] The first node determines the second model, which is trained using a second training set, including a portion of the third feature vector and the first feature vector. Based on the context information and the second model, the feature vector corresponding to the context information is determined.

[0120] This can be understood as follows: the first node is trained using the second training set to obtain the second model, and then the context information is used as the input to the second model to output the feature vector corresponding to the context information. Optionally, a portion of this third feature vector can be used as the label in the second training set.

[0121] The third feature vector can be selected and determined according to a certain rule. For example, a feature vector whose token attention is greater than a first threshold can be selected from the third feature vector as a selected feature vector, i.e., a part of the third feature vector. Optionally, the first threshold can be predefined and determined by the first node, which is not limited in this embodiment.

[0122] In one example, the second training set includes: feature vectors of text segment 1, text segment 2, ..., text segment M, and a portion of a third feature vector. The feature vectors of text segment 1, text segment 2, ..., text segment M can be the first feature vectors. The selection of the M text segments can refer to the above description and will not be repeated here. The first node can call the LLM to generate the second training set, input the second training set into a small model, and thus obtain the second model. This second model is a pre-trained dedicated small model. Optionally, this small model can be a model obtained by pruning some functions of the LLM. Then, the context information is used as the input to the second model to obtain the feature vectors corresponding to the context information.

[0123] In the above method, the connection between context paragraphs is supplemented and the input order between context paragraphs is preserved, thereby obtaining the feature vector corresponding to the complete context information and avoiding the problem of inference accuracy loss caused by segmented calculation. The second model has a fitting function; that is, it is trained on a second training set, which includes a portion of the third feature vector. This allows it to replace all third feature vectors with a portion of them, meaning the fitted short text segment represents the information of the complete text segment. See option 3 in Figure 4. Small white squares represent a feature vector within the first feature vector; all white squares form the first feature vector. Small black squares represent a feature vector within a portion of the third feature vector; all black squares form a portion of the third feature vector. Small striped squares represent a feature vector corresponding to the context information; all striped squares form the feature vector corresponding to the context information. Small gray squares represent a feature vector within the second feature vector; all gray squares form the second feature vector. A portion of the third feature vector can be selected from the third feature vector according to a certain rule. The second model is trained based on a portion of the third feature vector and the first feature vector. Then, based on the context information, the second model is called to obtain the feature vector corresponding to the context information. This approach saves database storage space, and the computational cost of inference using the first model is far less than that of inference using a large model. Moreover, it can further reduce the context length and computation while ensuring that the performance of the final inference meets the accuracy requirements of the agent.

[0124] Optionally, the method further includes: the first node sending the feature vector corresponding to the context information.

[0125] The first node can send the feature vector corresponding to the context information to the agent. The agent can then obtain the output result based on the feature vector and the second feature vector corresponding to the context information. For example, the agent can call a dedicated model, using the feature vector and the second feature vector as input to obtain the output result. If the agent is an orchestration agent, the output result can be the orchestration result task workflow plan; if the agent is a scheduling agent, the output result can be the execution unit computation node plan assigned to the scheduling result task. The agent can send the output result to the scheduler or executor for execution to obtain the execution result. After receiving the execution result, the agent sends it to the user.

[0126] In the method described in Figure 3, by having the first node determine the first feature vector based on the first and second information, the relevant information of frequently used tools and computing nodes can be preprocessed to obtain the first feature vector in advance. This avoids repeatedly calculating the relevant available tool information or computing node information to obtain feature quantities during each inference, saving computational load in real-time inference. Furthermore, by having the first node determine the context information related to the user request based on the second and first feature vectors, when a user request arrives, the context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the most relevant tool or computing node segment can be selected based on the user request, achieving a filtering effect. This means that a large amount of tool or node information in the network does not need to be input into the large model during application; only the highly relevant parts need to be selected. Then, the first node determines the feature vector corresponding to the context information. Model inference is performed using the feature vector corresponding to the context information, that is, selecting the feature vector corresponding to the highly relevant parts for model inference. This solves the problem of the context window length limitation of large models, thereby saving computational load, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0127] The following uses the system shown in Figure 2, i.e., the first node includes three modules, namely the preprocessing module, the selection module, and the merging module, and the orchestration agent is not deployed on the first node, as an example to provide a detailed description of the reasoning method for pre-stored context information of a large model provided in this application embodiment.

[0128] Please refer to Figure 5, which is a flowchart illustrating another reasoning method for pre-stored context information in a large model provided in an embodiment of this application. This method includes, but is not limited to, the following steps:

[0129] Step S501: The orchestration agent sends the first information to the preprocessing module in the first node.

[0130] The first piece of information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent's role may be orchestration; the rule information includes limitations on the agent, such as what functions it needs to perform, what inputs it requires, and what restrictions it faces; the output format information may be, for example, the workflow for calling the tool. For details, please refer to the relevant description in step S301, which will not be repeated here.

[0131] The orchestration agent may or may not be deployed on the first node; this embodiment does not impose any limitation. When the orchestration agent is deployed on the first node, the sending of first information by the orchestration agent to the first node can be understood as the first node determining the first information.

[0132] Step S502: The input segment provides direction for the preprocessing module in the first node to send the second information.

[0133] The provider of the input text can be the tool provider. The second information includes tool description information. Please refer to the relevant description in step S301 for details, which will not be repeated here.

[0134] Step S503: The preprocessing module in the first node determines the first feature vector based on the first information and the second information.

[0135] The first feature vector includes one or more of the following: the key-value vector of each token among multiple tokens, the attention of each token, and the embedding information. For details, please refer to the description in step S301, which will not be repeated here.

[0136] Step S504: The merging module in the first node determines the third feature vector.

[0137] The third feature vector is determined based on third information, which is generated by combining the first and / or second information. For details, please refer to the relevant description in step S303; it will not be repeated here.

[0138] Step S505: The user sends a user request to the orchestration agent.

[0139] Step S506: The orchestration agent generates a second feature vector based on the user request.

[0140] For details, please refer to the relevant description in step S302, which will not be repeated here.

[0141] Step S507: The orchestration agent sends the second feature vector to the selection module in the first node.

[0142] Step S508: The selection module in the first node determines the context information related to the user request based on the second feature vector and the first feature vector.

[0143] For details, please refer to the description in step S302, which will not be repeated here.

[0144] Step S509: The selection module in the first node sends the context information to the merging module in the first node.

[0145] Step S510: The merging module in the first node determines the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0146] The feature vector corresponding to this context information is used for model inference. For details, please refer to the relevant description in Method 1 of step S303, which will not be repeated here.

[0147] Step S511: The merging module in the first node sends the feature vector corresponding to the context information to the orchestration agent.

[0148] Step S512: The orchestration agent obtains the output result based on the feature vector and the second feature vector corresponding to the context information.

[0149] The output may include the orchestration result task workflow plan. Steps S512-S516 can be specifically described in step S303, and will not be repeated here.

[0150] Step S513: The orchestration agent sends the output results to the scheduler or executor.

[0151] Step S514: The scheduler or executor executes the output to obtain the execution result.

[0152] Step S515: The scheduler or executor sends the execution result to the orchestration agent.

[0153] Step S516: The orchestration agent sends the execution results to the user.

[0154] In the method described in Figure 5, by having the first node determine the first feature vector based on the first and second information, the relevant information of frequently used tools and computing nodes can be preprocessed to obtain the first feature vector in advance. This avoids repeatedly calculating the relevant available tool information or computing node information to obtain feature quantities during each inference, saving computational load in real-time inference. Furthermore, by having the first node determine the context information related to the user request based on the second and first feature vectors, when a user request arrives, the context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the most relevant tool or computing node segment can be selected based on the user request, achieving a filtering effect. This means that a large amount of tool or node information in the network does not need to be input into the large model during application; only the highly relevant parts need to be selected. Then, the first node determines the feature vector corresponding to the context information. Model inference is performed using the feature vector corresponding to the context information, that is, model inference is performed using the feature vector corresponding to the highly relevant parts. This solves the problem of the context window length limitation of large models, thereby saving computational load, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0155] The following uses the system shown in Figure 2, i.e., the first node includes three modules, namely the preprocessing module, the selection module, and the merging module, and the orchestration agent is not deployed on the first node, as an example to provide a detailed description of the reasoning method for pre-stored context information of a large model provided in this application embodiment.

[0156] Please refer to Figure 6, which is a flowchart illustrating another reasoning method for pre-stored context information in a large model provided in an embodiment of this application. This method includes, but is not limited to, the following steps:

[0157] Steps S601-S603 can be referred to the relevant descriptions in steps S501-S503, and will not be repeated here.

[0158] Step S604: The merging module in the first node determines the first model or the second model.

[0159] Steps S605-S609 can be referred to the relevant descriptions in steps S505-S509, and will not be repeated here.

[0160] Step S610: The merging module in the first node determines the feature vector corresponding to the context information based on the context information, the first model, or the second model.

[0161] The feature vector corresponding to this context information is used for model inference. For details, please refer to the relevant description in Method 2 of step S303, which will not be repeated here.

[0162] Steps S611-S616 can be referred to the relevant descriptions in steps S511-S516, and will not be repeated here.

[0163] In the method described in Figure 6, by having the first node determine the first feature vector based on the first and second information, the relevant information of frequently used tools and computing nodes can be preprocessed to obtain the first feature vector in advance. This avoids repeatedly calculating the relevant available tool information or computing node information to obtain feature quantities during each inference, saving computational load in real-time inference. Furthermore, by having the first node determine the context information related to the user request based on the second and first feature vectors, when a user request arrives, the context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the most relevant tool or computing node segment can be selected based on the user request, achieving a filtering effect. This means that a large amount of tool or node information in the network does not need to be input into the large model during application; only the highly relevant parts need to be selected. Then, the first node determines the feature vector corresponding to the context information. Model inference is performed using the feature vector corresponding to the context information, that is, selecting the feature vector corresponding to the highly relevant parts for model inference. This solves the problem of the context window length limitation of large models, thereby saving computational load, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0164] The following uses the system shown in Figure 2, i.e., the first node includes three modules, namely the preprocessing module, the selection module, and the merging module, and the orchestration agent is not deployed on the first node, as an example to provide a detailed description of the reasoning method for pre-stored context information of a large model provided in this application embodiment.

[0165] Please refer to Figure 7, which is a flowchart illustrating another reasoning method for pre-stored context information in a large model provided in an embodiment of this application. This method includes, but is not limited to, the following steps:

[0166] Step S701: The scheduling agent sends the first information to the preprocessing module in the first node.

[0167] The first piece of information may include one or more of the following: agent role information, rule information, or output format information. For example, the agent's role may be scheduling; the rule information may include limitations on the agent, such as what functions it needs to perform, what inputs it needs, and what restrictions it has; the output format information may be, for example, the computing power nodes allocated to the execution units required for the corresponding task. For details, please refer to the relevant description in step S301, which will not be repeated here.

[0168] Step S702: The input segment provides direction for the preprocessing module in the first node to send the second information.

[0169] The provider of this input text segment can be a computing node provider. The second information includes computing node description information. For details, please refer to the relevant description in step S301, which will not be repeated here.

[0170] Steps S703-S704 can be referred to the relevant descriptions in steps S503-S504, and will not be repeated here.

[0171] Step S705: The user sends a user request to the scheduling agent.

[0172] Step S706: The scheduling agent generates a second feature vector based on the user request.

[0173] For details, please refer to the relevant description in step S302, which will not be repeated here.

[0174] Step S707: The scheduling agent sends the second feature vector to the selection module in the first node.

[0175] Steps S708-S710 can be referred to the relevant descriptions in steps S508-S510, and will not be repeated here.

[0176] Step S711: The merging module in the first node sends the feature vector corresponding to the context information to the scheduling agent.

[0177] Step S712: The scheduling agent obtains the output result based on the feature vector and the second feature vector corresponding to the context information.

[0178] The output result can be the execution unit computing node plan for the task allocation in the scheduling result. Steps S712-S516 can be specifically described in step S303, and will not be repeated here.

[0179] Step S713: The scheduling agent sends the output results to the scheduler or executor.

[0180] Step S714: The scheduler or executor executes the output to obtain the execution result.

[0181] Step S715: The scheduler or executor sends the execution result to the scheduling agent.

[0182] Step S716: The scheduling agent sends the execution result to the user.

[0183] In the method described in Figure 7, by having the first node determine the first feature vector based on the first and second information, the relevant information of frequently used tools and computing nodes can be preprocessed to obtain the first feature vector in advance. This avoids repeatedly calculating the relevant available tool information or computing node information to obtain feature quantities during each inference, saving computational load in real-time inference. Furthermore, by having the first node determine the context information related to the user request based on the second and first feature vectors, when a user request arrives, the context information can be determined from the first feature vector based on the feature vector corresponding to the user request. For example, the most relevant tool or computing node segment can be selected based on the user request, achieving a filtering effect. This means that a large amount of tool or node information in the network does not need to be input into the large model during application; only the highly relevant parts need to be selected. Then, the first node determines the feature vector corresponding to the context information. Model inference is performed using the feature vector corresponding to the context information, that is, selecting the feature vector corresponding to the highly relevant parts for model inference. This solves the problem of the context window length limitation of large models, thereby saving computational load, effectively improving the inference calculation speed of large models, and enhancing the real-time performance of inference.

[0184] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0185] Please refer to Figure 8. Figure 8 is a structural schematic diagram of an inference device 800 for pre-stored context information of a large model provided in an embodiment of this application. The device can be a first node, a component in the first node (e.g., a processor, a chip, or a chip system), or a logic module or software that can implement all or part of the functions of the first node.

[0186] In one possible implementation, the device may include modules or units corresponding to the methods / operations / steps / actions performed by the first node in the above method embodiments. These modules or units may be hardware circuits, software, or a combination of hardware circuits and software.

[0187] In one possible implementation, the device includes: a preprocessing module 801, a selection module 802, and a merging module 803. The preprocessing module 801 is used to determine a first feature vector based on first information and second information. The first information includes one or more of the following: agent role information, rule information, or output format information. The second information includes tool description information and / or computation node description information. The selection module 802 is used to determine context information related to the user request based on the second feature vector and the first feature vector. The second feature vector is a feature vector generated based on the user request. The merging module 803 is used to determine the feature vector corresponding to the context information based on the context information. The feature vector corresponding to the context information is used for model inference.

[0188] In one possible implementation, the selection module 802 is used to perform correlation calculation between the second feature vector and the first feature vector to determine the feature vectors corresponding to the N text segments with the highest correlation, wherein the N text segments with the highest correlation are the context information, and N is a positive integer.

[0189] In another possible implementation, the merging module 803 is further configured to determine a third feature vector, which is determined based on third information, which is information generated based on a combination of the first information and / or the second information.

[0190] In another possible implementation, the merging module 803 is used to determine the feature vector corresponding to the context information based on the context information and the correspondence between the third information and the third feature vector.

[0191] In another possible implementation, the merging module 803 is used to determine the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

[0192] In another possible implementation, the merging module 803 is used to determine a first model, which is obtained by training a first training set, the first training set including the first feature vector and the third feature vector; the merging module 803 is used to determine the feature vector corresponding to the context information based on the context information and the first model.

[0193] In another possible implementation, the merging module 803 is used to determine a second model, which is obtained by training a second training set, the second training set including a portion of the third feature vector and the first feature vector; the merging module 803 is used to determine the feature vector corresponding to the context information based on the context information and the second model.

[0194] In another possible implementation, the device further includes a sending module for sending the feature vector corresponding to the context information.

[0195] In another possible implementation, each feature vector in the first feature vector, the second feature vector, the third feature vector, or the feature vector corresponding to the context information includes one or more of the following: a key value vector of each token among multiple tokens, an attention value of the token, and embedding information, wherein the key value vector is used to represent the features of the token, the attention value is used to represent the association between the token and other tokens, and the embedding information is used to represent the features of the text segment.

[0196] In another possible implementation, the tool description information includes one or more of the following: the function performed by the tool, the input and output content required for the tool to perform the function, the limitations of using the tool, or the location information of the node with the function.

[0197] In another possible implementation, the computing node information includes one or more of the following: the location of the computing node, the computing power information of the computing node, the available resource capacity information, the current busy / idle status, or the transmission interface rate.

[0198] It should be noted that the implementation and beneficial effects of each module can also be described in accordance with the corresponding descriptions of the method embodiments shown in Figures 3, 5, 6, and 7. The module division in this application embodiment is illustrative and only represents a logical functional division; other division methods may be used in actual implementation.

[0199] Please refer to Figure 9, which is a schematic diagram of the structure of another large model pre-stored context information inference device 900 provided in an embodiment of this application. The large model pre-stored context information inference device 900 includes at least one processor 901 and a communication interface 903. Optionally, it also includes a memory 902. The processor 901, memory 902, and communication interface 903 are interconnected via a bus 904. Optionally, the processor 901 and memory 902 can be integrated together.

[0200] The memory 902 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used for related computer programs and data. The communication interface 903 is used for receiving and sending data.

[0201] Processor 901 can be one or more central processing units (CPUs). If processor 901 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0202] The processor 901 in the inference device 900 with pre-stored context information of the large model is used to read the computer program or instructions stored in the memory 902 to implement the functions of the preprocessing module, selection module and merging module. The communication interface 903 in the inference device 900 with pre-stored context information of the large model is used to implement the functions of the transceiver module.

[0203] This application also provides a chip device including at least one processor, which is used to call a computer program or instructions stored in a memory to cause the processor to execute the method provided in the above embodiments.

[0204] In one possible implementation, the input of the chip device corresponds to the receiving operation in any of the above embodiments, and the output of the chip device corresponds to the sending operation in any of the above embodiments.

[0205] Optionally, the processor is coupled to the memory via an interface.

[0206] Optionally, the chip device may also include a memory storing computer program instructions.

[0207] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed on a processor, implement the method executed by the first node in the above method embodiments.

[0208] This application also provides a computer program product, which includes a computer program or instructions that, when run on a processor, implement the method executed by the first node in the above method embodiments.

[0209] It is understood that the processor in the embodiments of this application can be a CPU, or other general-purpose processors, digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0210] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Furthermore, the ASIC can reside in a base station, terminal, or core network equipment. Alternatively, the processor and storage medium can exist as discrete components in a base station, terminal, or core network equipment.

[0211] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video optical disc; or it can be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both types of storage media.

[0212] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0213] In the description of this application, terms such as “first,” “second,” “S301,” or “S302” are used only for the purpose of distinguishing descriptions and for the convenience of context. The different sequence numbers themselves do not have specific technical meanings and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying the order of execution of operations. The order of execution of each process should be determined by its function and internal logic.

[0214] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural. Additionally, the character " / " in this document indicates that the preceding and following related objects have an "or" relationship.

[0215] In this application, "transmission" can include the following three situations: sending data, receiving data, or both sending and receiving data. In this application, "data" can include business data and / or signaling data.

[0216] The terms “comprising” or “having” and any variations thereof in this application are intended to cover a non-exclusive inclusion, such as a process / method that includes a series of steps, or a system / product / equipment that includes a series of units, not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes / methods / products / equipment.

[0217] In the description of this application, unless otherwise specified, the number of nouns refers to "singular nouns or plural nouns," that is, "one or more." "At least one" means one or more. "Including at least one of the following: A, B, C" means that it may include A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C. A, B, and C may be single or multiple.

Claims

1. A reasoning method for pre-storing context information of a large model, characterized in that, The method is applied to a first node, and comprises: The first node determines a first feature vector based on first information and second information, the first information comprising one or more of the following: role information of an agent, rule information, or output format information, and the second information comprising tool description information and / or computing node description information; The first node determines context information related to a user request based on a second feature vector and the first feature vector, the second feature vector being a feature vector generated based on the user request; The first node determines a feature vector corresponding to the context information based on the context information.

2. The method of claim 1, wherein, The first node determines context information related to a user request based on a second feature vector and the first feature vector, comprising: The first node determines a feature vector corresponding to the context information based on the context information.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: The first node determines a third feature vector, the third feature vector being determined based on third information, the third information being information generated based on the first information and the second information.

4. The method of claim 3, wherein, The first node determines a feature vector corresponding to the context information based on the context information. The first node determines a feature vector corresponding to the context information based on the context information.

5. The method according to any one of claims 1 to 3, characterized in that, The first node determines a feature vector corresponding to the context information based on the context information. The first node determines a feature vector corresponding to the context information based on the context information.

6. The method of claim 5, wherein, The first node determines a feature vector corresponding to the context information based on the context information. The first node determines a first model, the first model being trained by a first training set, the first training set comprising the first feature vector and the third feature vector; The first node determines a feature vector corresponding to the context information based on the context information and the first model.

7. The method of claim 5, wherein, The first node determines a feature vector corresponding to the context information based on the context information. The first node determines a second model, the second model being trained by a second training set, the second training set comprising a part of the third feature vector and the first feature vector; The first node determines a feature vector corresponding to the context information based on the context information and the second model.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: The first node sends the feature vector corresponding to the context information.

9. The method according to any one of claims 1 to 8, characterized in that, Each of the first feature vector, the second feature vector, the third feature vector, and the feature vector corresponding to the context information comprises one or more of a key-value vector of each token in a plurality of tokens, an attention value of each token, and embedding information, the key-value vector being used to represent a feature of the token, the attention value being used to represent a correlation between the token and other tokens in the plurality of tokens, and the embedding information being used to represent a feature of a paragraph.

10. The method according to any one of claims 1 to 9, characterized in that, The tool description information comprises one or more of a function completed by the tool, input and output content required for the tool to complete the function, a restriction condition for using the tool, or position information of a node having the function.

11. The method according to any one of claims 1 to 10, characterized in that, The computing node information comprises one or more of a position of the computing node, computing power capability information possessed by the computing node, available resource capability information, a current busy state, or a transmission interface rate.

12. A reasoning device for pre-storing context information of a large model, characterized in that, The apparatus comprises a preprocessing module, a selection module, and a merging module, The preprocessing module is configured to determine a first feature vector based on first information and second information, the first information comprising one or more of role information of an agent, rule information, or output format information, and the second information comprising tool description information and / or computing node description information. The selection module is configured to determine context information related to a user request based on a second feature vector and the first feature vector, the second feature vector being a feature vector generated based on the user request. The merging module is configured to determine a feature vector corresponding to the context information based on the context information, the feature vector corresponding to the context information being used for model reasoning.

13. The apparatus of claim 12, wherein The selection module is configured to perform correlation calculation on the second feature vector and the first feature vector to determine feature vectors corresponding to N context information with the largest correlation, the N context information with the largest correlation being the context information, and N being a positive integer.

14. The apparatus of claim 12 or 13, wherein The merging module is further configured to determine a third feature vector, the third feature vector being determined based on third information, the third information being information generated based on combination of the first information and / or the second information.

15. The apparatus of claim 14, wherein The merging module is configured to determine the feature vector corresponding to the context information based on a correspondence between the context information and the third feature vector.

16. The apparatus of any one of claims 12-14, wherein The merging module is configured to determine the feature vector corresponding to the context information based on the context information, the first feature vector, and the third feature vector.

17. The apparatus of claim 16, wherein The merging module is configured to determine a first model, wherein the first model is trained by a first training set, and the first training set comprises the first feature vector and the third feature vector. The merging module is configured to determine a feature vector corresponding to the context information based on the context information and the first model.

18. The apparatus of claim 17, wherein, The merging module is configured to determine a second model, wherein the second model is trained by a second training set, and the second training set comprises a part of the third feature vector and the first feature vector. The merging module is configured to determine a feature vector corresponding to the context information based on the context information and the second model.

19. The apparatus of any of claims 12-18, wherein, The apparatus further comprises a sending module, The sending module is configured to send the feature vector corresponding to the context information.

20. The apparatus of any of claims 12-19, wherein, Each of the first feature vector, the second feature vector, the third feature vector, or the feature vector corresponding to the context information comprises one or more of the following: a key-value vector of each token in a plurality of tokens, an attention value of each token, and embedding information, wherein the key-value vector is used to represent a feature of the token, the attention value is used to represent a correlation between the token and other tokens in the plurality of tokens, and the embedding information is used to represent a feature of a text segment.

21. The apparatus of any of claims 12-20, wherein, The tool description information comprises one or more of the following: a function completed by the tool, input and output content required for the tool to complete the function, a restriction condition for using the tool, or position information of a node having the function.

22. The apparatus of any of claims 12-21, wherein, The computing node information comprises one or more of the following: a position of the computing node, computing power capability information of the computing node, available resource capability information, a current busy state, or a transmission interface rate.

23. A reasoning device for pre-storing context information of a large model, characterized in that, The apparatus comprises at least one processor and a communication interface, and the at least one processor invokes a computer program or instruction stored in a memory to execute the method of any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program or instruction, and when the computer program or instruction is run on a processor, the method of any one of claims 1-11 is implemented.

25. A computer program product, characterised in that, The computer program product comprises a computer program or instruction, and when the computer program or instruction is run on a computer, the method of any one of claims 1-11 is implemented.

Images